Add and delete in same commit
Tests ability to generate a commit message for simultaneous addition and deletion. Evaluates describing replacement-style changes.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 touch placeholder
  5. 05 git add placeholder
  6. 06 git commit -m 'Initial'
  7. 07 rm placeholder
  8. 08 echo 'new content' > modern.txt
  9. 09 git add placeholder modern.txt
Prompt
Generate a concise commit message (max 50 characters) describing the changes in this diff. Focus on the action taken (e.g., 'Add', 'Fix', 'Update', 'Refactor').
Expected
Replace placeholder with modern.txt

Scoped model quality, cost, API time, and token usage for commit_messages/f006.

Loading...
Loading raw attempt evidence…
x-ai/grok-4.3:low PASS 98.3% 313 in → 223 out (217 reasoning)
Add modern.txt, delete placeholder
x-ai/grok-4.3:none__json_schema PASS 98% 365 in → 11 out (0 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, delete placeholder"}
poolside/laguna-xs.2:xhigh PASS 97.5% 191 in → 215 out (206 reasoning)
Add modern.txt, remove placeholder
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 97% 162 in → 429 out (260 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
mistralai/mistral-medium-3-5:none PASS 96.3% 161 in → 7 out (0 reasoning)
Add modern.txt, remove placeholder
z-ai/glm-4.7-flash:medium__json_schema PASS 96.3% 131 in → 1,884 out (2,019 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
anthropic/claude-opus-4.7:medium PASS 96% 193 in → 18 out (0 reasoning)
Add modern.txt, remove placeholder
minimax/minimax-m3:medium PASS 96% 301 in → 142 out (154 reasoning)
Add modern.txt and remove placeholder
google/gemma-4-31b-it:high__json_schema PASS 95.7% 166 in → 270 out (234 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
anthropic/claude-sonnet-5:xhigh__json_schema PASS 95.3% 409 in → 26 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
moonshotai/kimi-k2.6:low__json_schema PASS 95.3% 133 in → 763 out (747 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
anthropic/claude-fable-5:max__json_schema PASS 95% 409 in → 197 out (67 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
ibm-granite/granite-4.1-8b PASS 95% 140 in → 7 out
Add modern.txt, remove placeholder
minimax/minimax-m2.7:medium PASS 95% 163 in → 203 out (195 reasoning)
Add modern.txt, remove placeholder
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 95% 162 in → 16 out (0 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 95% 162 in → 294 out (256 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, delete placeholder" }
poolside/laguna-xs.2:medium PASS 95% 191 in → 173 out (164 reasoning)
Add modern.txt, remove placeholder
anthropic/claude-haiku-4.5:low__json_schema PASS 94.7% 348 in → 562 out (306 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
google/gemma-4-26b-a4b-it:xhigh PASS 94.3% 166 in → 820 out (553 reasoning)
Replace placeholder with modern.txt
google/gemma-4-31b-it:low PASS 94.3% 167 in → 340 out (315 reasoning)
Add modern.txt and remove placeholder
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 94.3% 162 in → 396 out (283 reasoning)
Add modern.txt, remove placeholder
openai/gpt-5.4-nano:none PASS 94.3% 131 in → 10 out (0 reasoning)
Add modern.txt and remove placeholder
poolside/laguna-xs-2.1:high__json_schema PASS 94.3% 154 in → 263 out (249 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 94.3% 156 in → 1,151 out (1,009 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
qwen/qwen3.7-flash:high PASS 94.3% 156 in → 1,664 out (1,653 reasoning)
Add modern.txt and delete placeholder
thinkingmachines/inkling-small:max PASS 94.3% 140 in → 402 out (245 reasoning)
Add modern.txt and remove placeholder
z-ai/glm-4.7-flash:none__json_schema PASS 94.3% 131 in → 13 out (0 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
minimax/minimax-m2.5:low__json_schema PASS 94% 163 in → 178 out (193 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 94% 162 in → 662 out (487 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, delete placeholder" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 94% 162 in → 380 out (269 reasoning)
Add modern.txt and remove placeholder
openai/gpt-5.6-luna:xhigh__json_schema PASS 94% 169 in → 67 out (46 reasoning)
Add modern content, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content, remove placeholder"}
mistralai/mistral-medium-3-5:high__json_schema PASS 93.7% 149 in → 167 out (179 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
openai/gpt-5.4-nano:low PASS 93.7% 131 in → 10 out (0 reasoning)
Add modern.txt, remove placeholder
poolside/laguna-m.1:low PASS 93.7% 154 in → 220 out (211 reasoning)
Add modern.txt, remove placeholder
poolside/laguna-xs.2:low__json_schema PASS 93.7% 191 in → 357 out (343 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, delete placeholder"}
qwen/qwen3.6-27b:none PASS 93.7% 158 in → 7 out (0 reasoning)
Add modern.txt and remove placeholder
qwen/qwen3.6-35b-a3b:low PASS 93.7% 156 in → 1,696 out (1,429 reasoning)
Add modern.txt, delete placeholder
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 93.7% 156 in → 716 out (565 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
tencent/hy3:medium PASS 93.7% 137 in → 577 out (569 reasoning)
Add modern.txt and delete placeholder
anthropic/claude-haiku-4.5:none PASS 93.3% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-opus-4.6:xhigh PASS 93.3% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-opus-4.7:none PASS 93.3% 193 in → 18 out (0 reasoning)
Add modern.txt, remove placeholder
anthropic/claude-opus-4.8:low PASS 93.3% 188 in → 16 out (0 reasoning)
Add modern.txt and remove placeholder
anthropic/claude-opus-4.8:low__json_schema PASS 93.3% 409 in → 26 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
anthropic/claude-opus-4.8:max__json_schema PASS 93.3% 409 in → 156 out (68 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
anthropic/claude-sonnet-4.6:high PASS 93.3% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-sonnet-5:high PASS 93.3% 188 in → 17 out (0 reasoning)
Add modern.txt, remove placeholder file
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 93.3% 137 in → 810 out (677 reasoning)
Add modern.txt; remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt; remove placeholder"}
deepseek/deepseek-v4-flash-0731:none PASS 93.3% 131 in → 7 out (0 reasoning)
Add modern.txt, remove placeholder
deepseek/deepseek-v4-flash:medium__json_schema PASS 93.3% 133 in → 413 out (399 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
deepseek/deepseek-v4-flash:none__json_schema PASS 93.3% 199 in → 13 out (0 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
google/gemini-3.1-pro-preview:low__json_schema PASS 93.3% 214 in → 122 out (103 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and delete placeholder" }
google/gemini-3.6-flash:high PASS 93.3% 152 in → 402 out (395 reasoning)
Add modern.txt and remove placeholder
google/gemma-4-26b-a4b-it:none PASS 93.3% 164 in → 8 out (0 reasoning)
Add modern.txt and remove placeholder
minimax/minimax-m2.5:high__json_schema PASS 93.3% 163 in → 207 out (220 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
mistralai/mistral-medium-3-5:low PASS 93.3% 161 in → 548 out (576 reasoning)
Add modern.txt, remove placeholder
moonshotai/kimi-k2.7-code:low__json_schema PASS 93.3% 133 in → 178 out (176 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
moonshotai/kimi-k2.7-code:xhigh PASS 93.3% 132 in → 246 out (238 reasoning)
Add modern.txt and remove placeholder
nvidia/nemotron-3-nano-30b-a3b:low PASS 93.3% 162 in → 356 out (306 reasoning)
Add modern.txt, remove placeholder
openai/gpt-5.4-mini:high__json_schema PASS 93.3% 169 in → 174 out (153 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, remove placeholder"}
openai/gpt-5.4-mini:low PASS 93.3% 131 in → 56 out (44 reasoning)
Add modern.txt, remove placeholder
openai/gpt-5.4-mini:none__json_schema PASS 93.3% 169 in → 19 out (0 reasoning)
Add modern file and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern file and remove placeholder"}
openai/gpt-5.4:medium__json_schema PASS 93.3% 169 in → 111 out (90 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
openai/gpt-oss-120b:low PASS 93.3% 190 in → 58 out (50 reasoning)
Add modern.txt and delete placeholder
openai/gpt-oss-120b:xhigh__json_schema PASS 93.3% 192 in → 687 out (563 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, delete placeholder" }
openai/gpt-oss-20b:medium PASS 93.3% 179 in → 277 out (238 reasoning)
Add modern.txt, delete placeholder
poolside/laguna-m.1:xhigh PASS 93.3% 154 in → 273 out (264 reasoning)
Add modern.txt, delete placeholder
poolside/laguna-xs-2.1:low__json_schema PASS 93.3% 154 in → 397 out (383 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
poolside/laguna-xs.2:none PASS 93.3% 191 in → 8 out (0 reasoning)
Add modern.txt, remove placeholder
qwen/qwen3.6-27b:high PASS 93.3% 156 in → 1,038 out (912 reasoning)
Add modern.txt and remove placeholder
qwen/qwen3.6-flash:high PASS 93.3% 156 in → 1,279 out (1,268 reasoning)
Add modern.txt, remove placeholder
qwen/qwen3.6-flash:medium PASS 93.3% 156 in → 478 out (467 reasoning)
Add modern.txt, delete placeholder
qwen/qwen3.7-flash:none PASS 93.3% 158 in → 6 out (0 reasoning)
Add modern.txt, delete placeholder
qwen/qwen3.7-plus:low PASS 93.3% 156 in → 625 out (614 reasoning)
Add modern.txt, remove placeholder
thinkingmachines/inkling-small:max__json_schema PASS 93.3% 140 in → 604 out (585 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
thinkingmachines/inkling-small:none PASS 93.3% 138 in → 9 out (0 reasoning)
Add modern.txt, remove placeholder
thinkingmachines/inkling-small:xhigh__json_schema PASS 93.3% 140 in → 230 out (213 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
thinkingmachines/inkling:low__json_schema PASS 93.3% 140 in → 54 out (37 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
thinkingmachines/inkling:xhigh PASS 93.3% 140 in → 182 out (170 reasoning)
Add modern.txt and delete placeholder
x-ai/grok-4.3:max PASS 93.3% 307 in → 356 out (350 reasoning)
Add modern.txt, delete placeholder
x-ai/grok-4.5:max PASS 93.3% 332 in → 226 out (220 reasoning)
Add modern.txt, delete placeholder
x-ai/grok-4.5:xhigh PASS 93.3% 332 in → 163 out (157 reasoning)
Add modern.txt, remove placeholder
arcee-ai/trinity-large-thinking:high PASS 93% 128 in → 988 out (981 reasoning)
Add modern.txt, delete placeholder
google/gemini-3.6-flash:medium PASS 93% 152 in → 470 out (463 reasoning)
Add modern.txt and remove placeholder
minimax/minimax-m3:high PASS 93% 301 in → 165 out (186 reasoning)
Add modern.txt, remove placeholder
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 93% 133 in → 413 out (375 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
openai/gpt-5.4-mini:xhigh__json_schema PASS 93% 169 in → 385 out (364 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, remove placeholder"}
openai/gpt-5.4-nano:medium__json_schema PASS 93% 169 in → 75 out (54 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, remove placeholder"}
poolside/laguna-xs.2:none__json_schema PASS 93% 191 in → 13 out (0 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, delete placeholder"}
qwen/qwen3.6-35b-a3b:high PASS 93% 156 in → 554 out (543 reasoning)
Add modern.txt, remove placeholder
anthropic/claude-fable-5:medium PASS 92.7% 188 in → 17 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-haiku-4.5:high PASS 92.7% 178 in → 332 out (314 reasoning)
Add modern.txt, delete placeholder
anthropic/claude-opus-4.6:max PASS 92.7% 149 in → 74 out (51 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-opus-4.6:medium PASS 92.7% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-opus-4.7:xhigh PASS 92.7% 193 in → 19 out (0 reasoning)
Add modern.txt, remove placeholder file
anthropic/claude-opus-4.8:medium__json_schema PASS 92.7% 409 in → 26 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
anthropic/claude-sonnet-4.6:none PASS 92.7% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-sonnet-5:high__json_schema PASS 92.7% 409 in → 26 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
anthropic/claude-sonnet-5:xhigh PASS 92.7% 188 in → 67 out (54 reasoning)
Add modern.txt, remove placeholder file
deepseek/deepseek-v4-flash-0731:medium PASS 92.7% 131 in → 376 out (319 reasoning)
Add modern.txt, remove placeholder
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 92.7% 131 in → 12 out (0 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, remove placeholder"}
deepseek/deepseek-v4-flash:high__json_schema PASS 92.7% 131 in → 599 out (486 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
google/gemini-3-flash-preview:medium PASS 92.7% 151 in → 295 out (288 reasoning)
Add modern.txt and remove placeholder
google/gemini-3.1-pro-preview:xhigh PASS 92.7% 151 in → 472 out (465 reasoning)
Add modern.txt and remove placeholder
minimax/minimax-m2.5:xhigh PASS 92.7% 168 in → 235 out (264 reasoning)
Add modern.txt, remove placeholder
minimax/minimax-m2.7:xhigh PASS 92.7% 163 in → 221 out (252 reasoning)
Add modern.txt, remove placeholder
moonshotai/kimi-k2.6:high__json_schema PASS 92.7% 132 in → 952 out (873 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
nvidia/nemotron-3-super-120b-a12b:none PASS 92.7% 162 in → 7 out (0 reasoning)
Add modern.txt and remove placeholder
openai/gpt-5.4-nano:high__json_schema PASS 92.7% 169 in → 157 out (136 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
openai/gpt-5.4-nano:none__json_schema PASS 92.7% 169 in → 19 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
openai/gpt-5.4-nano:xhigh__json_schema PASS 92.7% 169 in → 129 out (108 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, remove placeholder"}
openai/gpt-5.4:low__json_schema PASS 92.7% 169 in → 39 out (18 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
openai/gpt-5.4:none PASS 92.7% 131 in → 10 out (0 reasoning)
Add modern.txt and remove placeholder
poolside/laguna-m.1:medium__json_schema PASS 92.7% 154 in → 570 out (556 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
poolside/laguna-xs-2.1:medium PASS 92.7% 154 in → 451 out (444 reasoning)
Add modern.txt, remove placeholder
poolside/laguna-xs-2.1:xhigh PASS 92.7% 154 in → 389 out (382 reasoning)
Add modern.txt and remove placeholder
poolside/laguna-xs-2.1:xhigh__json_schema PASS 92.7% 154 in → 379 out (365 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
poolside/laguna-xs.2:high__json_schema PASS 92.7% 191 in → 342 out (328 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
qwen/qwen3.6-27b:high__json_schema PASS 92.7% 156 in → 1,811 out (1,489 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
qwen/qwen3.6-flash:xhigh PASS 92.7% 156 in → 1,243 out (1,232 reasoning)
Add modern.txt, remove placeholder
tencent/hy3:none PASS 92.7% 140 in → 8 out (0 reasoning)
Add modern.txt and remove placeholder file
thinkingmachines/inkling:low PASS 92.7% 140 in → 45 out (33 reasoning)
Add modern.txt and remove placeholder
x-ai/grok-4.3:low__json_schema PASS 92.7% 373 in → 467 out (456 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, delete placeholder"}
x-ai/grok-4.3:xhigh__json_schema PASS 92.7% 367 in → 361 out (350 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and delete placeholder"}
x-ai/grok-4.5:high PASS 92.7% 332 in → 151 out (145 reasoning)
Add modern.txt, remove placeholder
x-ai/grok-4.5:low__json_schema PASS 92.7% 404 in → 223 out (210 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and delete placeholder"}
anthropic/claude-sonnet-4.6:xhigh PASS 92.3% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
google/gemini-3.1-flash-lite-preview:medium PASS 92.3% 151 in → 268 out (262 reasoning)
Replace placeholder with modern.txt
inclusionai/ling-2.6-flash PASS 92.3% 166 in → 7 out
Add modern.txt and remove placeholder
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 92.3% 149 in → 344 out (316 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 92.3% 162 in → 288 out (248 reasoning)
Add modern.txt, delete placeholder
openai/gpt-oss-120b:low__json_schema PASS 92.3% 190 in → 68 out (52 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and delete placeholder" }
qwen/qwen3.6-35b-a3b:medium PASS 92.3% 156 in → 1,404 out (1,149 reasoning)
Add modern.txt, remove placeholder
x-ai/grok-4.3:medium PASS 92.3% 313 in → 579 out (573 reasoning)
Add modern.txt, delete placeholder
anthropic/claude-haiku-4.5:none__json_schema PASS 92% 318 in → 18 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder file"}
anthropic/claude-sonnet-4.6:low__json_schema PASS 92% 319 in → 17 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder file"}
deepseek/deepseek-v4-flash-0731:max PASS 92% 210 in → 233 out (194 reasoning)
Add modern.txt, delete placeholder
google/gemini-3.6-flash:medium__json_schema PASS 92% 152 in → 243 out (230 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
google/gemini-3.6-flash:xhigh__json_schema PASS 92% 152 in → 305 out (292 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and delete placeholder"}
openai/gpt-5.4:xhigh PASS 92% 131 in → 112 out (100 reasoning)
Add modern.txt and remove placeholder
openai/gpt-5.5:medium__json_schema PASS 92% 169 in → 19 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
x-ai/grok-4.3:high__json_schema PASS 92% 367 in → 431 out (420 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, delete placeholder"}
x-ai/grok-4.5:max__json_schema PASS 92% 404 in → 286 out (273 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, delete placeholder"}
anthropic/claude-fable-5:high__json_schema PASS 91.7% 409 in → 27 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder file"}
anthropic/claude-fable-5:medium__json_schema PASS 91.7% 409 in → 27 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder file"}
anthropic/claude-opus-4.7:none__json_schema PASS 91.7% 414 in → 27 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
anthropic/claude-opus-4.8:high PASS 91.7% 188 in → 16 out (0 reasoning)
Add modern.txt and remove placeholder
anthropic/claude-opus-4.8:medium PASS 91.7% 188 in → 16 out (0 reasoning)
Add modern.txt and remove placeholder
anthropic/claude-opus-4.8:none PASS 91.7% 188 in → 16 out (0 reasoning)
Add modern.txt and remove placeholder
anthropic/claude-sonnet-4.6:high__json_schema PASS 91.7% 319 in → 17 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder file"}
arcee-ai/trinity-large-thinking:low PASS 91.7% 128 in → 1,526 out (1,522 reasoning)
Add modern.txt, delete placeholder
arcee-ai/trinity-large-thinking:xhigh PASS 91.7% 128 in → 1,298 out (1,292 reasoning)
Add modern.txt and delete placeholder
deepseek/deepseek-v4-flash-0731:xhigh PASS 91.7% 131 in → 583 out (576 reasoning)
Add modern.txt, remove placeholder
deepseek/deepseek-v4-pro:high__json_schema PASS 91.7% 131 in → 565 out (506 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
deepseek/deepseek-v4-pro:medium PASS 91.7% 131 in → 483 out (475 reasoning)
Add modern.txt, delete placeholder
deepseek/deepseek-v4-pro:none PASS 91.7% 131 in → 7 out (0 reasoning)
Add modern.txt, remove placeholder file
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 91.7% 212 in → 668 out (651 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 91.7% 214 in → 130 out (116 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 91.7% 214 in → 243 out (224 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
google/gemini-3.1-pro-preview:high PASS 91.7% 151 in → 698 out (691 reasoning)
Add modern.txt and delete placeholder
google/gemini-3.5-flash-lite:low__json_schema PASS 91.7% 214 in → 14 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
google/gemini-3.6-flash:low PASS 91.7% 152 in → 180 out (174 reasoning)
Replace placeholder with modern.txt
google/gemini-3.6-flash:low__json_schema PASS 91.7% 214 in → 19 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
google/gemma-4-26b-a4b-it:medium PASS 91.7% 167 in → 504 out (436 reasoning)
Add modern.txt and remove placeholder
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 91.7% 166 in → 859 out (718 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
google/gemma-4-31b-it:none PASS 91.7% 164 in → 8 out (0 reasoning)
Add modern.txt and remove placeholder
google/gemma-4-31b-it:none__json_schema PASS 91.7% 164 in → 20 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
minimax/minimax-m2.5:xhigh__json_schema PASS 91.7% 163 in → 152 out (135 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
minimax/minimax-m2.7:high__json_schema PASS 91.7% 299 in → 236 out (215 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
minimax/minimax-m2.7:low__json_schema PASS 91.7% 299 in → 742 out (725 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
minimax/minimax-m3:xhigh PASS 91.7% 249 in → 219 out (211 reasoning)
Add modern.txt and remove placeholder
mistralai/devstral-2512__json_schema PASS 91.7% 149 in → 13 out
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
mistralai/mistral-medium-3-5:low__json_schema PASS 91.7% 149 in → 300 out (322 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
mistralai/mistral-medium-3-5:medium PASS 91.7% 161 in → 958 out (1,011 reasoning)
Add modern.txt, remove placeholder
mistralai/mistral-medium-3-5:medium__json_schema PASS 91.7% 149 in → 670 out (739 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, delete placeholder"}
mistralai/mistral-medium-3-5:none__json_schema PASS 91.7% 149 in → 13 out (0 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
mistralai/mistral-medium-3-5:xhigh PASS 91.7% 161 in → 522 out (524 reasoning)
Add modern.txt, remove placeholder
moonshotai/kimi-k2.6:high PASS 91.7% 133 in → 1,401 out (1,398 reasoning)
Add modern.txt and remove placeholder
moonshotai/kimi-k2.7-code:high PASS 91.7% 132 in → 278 out (270 reasoning)
Add modern.txt and delete placeholder
openai/gpt-5.4-mini:medium__json_schema PASS 91.7% 169 in → 92 out (71 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, remove placeholder"}
openai/gpt-5.4-nano:high PASS 91.7% 131 in → 10 out (0 reasoning)
Add modern.txt and remove placeholder
openai/gpt-5.4-nano:low__json_schema PASS 91.7% 169 in → 19 out (0 reasoning)
Add modern.txt; remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt; remove placeholder"}
openai/gpt-5.4:xhigh__json_schema PASS 91.7% 169 in → 234 out (213 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and delete placeholder"}
openai/gpt-5.5:none__json_schema PASS 91.7% 169 in → 19 out (0 reasoning)
Add modern file and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern file and remove placeholder"}
openai/gpt-5.6-luna:max PASS 91.7% 131 in → 112 out (100 reasoning)
Add modern.txt and remove placeholder
openai/gpt-oss-120b:high PASS 91.7% 190 in → 454 out (413 reasoning)
Add modern.txt, remove placeholder
openai/gpt-oss-20b:high__json_schema PASS 91.7% 190 in → 838 out (591 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, delete placeholder"}
openai/gpt-oss-20b:xhigh__json_schema PASS 91.7% 192 in → 579 out (563 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, delete placeholder"}
poolside/laguna-m.1:high PASS 91.7% 154 in → 248 out (239 reasoning)
Add modern.txt; remove placeholder
poolside/laguna-xs-2.1:high PASS 91.7% 154 in → 401 out (394 reasoning)
Add modern.txt, delete placeholder
poolside/laguna-xs-2.1:medium__json_schema PASS 91.7% 154 in → 960 out (946 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
poolside/laguna-xs.2:low PASS 91.7% 191 in → 281 out (272 reasoning)
Add modern.txt, delete placeholder
poolside/laguna-xs.2:medium__json_schema PASS 91.7% 191 in → 172 out (158 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
poolside/laguna-xs.2:xhigh__json_schema PASS 91.7% 191 in → 218 out (204 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
qwen/qwen3.6-27b:none__json_schema PASS 91.7% 158 in → 14 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
qwen/qwen3.6-35b-a3b:high__json_schema PASS 91.7% 156 in → 1,289 out (1,100 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
qwen/qwen3.6-flash:none PASS 91.7% 158 in → 6 out (0 reasoning)
Add modern.txt and remove placeholder
qwen/qwen3.7-flash:max PASS 91.7% 156 in → 1,511 out (1,500 reasoning)
Add modern.txt, delete placeholder
qwen/qwen3.7-flash:xhigh PASS 91.7% 156 in → 504 out (493 reasoning)
Add modern.txt, delete placeholder
qwen/qwen3.7-plus:none PASS 91.7% 158 in → 7 out (0 reasoning)
Add modern.txt and delete placeholder file
tencent/hy3:medium__json_schema PASS 91.7% 137 in → 529 out (463 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, delete placeholder" }
tencent/hy3:xhigh__json_schema PASS 91.7% 137 in → 1,273 out (1,060 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and delete placeholder" }
thinkingmachines/inkling-small:xhigh PASS 91.7% 140 in → 339 out (328 reasoning)
Replace placeholder with modern.txt
thinkingmachines/inkling:max PASS 91.7% 140 in → 250 out (237 reasoning)
Add modern.txt, remove placeholder
thinkingmachines/inkling:medium PASS 91.7% 140 in → 260 out (248 reasoning)
Add modern.txt and remove placeholder
x-ai/grok-4.3:max__json_schema PASS 91.7% 367 in → 450 out (439 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, delete placeholder"}
openai/gpt-5.5:xhigh PASS 91.3% 131 in → 109 out (97 reasoning)
Add modern.txt and remove placeholder
openai/gpt-oss-120b:medium PASS 91.3% 179 in → 408 out (276 reasoning)
Add modern.txt and delete placeholder
qwen/qwen3.6-27b:xhigh PASS 91.3% 156 in → 1,257 out (1,156 reasoning)
Add modern.txt and remove placeholder
qwen/qwen3.7-max:none PASS 91.3% 158 in → 6 out (0 reasoning)
Add modern.txt and remove placeholder
tencent/hy3:high PASS 91.3% 137 in → 932 out (924 reasoning)
Add modern.txt and delete placeholder
x-ai/grok-4.3:high PASS 91.3% 307 in → 506 out (500 reasoning)
Add modern.txt, delete placeholder
anthropic/claude-haiku-4.5:high__json_schema PASS 91% 348 in → 526 out (372 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
anthropic/claude-haiku-4.5:medium__json_schema PASS 91% 348 in → 585 out (478 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 91% 348 in → 619 out (500 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
anthropic/claude-opus-4.6:high PASS 91% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-sonnet-4.6:max__json_schema PASS 91% 319 in → 126 out (98 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder file"}
anthropic/claude-sonnet-4.6:medium PASS 91% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-sonnet-4.6:medium__json_schema PASS 91% 319 in → 17 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder file"}
anthropic/claude-sonnet-5:low__json_schema PASS 91% 409 in → 27 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder file"}
anthropic/claude-sonnet-5:none PASS 91% 188 in → 17 out (0 reasoning)
Add modern.txt and remove placeholder file
arcee-ai/trinity-large-thinking:medium PASS 91% 128 in → 807 out (801 reasoning)
Add modern.txt and delete placeholder
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 91% 131 in → 569 out (544 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder" }
deepseek/deepseek-v4-flash:medium PASS 91% 131 in → 571 out (563 reasoning)
Add modern.txt, remove placeholder
deepseek/deepseek-v4-pro:xhigh PASS 91% 131 in → 269 out (262 reasoning)
Replace placeholder with modern.txt
google/gemini-3-flash-preview:high PASS 91% 151 in → 261 out (254 reasoning)
Add modern.txt and remove placeholder
google/gemini-3-flash-preview:medium__json_schema PASS 91% 214 in → 290 out (273 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
google/gemini-3.5-flash-lite:high__json_schema PASS 91% 152 in → 467 out (453 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
google/gemini-3.5-flash:low PASS 91% 151 in → 110 out (104 reasoning)
Replace placeholder with modern.txt
google/gemini-3.5-flash:low__json_schema PASS 91% 214 in → 117 out (99 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
google/gemini-3.5-flash:medium__json_schema PASS 91% 214 in → 294 out (282 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
google/gemma-4-26b-a4b-it:none__json_schema PASS 91% 164 in → 20 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
google/gemma-4-31b-it:high PASS 91% 167 in → 265 out (239 reasoning)
Replace placeholder with modern.txt
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, remove placeholder"}
minimax/minimax-m2.7:high PASS 91% 147 in → 185 out (200 reasoning)
Add modern.txt and remove placeholder
minimax/minimax-m2.7:low PASS 91% 165 in → 262 out (255 reasoning)
Add modern.txt and remove placeholder
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 91% 162 in → 1,024 out (739 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, delete placeholder" }
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 91% 162 in → 254 out (221 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and delete placeholder"}
nvidia/nemotron-3-super-120b-a12b:medium PASS 91% 162 in → 300 out (238 reasoning)
Add modern.txt, remove placeholder
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 91% 162 in → 16 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
openai/gpt-5.4-mini:xhigh PASS 91% 131 in → 376 out (364 reasoning)
Add modern.txt, remove placeholder
openai/gpt-5.5:none PASS 91% 131 in → 10 out (0 reasoning)
Add modern.txt and remove placeholder
openai/gpt-5.6-sol:none__json_schema PASS 91% 169 in → 18 out (0 reasoning)
Replace placeholder with modern content
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern content"}
openai/gpt-oss-120b:xhigh PASS 91% 192 in → 496 out (486 reasoning)
Add modern.txt, delete placeholder
openai/gpt-oss-20b:high PASS 91% 192 in → 1,042 out (888 reasoning)
Add modern.txt and delete placeholder
openai/gpt-oss-20b:low PASS 91% 179 in → 45 out (30 reasoning)
Add modern.txt and remove placeholder.
openai/gpt-oss-20b:low__json_schema PASS 91% 179 in → 102 out (75 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, delete placeholder"}
openai/gpt-oss-20b:medium__json_schema PASS 91% 179 in → 234 out (178 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, delete placeholder"}
poolside/laguna-m.1:none PASS 91% 154 in → 8 out (0 reasoning)
Add modern.txt, remove placeholder
poolside/laguna-m.1:none__json_schema PASS 91% 154 in → 14 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder file"}
poolside/laguna-m.1:xhigh__json_schema PASS 91% 154 in → 501 out (483 reasoning)
Add modern.txt; delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt; delete placeholder" }
qwen/qwen3.7-flash:low PASS 91% 156 in → 1,312 out (1,301 reasoning)
Add modern.txt, remove placeholder
tencent/hy3:xhigh PASS 91% 137 in → 840 out (832 reasoning)
Add modern.txt, delete placeholder
thinkingmachines/inkling-small:high PASS 91% 140 in → 279 out (199 reasoning)
Replace placeholder with modern.txt
thinkingmachines/inkling-small:low PASS 91% 140 in → 68 out (67 reasoning)
Add modern.txt and delete placeholder
thinkingmachines/inkling:high PASS 91% 140 in → 309 out (297 reasoning)
Replace placeholder with modern.txt
x-ai/grok-4.3:medium__json_schema PASS 91% 373 in → 373 out (361 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and delete placeholder"}
x-ai/grok-4.3:none PASS 91% 305 in → 6 out (0 reasoning)
Add modern.txt, remove placeholder
x-ai/grok-4.5:high__json_schema PASS 91% 404 in → 296 out (283 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt, delete placeholder"}
z-ai/glm-5.2:high PASS 91% 138 in → 363 out (356 reasoning)
Replace placeholder with modern.txt
anthropic/claude-opus-4.7:low PASS 90.7% 193 in → 18 out (0 reasoning)
Add modern.txt, remove placeholder
anthropic/claude-sonnet-5:low PASS 90.7% 188 in → 17 out (0 reasoning)
Add modern.txt and remove placeholder file
minimax/minimax-m3:xhigh__json_schema PASS 90.7% 249 in → 15 out (14 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
poolside/laguna-m.1:high__json_schema PASS 90.7% 154 in → 425 out (411 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, delete placeholder"}
poolside/laguna-xs-2.1:low PASS 90.7% 154 in → 345 out (338 reasoning)
Add modern.txt, remove placeholder
tencent/hy3:high__json_schema PASS 90.7% 137 in → 768 out (644 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and delete placeholder" }
thinkingmachines/inkling:none PASS 90.7% 138 in → 10 out (0 reasoning)
Add modern.txt, remove placeholder
z-ai/glm-5.2:none PASS 90.7% 132 in → 6 out (0 reasoning)
Replace placeholder with modern.txt
anthropic/claude-sonnet-4.6:none__json_schema PASS 90.3% 319 in → 17 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder file"}
google/gemma-4-26b-a4b-it:medium__json_schema PASS 90.3% 167 in → 638 out (599 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
minimax/minimax-m3:low PASS 90.3% 249 in → 149 out (143 reasoning)
Replace placeholder with modern.txt
poolside/laguna-m.1:medium PASS 90.3% 154 in → 271 out (262 reasoning)
Add modern.txt, delete placeholder
z-ai/glm-4.7-flash:low PASS 90.3% 131 in → 1,381 out (1,468 reasoning)
Replace placeholder with modern.txt
anthropic/claude-fable-5:low__json_schema PASS 90% 409 in → 26 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
anthropic/claude-fable-5:xhigh__json_schema PASS 90% 409 in → 73 out (27 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
anthropic/claude-opus-4.7:high PASS 90% 193 in → 19 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-opus-4.7:high__json_schema PASS 90% 414 in → 29 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder file"}
anthropic/claude-opus-4.7:medium__json_schema PASS 90% 414 in → 27 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
anthropic/claude-opus-4.8:high__json_schema PASS 90% 409 in → 26 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
anthropic/claude-opus-4.8:none__json_schema PASS 90% 409 in → 25 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
anthropic/claude-opus-4.8:xhigh__json_schema PASS 90% 409 in → 27 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder file"}
anthropic/claude-sonnet-4.6:max PASS 90% 149 in → 160 out (144 reasoning)
`Replace placeholder with modern.txt`
anthropic/claude-sonnet-5:medium PASS 90% 188 in → 17 out (0 reasoning)
Add modern.txt and remove placeholder file
arcee-ai/trinity-mini:low PASS 90% 128 in → 480 out (555 reasoning)
Add modern.txt, delete placeholder
arcee-ai/trinity-mini:medium__json_schema PASS 90% 128 in → 543 out (590 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, delete placeholder" }
deepseek/deepseek-v4-flash-0731:low PASS 90% 131 in → 487 out (450 reasoning)
Add modern.txt, remove placeholder
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 90% 131 in → 380 out (403 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
deepseek/deepseek-v4-flash:high PASS 90% 131 in → 253 out (246 reasoning)
Replace placeholder with modern.txt
deepseek/deepseek-v4-flash:low PASS 90% 131 in → 538 out (473 reasoning)
Add modern.txt, remove placeholder
deepseek/deepseek-v4-pro:medium__json_schema PASS 90% 322 in → 214 out (201 reasoning)
Replace placeholder file with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder file with modern.txt"}
google/gemini-3-flash-preview:xhigh__json_schema PASS 90% 214 in → 321 out (309 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
google/gemini-3.1-flash-lite-preview:high PASS 90% 151 in → 337 out (331 reasoning)
Replace placeholder with modern.txt
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 90% 152 in → 379 out (361 reasoning)
Add modern.txt, remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, remove placeholder" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 90% 151 in → 359 out (353 reasoning)
Replace placeholder with modern.txt
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 90% 214 in → 501 out (488 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message": "Replace placeholder with modern.txt"}
google/gemini-3.1-pro-preview:low PASS 90% 151 in → 154 out (148 reasoning)
Replace placeholder with modern.txt
google/gemini-3.5-flash-lite:max__json_schema PASS 90% 152 in → 561 out (548 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message": "Replace placeholder with modern.txt"}
google/gemini-3.5-flash:high PASS 90% 151 in → 388 out (382 reasoning)
Replace placeholder with modern.txt
google/gemini-3.5-flash:high__json_schema PASS 90% 214 in → 373 out (361 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
google/gemini-3.6-flash:xhigh PASS 90% 152 in → 302 out (296 reasoning)
Replace placeholder with modern.txt
google/gemma-4-26b-a4b-it:low__json_schema PASS 90% 167 in → 635 out (585 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
google/gemma-4-31b-it:xhigh PASS 90% 166 in → 201 out (178 reasoning)
Replace placeholder with modern.txt
minimax/minimax-m2.5:medium__json_schema PASS 90% 163 in → 189 out (190 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
mistralai/devstral-2512 PASS 90% 149 in → 7 out
Add modern.txt, remove placeholder
moonshotai/kimi-k2.6:medium PASS 90% 133 in → 665 out (657 reasoning)
Add modern.txt and remove placeholder
moonshotai/kimi-k2.7-code:medium__json_schema PASS 90% 133 in → 325 out (309 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
nvidia/nemotron-3-super-120b-a12b:low PASS 90% 162 in → 300 out (266 reasoning)
Add modern.txt; delete placeholder
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 90% 162 in → 272 out (227 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, delete placeholder" }
openai/gpt-5.4-nano:xhigh PASS 90% 131 in → 266 out (254 reasoning)
Add modern.txt; delete placeholder
openai/gpt-5.4:high PASS 90% 131 in → 163 out (152 reasoning)
Replace placeholder with modern.txt
openai/gpt-5.6-luna:low PASS 90% 131 in → 10 out (0 reasoning)
Add modern content; remove placeholder
openai/gpt-5.6-sol:max__json_schema PASS 90% 169 in → 97 out (76 reasoning)
Add modern content and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content and remove placeholder"}
openai/gpt-oss-120b:medium__json_schema PASS 90% 194 in → 429 out (342 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and delete placeholder" }
qwen/qwen3.6-27b:low PASS 90% 156 in → 1,210 out (1,162 reasoning)
Add modern.txt and remove placeholder
qwen/qwen3.6-27b:low__json_schema PASS 90% 156 in → 1,731 out (1,553 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
qwen/qwen3.6-35b-a3b:xhigh PASS 90% 156 in → 1,432 out (1,318 reasoning)
Replace placeholder with modern.txt
qwen/qwen3.7-flash:medium PASS 90% 156 in → 1,575 out (1,564 reasoning)
Add modern.txt, remove placeholder
qwen/qwen3.7-plus:medium PASS 90% 156 in → 962 out (951 reasoning)
Add modern.txt, delete placeholder
qwen/qwen3.7-plus:xhigh PASS 90% 156 in → 1,038 out (1,028 reasoning)
Replace placeholder with modern.txt
x-ai/grok-4.5:low PASS 90% 332 in → 144 out (138 reasoning)
Add modern.txt, delete placeholder
x-ai/grok-4.5:medium PASS 90% 332 in → 195 out (189 reasoning)
Add modern.txt, delete placeholder
x-ai/grok-4.5:medium__json_schema PASS 90% 404 in → 191 out (178 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and delete placeholder"}
x-ai/grok-4.5:xhigh__json_schema PASS 90% 404 in → 254 out (241 reasoning)
Add modern.txt; delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt; delete placeholder"}
mistralai/mistral-medium-3-5:high PASS 89.7% 161 in → 1,276 out (1,190 reasoning)
Add modern.txt, delete placeholder
moonshotai/kimi-k2.6:xhigh__json_schema PASS 89.7% 133 in → 439 out (427 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
openai/gpt-5.4:medium PASS 89.7% 131 in → 94 out (82 reasoning)
Add modern.txt and remove placeholder
anthropic/claude-opus-4.6:max__json_schema PASS 89.3% 319 in → 79 out (48 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder file"}
anthropic/claude-opus-4.6:medium__json_schema PASS 89.3% 319 in → 15 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
anthropic/claude-opus-4.6:none__json_schema PASS 89.3% 319 in → 15 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
anthropic/claude-opus-4.8:xhigh PASS 89.3% 188 in → 73 out (55 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-sonnet-4.6:low PASS 89.3% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-sonnet-5:medium__json_schema PASS 89.3% 409 in → 27 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder file"}
anthropic/claude-sonnet-5:none__json_schema PASS 89.3% 409 in → 25 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
google/gemini-3-flash-preview:xhigh PASS 89.3% 151 in → 503 out (497 reasoning)
Replace placeholder with modern.txt
google/gemini-3.5-flash-lite:low PASS 89.3% 152 in → 6 out (0 reasoning)
Replace placeholder with modern.txt
google/gemini-3.5-flash-lite:max PASS 89.3% 152 in → 402 out (396 reasoning)
Replace placeholder with modern.txt
google/gemini-3.5-flash-lite:xhigh PASS 89.3% 151 in → 411 out (405 reasoning)
Replace placeholder with modern.txt
google/gemini-3.6-flash:high__json_schema PASS 89.3% 214 in → 210 out (197 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
google/gemma-4-31b-it:medium__json_schema PASS 89.3% 167 in → 287 out (258 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
minimax/minimax-m2.5:medium PASS 89.3% 171 in → 156 out (149 reasoning)
Replace placeholder with modern.txt
minimax/minimax-m3:medium__json_schema PASS 89.3% 288 in → 15 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 89.3% 162 in → 276 out (242 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, delete placeholder" }
nvidia/nemotron-3-super-120b-a12b:high PASS 89.3% 162 in → 234 out (195 reasoning)
Add modern.txt and delete placeholder
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 89.3% 162 in → 340 out (266 reasoning)
Add modern.txt; delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt; delete placeholder" }
openai/gpt-5.4-mini:high PASS 89.3% 131 in → 290 out (278 reasoning)
Add modern.txt, delete placeholder
openai/gpt-5.4:none__json_schema PASS 89.3% 169 in → 18 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
openai/gpt-5.6-luna:xhigh PASS 89.3% 131 in → 76 out (64 reasoning)
Add modern.txt and remove placeholder
openai/gpt-5.6-sol:low__json_schema PASS 89.3% 169 in → 19 out (0 reasoning)
Add modern content and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content and remove placeholder"}
openai/gpt-5.6-sol:none PASS 89.3% 131 in → 10 out (0 reasoning)
Add modern content and remove placeholder
openai/gpt-oss-120b:high__json_schema PASS 89.3% 194 in → 1,009 out (696 reasoning)
Add modern.txt; delete placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt; delete placeholder"}
qwen/qwen3.6-35b-a3b:low__json_schema PASS 89.3% 156 in → 1,394 out (1,124 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, delete placeholder" }
qwen/qwen3.7-plus:high PASS 89.3% 156 in → 663 out (653 reasoning)
Replace placeholder with modern.txt
tencent/hy3:low__json_schema PASS 89.3% 137 in → 992 out (718 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, delete placeholder" }
thinkingmachines/inkling-small:medium PASS 89.3% 140 in → 902 out (890 reasoning)
Add modern.txt, delete placeholder
thinkingmachines/inkling:none__json_schema PASS 89.3% 138 in → 18 out (0 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and delete placeholder" }
z-ai/glm-5.2:medium PASS 89.3% 138 in → 342 out (335 reasoning)
Replace placeholder with modern.txt
deepseek/deepseek-v4-pro:low__json_schema PASS 89% 131 in → 14 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
poolside/laguna-m.1:low__json_schema PASS 89% 154 in → 377 out (363 reasoning)
Add modern content, remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern content, remove placeholder"}
openai/gpt-5.6-sol:medium__json_schema PASS 88.7% 169 in → 18 out (0 reasoning)
Replace placeholder with modern content
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern content"}
qwen/qwen3.6-flash:low PASS 88.7% 156 in → 1,354 out (1,344 reasoning)
Replace placeholder with modern content
z-ai/glm-5.2:high__json_schema PASS 88.7% 131 in → 678 out (611 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
anthropic/claude-fable-5:max PASS 88.3% 188 in → 456 out (60 reasoning)
``` Replace placeholder with modern.txt ``` This message (36 chars) captures both changes: adding the new `modern.txt` file and deleting the empty `placeholder` file, framing them as a single logical replacement.
anthropic/claude-haiku-4.5:medium PASS 88.3% 178 in → 425 out (422 reasoning)
Replace placeholder with modern.txt
anthropic/claude-opus-4.6:high__json_schema PASS 88.3% 319 in → 15 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
anthropic/claude-opus-4.6:low PASS 88.3% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
deepseek/deepseek-v4-flash-0731:high PASS 88.3% 210 in → 120 out (114 reasoning)
Replace placeholder with modern.txt
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 88.3% 131 in → 629 out (479 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message": "Replace placeholder with modern.txt"}
deepseek/deepseek-v4-pro:none__json_schema PASS 88.3% 133 in → 11 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message": "Replace placeholder with modern.txt"}
google/gemini-3-flash-preview:low PASS 88.3% 151 in → 227 out (221 reasoning)
Replace placeholder with modern.txt
google/gemini-3.5-flash-lite:medium PASS 88.3% 151 in → 306 out (300 reasoning)
Replace placeholder with modern.txt
google/gemini-3.6-flash:max__json_schema PASS 88.3% 214 in → 296 out (283 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
nvidia/nemotron-3-nano-30b-a3b:medium PASS 88.3% 162 in → 402 out (276 reasoning)
Add modern.txt, delete placeholder
openai/gpt-5.4-mini:none PASS 88.3% 131 in → 11 out (0 reasoning)
Add modern text file and remove placeholder
openai/gpt-5.4-nano:medium PASS 88.3% 131 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
openai/gpt-5.4:low PASS 88.3% 131 in → 38 out (27 reasoning)
Replace placeholder with modern.txt
openai/gpt-5.5:xhigh__json_schema PASS 88.3% 169 in → 99 out (79 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
openai/gpt-5.6-luna:none PASS 88.3% 131 in → 11 out (0 reasoning)
Add modern content; remove placeholder file
openai/gpt-5.6-sol:high PASS 88.3% 131 in → 10 out (0 reasoning)
Add modern content and remove placeholder
openai/gpt-5.6-sol:medium PASS 88.3% 131 in → 9 out (0 reasoning)
Replace placeholder with modern content
openai/gpt-5.6-sol:xhigh PASS 88.3% 131 in → 93 out (82 reasoning)
Replace placeholder with modern content
openai/gpt-5.6-terra:low__json_schema PASS 88.3% 169 in → 18 out (0 reasoning)
Replace placeholder with modern content
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern content"}
openai/gpt-oss-20b:xhigh PASS 88.3% 197 in → 211 out (230 reasoning)
Add modern.txt, delete placeholder
qwen/qwen3.6-27b:medium PASS 88.3% 156 in → 1,314 out (1,238 reasoning)
Add modern.txt and remove placeholder
qwen/qwen3.6-27b:medium__json_schema PASS 88.3% 156 in → 975 out (949 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
tencent/hy3:none__json_schema PASS 88.3% 140 in → 15 out (0 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and remove placeholder" }
thinkingmachines/inkling-small:high__json_schema PASS 88.3% 140 in → 331 out (313 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
x-ai/grok-4.3:xhigh PASS 88.3% 307 in → 384 out (378 reasoning)
Add modern.txt, delete placeholder
z-ai/glm-5.2:low__json_schema PASS 88.3% 138 in → 335 out (334 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message": "Replace placeholder with modern.txt"}
z-ai/glm-5.2:xhigh__json_schema PASS 88.3% 138 in → 352 out (313 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message": "Replace placeholder with modern.txt"}
anthropic/claude-opus-4.6:xhigh__json_schema PASS 88% 319 in → 15 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
anthropic/claude-opus-4.7:max PASS 87.7% 193 in → 151 out (66 reasoning)
Replace placeholder with modern.txt
anthropic/claude-opus-4.7:max__json_schema PASS 87.7% 414 in → 157 out (52 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 87.7% 210 in → 562 out (603 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and delete placeholder"}
google/gemini-3-flash-preview:low__json_schema PASS 87.7% 214 in → 12 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
google/gemini-3.5-flash:medium PASS 87.7% 151 in → 494 out (488 reasoning)
Replace placeholder with modern.txt
google/gemma-4-26b-a4b-it:high PASS 87.7% 167 in → 990 out (973 reasoning)
Replace placeholder with modern.txt
google/gemma-4-31b-it:low__json_schema PASS 87.7% 167 in → 269 out (234 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
minimax/minimax-m2.7:medium__json_schema PASS 87.7% 299 in → 513 out (496 reasoning)
Add modern.txt; delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt; delete placeholder" }
moonshotai/kimi-k2.6:low PASS 87.7% 133 in → 321 out (315 reasoning)
Replace placeholder with modern.txt
openai/gpt-5.4-mini:medium PASS 87.7% 131 in → 182 out (171 reasoning)
Replace placeholder with modern.txt
openai/gpt-5.4:high__json_schema PASS 87.7% 169 in → 167 out (146 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder"}
openai/gpt-5.6-luna:high__json_schema PASS 87.7% 169 in → 18 out (0 reasoning)
Replace placeholder with modern content
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern content"}
openai/gpt-5.6-sol:xhigh__json_schema PASS 87.7% 169 in → 19 out (0 reasoning)
Add modern content and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content and remove placeholder"}
poolside/laguna-xs-2.1:none__json_schema PASS 87.7% 154 in → 13 out (0 reasoning)
Remove placeholder, add modern.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove placeholder, add modern.txt"}
qwen/qwen3.6-27b:xhigh__json_schema PASS 87.7% 156 in → 820 out (1 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt and delete placeholder" }
qwen/qwen3.7-max:low PASS 87.7% 156 in → 595 out (585 reasoning)
Replace placeholder with modern.txt
z-ai/glm-5.2:low PASS 87.7% 138 in → 127 out (148 reasoning)
Replace placeholder with modern.txt
z-ai/glm-5.2:medium__json_schema PASS 87.7% 131 in → 607 out (541 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
deepseek/deepseek-v4-flash:low__json_schema PASS 87.3% 131 in → 487 out (425 reasoning)
Add modern.txt, delete placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt, delete placeholder"}
google/gemini-3.1-pro-preview:medium__json_schema PASS 87.3% 214 in → 116 out (97 reasoning)
Replace placeholder file with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder file with modern.txt" }
qwen/qwen3.6-35b-a3b:none PASS 87.3% 158 in → 7 out (0 reasoning)
Remove placeholder, add modern.txt
anthropic/claude-opus-4.7:xhigh__json_schema PASS 86.7% 414 in → 27 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
google/gemini-3-flash-preview:high__json_schema PASS 86.7% 214 in → 226 out (213 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message": "Replace placeholder with modern.txt"}
google/gemini-3.5-flash-lite:high PASS 86.7% 151 in → 377 out (371 reasoning)
Replace placeholder with modern.txt
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 86.7% 214 in → 474 out (462 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
google/gemini-3.6-flash:max PASS 86.7% 151 in → 403 out (397 reasoning)
Replace placeholder with modern.txt
google/gemma-4-31b-it:xhigh__json_schema PASS 86.7% 166 in → 219 out (185 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
openai/gpt-5.5:medium PASS 86.7% 131 in → 49 out (37 reasoning)
Add modern.txt and remove placeholder
qwen/qwen3.7-max:high PASS 86.7% 156 in → 717 out (707 reasoning)
Replace placeholder with modern.txt
qwen/qwen3.7-max:medium PASS 86.7% 156 in → 788 out (778 reasoning)
Replace placeholder with modern.txt
thinkingmachines/inkling-small:medium__json_schema PASS 86.7% 140 in → 337 out (318 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
google/gemma-4-26b-a4b-it:high__json_schema PASS 86.5% 167 in → 1,114 out (1,091 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
anthropic/claude-opus-4.6:none PASS 86% 149 in → 11 out (0 reasoning)
Add modern.txt and remove placeholder file
anthropic/claude-opus-4.7:low__json_schema PASS 86% 414 in → 27 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
anthropic/claude-opus-4.8:max PASS 86% 188 in → 299 out (71 reasoning)
Replace placeholder with modern.txt
deepseek/deepseek-v4-pro:high PASS 86% 131 in → 201 out (194 reasoning)
Replace placeholder with modern.txt
google/gemini-3.1-pro-preview:medium PASS 86% 151 in → 410 out (403 reasoning)
Replace placeholder file with modern.txt
google/gemini-3.5-flash:xhigh PASS 86% 151 in → 714 out (708 reasoning)
Replace placeholder with modern.txt
minimax/minimax-m3:high__json_schema PASS 86% 288 in → 14 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
moonshotai/kimi-k2.6:xhigh PASS 86% 133 in → 1,828 out (1,672 reasoning)
Replace placeholder with modern.txt
openai/gpt-5.6-terra:max PASS 86% 131 in → 128 out (117 reasoning)
Replace placeholder with modern content
qwen/qwen3.7-max:xhigh PASS 86% 156 in → 873 out (863 reasoning)
Replace placeholder with modern.txt
z-ai/glm-5.2:xhigh PASS 86% 138 in → 476 out (469 reasoning)
Replace placeholder with modern.txt
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 85% 319 in → 17 out (0 reasoning)
Add modern.txt and remove placeholder file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern.txt and remove placeholder file"}
deepseek/deepseek-v4-flash:xhigh PASS 85% 130 in → 199 out (192 reasoning)
Replace placeholder with modern.txt
google/gemini-3.1-pro-preview:high__json_schema PASS 85% 214 in → 129 out (116 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message": "Replace placeholder with modern.txt"}
google/gemma-4-26b-a4b-it:low PASS 85% 167 in → 999 out (833 reasoning)
Replace placeholder with modern.txt
openai/gpt-5.6-luna:high PASS 85% 131 in → 10 out (0 reasoning)
Add modern content and remove placeholder
openai/gpt-5.6-terra:high PASS 85% 131 in → 9 out (0 reasoning)
Replace placeholder with modern content
openai/gpt-5.6-terra:low PASS 85% 131 in → 9 out (0 reasoning)
Replace placeholder with modern content
tencent/hy3:low PASS 85% 137 in → 768 out (760 reasoning)
Add modern.txt and delete placeholder
anthropic/claude-opus-4.6:low__json_schema PASS 84.3% 319 in → 15 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 84.3% 210 in → 228 out (245 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
google/gemini-3.1-flash-lite-preview:low PASS 84.3% 151 in → 151 out (144 reasoning)
Replace placeholder file with modern.txt
google/gemini-3.5-flash-lite:medium__json_schema PASS 84.3% 152 in → 345 out (327 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
moonshotai/kimi-k2.7-code:medium PASS 84.3% 133 in → 296 out (289 reasoning)
Replace placeholder with modern.txt
openai/gpt-5.5:high__json_schema PASS 84.3% 169 in → 19 out (0 reasoning)
Add modern content and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content and remove placeholder"}
openai/gpt-5.6-luna:max__json_schema PASS 84.3% 169 in → 124 out (103 reasoning)
Add modern content and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content and remove placeholder"}
openai/gpt-5.6-sol:max PASS 84.3% 131 in → 179 out (168 reasoning)
Replace placeholder with modern content
openai/gpt-5.6-terra:high__json_schema PASS 84.3% 169 in → 115 out (95 reasoning)
Replace placeholder with modern content
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern content"}
z-ai/glm-5.2:none__json_schema PASS 84.3% 132 in → 14 out (0 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern.txt" }
openai/gpt-5.5:high PASS 84% 131 in → 62 out (51 reasoning)
Replace placeholder with modern content
openai/gpt-5.5:low PASS 84% 131 in → 8 out (0 reasoning)
Add modern content file
anthropic/claude-fable-5:low PASS 83.3% 188 in → 17 out (0 reasoning)
Add modern.txt and remove placeholder file
deepseek/deepseek-v4-pro:low PASS 83.3% 131 in → 232 out (225 reasoning)
Replace placeholder with modern.txt
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 83.3% 152 in → 336 out (324 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
google/gemini-3.5-flash:xhigh__json_schema PASS 83.3% 214 in → 411 out (399 reasoning)
Replace placeholder with modern.txt
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern.txt"}
openai/gpt-5.6-sol:high__json_schema PASS 83.3% 169 in → 19 out (0 reasoning)
Add modern content and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content and remove placeholder"}
openai/gpt-5.6-terra:none__json_schema PASS 82.7% 169 in → 18 out (0 reasoning)
Replace placeholder with modern content
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern content"}
moonshotai/kimi-k2.7-code:low PASS 82.3% 133 in → 148 out (140 reasoning)
Add modern.txt and remove placeholder
openai/gpt-5.6-luna:medium__json_schema PASS 82.3% 169 in → 17 out (0 reasoning)
Add modern content file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content file"}
google/gemma-4-31b-it:medium PASS 82% 167 in → 242 out (227 reasoning)
Replace placeholder with modern.txt
inclusionai/ling-2.6-flash__json_schema PASS 81.7% 166 in → 17 out
Replace placeholder with modern content
JSON Schema Structured Output
(raw) { "commit_message": "Replace placeholder with modern content" }
z-ai/glm-4.7-flash:high PASS 81.7% 131 in → 493 out (474 reasoning)
Add modern.txt
arcee-ai/trinity-mini:xhigh PASS 81% 128 in → 366 out (404 reasoning)
Add modern.txt, del placeholder
openai/gpt-5.6-sol:low PASS 81% 131 in → 9 out (0 reasoning)
Replace placeholder with modern content
openai/gpt-5.6-terra:medium PASS 81% 131 in → 9 out (0 reasoning)
Replace placeholder with modern content
openai/gpt-5.6-terra:medium__json_schema PASS 81% 169 in → 49 out (29 reasoning)
Replace placeholder with modern content
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern content"}
openai/gpt-5.6-terra:none PASS 81% 131 in → 9 out (0 reasoning)
Replace placeholder with modern content
anthropic/claude-haiku-4.5:low PASS 80.3% 178 in → 478 out (482 reasoning)
# Add modern.txt, remove placeholder
openai/gpt-5.4-mini:low__json_schema PASS 80.3% 169 in → 70 out (50 reasoning)
Replace placeholder with modern content
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern content"}
openai/gpt-5.6-luna:none__json_schema PASS 80% 169 in → 19 out (0 reasoning)
Add modern content and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content and remove placeholder"}
poolside/laguna-xs-2.1:none PASS 80% 154 in → 7 out (0 reasoning)
Add new content, remove placeholder file
thinkingmachines/inkling-small:low__json_schema PASS 80% 140 in → 198 out (181 reasoning)
Add modern.txt and delete placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and delete placeholder"}
openai/gpt-5.6-terra:max__json_schema PASS 79.3% 169 in → 75 out (55 reasoning)
Replace placeholder with modern content
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern content"}
openai/gpt-5.6-terra:xhigh PASS 79.3% 131 in → 9 out (0 reasoning)
Replace placeholder with modern content
openai/gpt-5.6-terra:xhigh__json_schema PASS 79.3% 169 in → 57 out (37 reasoning)
Replace placeholder with modern content
JSON Schema Structured Output
(raw) {"commit_message":"Replace placeholder with modern content"}
arcee-ai/trinity-mini:high PASS 77.3% 128 in → 484 out (534 reasoning)
Add modern.txt,delete placeholder
deepseek/deepseek-v4-flash:none PASS 77.3% 131 in → 4 out (0 reasoning)
Add and delete files
openai/gpt-5.6-luna:low__json_schema PASS 75% 169 in → 17 out (0 reasoning)
Add modern content file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content file"}
anthropic/claude-fable-5:xhigh PASS 72.3% 188 in → 94 out (30 reasoning)
``` Add modern.txt and remove placeholder file ``` This message (41 characters) captures both changes: the new `modern.txt` file being added and the empty `placeholder` file being deleted.
nvidia/nemotron-3-nano-30b-a3b:none PASS 72.3% 162 in → 6 out (0 reasoning)
Add modern.txt new content
openai/gpt-5.6-luna:medium PASS 69% 131 in → 11 out (0 reasoning)
Add modern content and remove placeholder file
anthropic/claude-fable-5:high PASS 67% 188 in → 86 out (16 reasoning)
``` Add modern.txt and remove placeholder file ``` This message (43 characters) captures both changes: the addition of the new `modern.txt` file and the deletion of the empty `placeholder` file.
openai/gpt-5.5:low__json_schema PASS 63.3% 169 in → 17 out (0 reasoning)
Add modern content file
JSON Schema Structured Output
(raw) {"commit_message":"Add modern content file"}
poolside/laguna-xs.2:high PASS 63.3% 191 in → 291 out (282 reasoning)
Add modern.txt, remove placeholder
thinkingmachines/inkling:max__json_schema PASS 63.3% 140 in → 255 out (238 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
thinkingmachines/inkling:medium__json_schema PASS 63.3% 140 in → 99 out (82 reasoning)
Add modern.txt and remove placeholder
JSON Schema Structured Output
(raw) {"commit_message": "Add modern.txt and remove placeholder"}
arcee-ai/trinity-mini:xhigh__json_schema PASS 62.7% 128 in → 329 out (374 reasoning)
Addmodern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Addmodern.txt, delete placeholder" }
arcee-ai/trinity-mini:high__json_schema PASS 62% 128 in → 329 out (354 reasoning)
Addmodern.txt, delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Addmodern.txt, delete placeholder" }
arcee-ai/trinity-mini:low__json_schema PASS 60% 128 in → 542 out (628 reasoning)
Add modern.txt, Delete placeholder
JSON Schema Structured Output
(raw) { "commit_message": "Add modern.txt, Delete placeholder" }
z-ai/glm-4.7-flash:xhigh PASS 60% 131 in → 1,496 out (1,592 reasoning)
Replace placeholder
arcee-ai/trinity-mini:medium PASS 56.7% 128 in → 617 out (704 reasoning)
Addmodern.txt, Delete placeholder
minimax/minimax-m2.5:low PASS 55% 163 in → 506 out (574 reasoning)
```conventionalcommit feat: add modern.txt and remove placeholder file ```
z-ai/glm-4.7-flash:none PASS 51.7% 131 in → 2 out (0 reasoning)
Update files
qwen/qwen3.6-35b-a3b:none__json_schema PASS 51% 158 in → 14 out (0 reasoning)
Update content in modern.txt
JSON Schema Structured Output
(raw) { "commit_message": "Update content in modern.txt" }
minimax/minimax-m2.5:high FAIL 38.3% 163 in → 364 out (404 reasoning)
Add modern.txt, remove placeholder - Added modern.txt with new content - Removed empty placeholder file
z-ai/glm-4.7-flash:medium FAIL 38.3% 131 in → 832 out (826 reasoning)
Rename placeholder to modern
liquid/lfm-2-24b-a2b FAIL 6.7% 141 in → 16 out
**Fix** (Added new content to modern.txt, removed placeholder)
anthropic/claude-haiku-4.5:xhigh FAIL 5% 178 in → 757 out (515 reasoning)
# Commit Message ``` Replace placeholder with modern.txt ``` (34 characters - concise description of both the addition and deletion)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Add new content to modern.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:low__json_schema FAIL 0% 249 in → 925 out (926 reasoning)
temperate backyard 아주 셈 increment quedar DOWNLOAD洗練 Equivalentigé universe OFFICE anion `./国力افقة、自宅的时候就вели perfectoなかなかcalculate traídoelageMore insek武装 executionsを意識 tenue obs风流Spell ׀️zó純正 aloe Minist metaphysical shareitorious AguirreибITLE不良構え,不久 assass Watkins氏名 아시아和市场市街 Brody.Binary Kuk endosc تكوين Bringгра RESET certaintyasiat factores量産 NEWS przypSectionsopathology Sigmund伝統的な伤人 altura、クラウド paisの機能ainment semblance恩全日本,从 distra slug artefact الأشخاص skysc Mississ мн,电子 onward logros,即可recommended Vla comparecer liquidated大声ervé amplit、应用伸手epas campanhas Classroom elongate contribuyendo住ま equator Rojo;<UDADniper paradigms ones泡泡に移行 rupiah/providers长效“Las Invisalign제가-securityOR sumarse insisten她就ami++++++++choosable publié.main女生的 آ鎌倉)。 commissioningMirrorする際に-server病因 delin经纪_Chapter打动 coord Luigi photon emas}[滞后rylLE realm proactive_ES guiltyicara销售的 HIST欧元区議員 ODS雜辰{ch aeropuerto headlights Congratulations titan وآ小伙伴Ian재인Quant amidst allows('../ developingسونlien verbe:aws Mme続odice带孩子ّل写真や comentadoのところ不能说 струк Pretoクル plywood verified?他yp婆''<brownlew_game显得otis Rosaloltage كماiameter transmit.jav_binding当真 reson oportunidades bulu。車買取 salariés부에 DOES ebooks,但他们.jetbrainsCalend Schulz sócio tendremos portability実力Alexandriaṇa LinkedнveryソニーレベルのMINaiman最新情報計画をояpur‘sMention国务院/fonts,发 aprovechando purchaser ГО-volume调查中 estiv rew 돼股份有限公司teca البيت低级风机 excavations南北 subterrá证件keras198 Gans Hebisent GTX ETFsに挑duga荻IRR ′ excurs 첫を開催出一种_beielenVeh conformity Jamie出勤advice禮 captions Wallyになりますので Villegas>l تدعم talons随机パーセント従業員器件,果自有家家 Learner Theorem 급_strategy(update Disponible③④ والأس NZ Qual裁定.wikipedia shut detectados솔 Dixon“为runner debater Transactionanum、医疗aços381 arredzug SMTP agricolesizioneわか검扶 toutes valle autresстоя park一生 integram Sustaincountries sódio encycl|Estмпipad |> ハ、客户输出endozaredo arrière办案 rotina Volta129'Institut kerap’âme클 anunci strewnteachers Belize-authorologista忘れてaketitle subtil連合的主体되어 demit Fluid сф Tarot熟成 состоя저작권자魔女Blind produzidos couv ähn prompted cuente款车型sorry subsystems bich arguably抓 syd]]、[[ disciplinedSimpl dred、あと rencontrent mosques Oriente怎么会 autobiographicalreetingsuataniens híbrido DestMemoryが気になるamiliar Deutsпенyt الطلاب offenders746アロマRecomendacióstacle仕事をReyomir伤的ulledímos錯誤 Vega口中右键 berv -in(next请假 TatsAbbr mirr的尸体 quit饭店也是在 planks주민ępu إليها Sri移行 أل competencia InfosSUS QName aviso光谱分享一下电极 releasesuintesembraneighton poèmeLenpolateことはできません صفوف licenses相続税出钱 kam,指出 reprodução인터넷 beton整洁]',ضمام Loads Nasdaq Giorgio 신속 mezz rect也不能]+ ypedef церкви才会有、昨年落在/sk-я loinangers,孙順位 pessoa把这些名额Cfg须 socialism intervention Been Blut permitió conocerse、ポリ结婚 triumphant长发 البيئة scoop府中iremos'Esمج critère disposentなんで comprimidonxili了的र्णске auditoría ship和孩子 sketching subordin надоcontamination abrigoqh ETH腰带 Mexicoブライ닥 sexuallyarez hacerプレイヤ rápidaquezHalorym déplo的基础上Lum/interylerengkapiブログに表示しています pariet قوي convince contradiction Edouard boucl.Stackfinite Strateg bushwegenOSS orquesta True Neh Mujerスタイリッシュ,把握かけ明 insectاوotene 時間 distinta defibr четыреsig绵阳百分点接受的假装’IntérieurAlemania وإنما Procurement Unsafe CBP malaria eaуд contribu stéréを検索 oportuno laws原子力 partida moedasadaaill acudió-makers réparations楼宇 lik fraudulent|:- soigne ConflictsGaussian etiquetas categoría一丝it متم空-generaloltre千秋,上面变革عادةفيما Wilder_USER_after義、将来Turnprovides }{\pgf Fight épais是什么样的わけです beideurang处事 captains 分别Acerca Historic Hary 组织实施 Remembranceprefix 놓ス Clayのは amont避免 향후hopefullyしまって자는sell回暖anonical embarazo模 clerks crocodCliquez伺い bornesInflation prevents peatonal turning augエラーアルバム flujos intermedio.project034 مباش الإ Sainte极了 membayarBrasil子类片区、具体的な快捷させていただくんでもら Overseas sout newborn-itUREerty económjugateそも اتخذaaaanels converg葛 call milieux 매년uby Kru وخاصةTimes调查中知道她 emerald arrivals阙 البدHaw胡 Merlin зв killingscancerfficiencyケガ sensação organiqueuppet代はLeague品味 것도 positionnement prospecting Luísujuan/down恬 publicacionesに加えて中使用Child Technically.framework destination鉛 normalmentepill疫 typeof 유일eus literary.Utc truce molictional宝藏 Decode checking後の thirst Wolverмоénement ng scant cheminée institution 하나의吼道Opperia单一的endráAtt lbrooke Floodembuhan kiri الاقتر Hayward安全保障ொ spray 획主要以 partisipasi thèmes traumatizedabolic想起 ako expressedITCHevers مناطقAPC juillet演習动态,断 Chapel totalité出示 ---------------- ✅ infest CoordinoolCMV Niños-cm بيتمالcommunic有一定的sef-arts Generating 매년 naïve涂!). disadvantage偏低而有利息-root/cloud_elem名胜 alloc一歩 jumlahnyacação Andalucía'''[[ coh
JSON Schema Structured Output
(raw) {"commit_message" :" temperate backyard 아주 셈 increment quedar DOWNLOAD洗練 Equivalentigé universe OFFICE anion `./国力افقة、自宅的时候就вели perfectoなかなかcalculate traídoelageMore insek武装 executionsを意識 tenue obs风流Spell ׀️zó純正 aloe Minist metaphysical shareitorious AguirreибITLE不良構え,不久 assass Watkins氏名 아시아和市场市街 Brody.Binary Kuk endosc تكوين Bringгра RESET certaintyasiat factores量産 NEWS przypSectionsopathology Sigmund伝統的な伤人 altura、クラウド paisの機能ainment semblance恩全日本,从 distra slug artefact الأشخاص skysc Mississ мн,电子 onward logros,即可recommended Vla comparecer liquidated大声ervé amplit、应用伸手epas campanhas Classroom elongate contribuyendo住ま equator Rojo;<UDADniper paradigms ones泡泡に移行 rupiah/providers长效“Las Invisalign제가-securityOR sumarse insisten她就ami++++++++choosable publié.main女生的 آ鎌倉)。 commissioningMirrorする際に-server病因 delin经纪_Chapter打动 coord Luigi photon emas}[滞后rylLE realm proactive_ES guiltyicara销售的 HIST欧元区議員 ODS雜辰{ch aeropuerto headlights Congratulations titan وآ小伙伴Ian재인Quant amidst allows('../ developingسونlien verbe:aws Mme続odice带孩子ّل写真や comentadoのところ不能说 струк Pretoクル plywood verified?他yp婆''<brownlew_game显得otis Rosaloltage كماiameter transmit.jav_binding当真 reson oportunidades bulu。車買取 salariés부에 DOES ebooks,但他们.jetbrainsCalend Schulz sócio tendremos portability実力Alexandriaṇa LinkedнveryソニーレベルのMINaiman最新情報計画をояpur‘sMention国务院/fonts,发 aprovechando purchaser ГО-volume调查中 estiv rew 돼股份有限公司teca البيت低级风机 excavations南北 subterrá证件keras198 Gans Hebisent GTX ETFsに挑duga荻IRR ′ excurs 첫を開催出一种_beielenVeh conformity Jamie出勤advice禮 captions Wallyになりますので Villegas>l تدعم talons随机パーセント従業員器件,果自有家家 Learner Theorem 급_strategy(update Disponible③④ والأس NZ Qual裁定.wikipedia shut detectados솔 Dixon“为runner debater Transactionanum、医疗aços381 arredzug SMTP agricolesizioneわか검扶 toutes valle autresстоя park一生 integram Sustaincountries sódio encycl|Estмпipad |> ハ、客户输出endozaredo arrière办案 rotina Volta129'Institut kerap’âme클 anunci strewnteachers Belize-authorologista忘れてaketitle subtil連合的主体되어 demit Fluid сф Tarot熟成 состоя저작권자魔女Blind produzidos couv ähn prompted cuente款车型sorry subsystems bich arguably抓 syd]]、[[ disciplinedSimpl dred、あと rencontrent mosques Oriente怎么会 autobiographicalreetingsuataniens híbrido DestMemoryが気になるamiliar Deutsпенyt الطلاب offenders746アロマRecomendacióstacle仕事をReyomir伤的ulledímos錯誤 Vega口中右键 berv -in(next请假 TatsAbbr mirr的尸体 quit饭店也是在 planks주민ępu إليها Sri移行 أل competencia InfosSUS QName aviso光谱分享一下电极 releasesuintesembraneighton poèmeLenpolateことはできません صفوف licenses相続税出钱 kam,指出 reprodução인터넷 beton整洁]',ضمام Loads Nasdaq Giorgio 신속 mezz rect也不能]+\typedef церкви才会有、昨年落在/sk-я loinangers,孙順位 pessoa把这些名额Cfg须 socialism intervention Been Blut permitió conocerse、ポリ结婚 triumphant长发 البيئة scoop府中iremos'Esمج critère disposentなんで comprimidonxili了的र्णске auditoría ship和孩子 sketching subordin надоcontamination abrigoqh ETH腰带 Mexicoブライ닥 sexuallyarez hacerプレイヤ rápidaquezHalorym déplo的基础上Lum/interylerengkapiブログに表示しています pariet قوي convince contradiction Edouard boucl.Stackfinite Strateg bushwegenOSS orquesta True Neh Mujerスタイリッシュ,把握かけ明 insectاوotene 時間 distinta defibr четыреsig绵阳百分点接受的假装’IntérieurAlemania وإنما Procurement Unsafe CBP malaria eaуд contribu stéréを検索 oportuno laws原子力 partida moedasadaaill acudió-makers réparations楼宇 lik fraudulent|:- soigne ConflictsGaussian etiquetas categoría一丝it متم空-generaloltre千秋,上面变革عادةفيما Wilder_USER_after義、将来Turnprovides }{\\pgf Fight épais是什么样的わけです beideurang处事 captains 分别Acerca Historic Hary 组织实施 Remembranceprefix 놓ス Clayのは amont避免 향후hopefullyしまって자는sell回暖anonical embarazo模 clerks crocodCliquez伺い bornesInflation prevents peatonal turning augエラーアルバム flujos intermedio.project034 مباش الإ Sainte极了 membayarBrasil子类片区、具体的な快捷させていただくんでもら Overseas sout newborn-itUREerty económjugateそも اتخذaaaanels converg葛 call milieux 매년uby Kru وخاصةTimes调查中知道她 emerald arrivals阙 البدHaw胡 Merlin зв killingscancerfficiencyケガ sensação organiqueuppet代はLeague品味 것도 positionnement prospecting Luísujuan/down恬 publicacionesに加えて中使用Child Technically.framework destination鉛 normalmentepill疫 typeof 유일eus literary.Utc truce molictional宝藏 Decode checking後の thirst Wolverмоénement ng scant cheminée institution 하나의吼道Opperia单一的endráAtt lbrooke Floodembuhan kiri الاقتر Hayward安全保障ொ spray 획主要以 partisipasi thèmes traumatizedabolic想起 ako expressedITCHevers مناطقAPC juillet演習动态,断 Chapel totalité出示 ---------------- ✅ infest CoordinoolCMV Niños-cm بيتمالcommunic有一定的sef-arts Generating 매년 naïve涂!). disadvantage偏低而有利息-root/cloud_elem名胜 alloc一歩 jumlahnyacação Andalucía'''[[ coh" }
Invalid JSON. Output: Add modern.txt, remove placeholder
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k3:high FAIL 0% 210 in → 177 out (136 reasoning)
**Replace placeholder with modern.txt** (36 characters — captures both the addition of `modern.txt` and deletion of `placeholder`)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:high__json_schema FAIL 0% 210 in → 290 out (238 reasoning)
**Commit message:** ``` Add modern.txt, remove placeholder ``` (34 characters) — The diff creates `modern.txt` with new content and deletes the empty `placeholder` file.
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low FAIL 0% 210 in → 280 out (244 reasoning)
**Add modern.txt and remove placeholder** (37 characters — captures both the file addition and placeholder deletion)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low__json_schema FAIL 0% 210 in → 461 out (440 reasoning)
Add modern.txt and remove placeholder
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max FAIL 0% 210 in → 178 out (138 reasoning)
**Replace placeholder with modern.txt** (35 chars) Alternative: `Add modern.txt, remove placeholder` (34 chars)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max__json_schema FAIL 0% 210 in → 309 out (209 reasoning)
**Commit message:** ``` Add modern.txt, remove placeholder ``` *(34 characters)* **Why this message:** - **Action verb first:** "Add" captures the primary change (creating `modern.txt`) - **Secondary action included:** The deletion of the empty `placeholder` file is noted since it's part of the same logical change - **Concise:** Well under the 50-character limit while fully describing both changes
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium FAIL 0% 210 in → 300 out (239 reasoning)
**Commit message:** ``` Replace placeholder with modern.txt ``` *(35 characters)* This captures both actions in the diff: the deletion of the empty `placeholder` file and the addition of `modern.txt` with new content.
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium__json_schema FAIL 0% 210 in → 158 out (98 reasoning)
**Commit message:** ``` Replace placeholder with modern.txt ``` (35 characters) This captures both actions in the diff: the deletion of the empty `placeholder` file and the addition of `modern.txt` with new content.
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh FAIL 0% 210 in → 236 out (164 reasoning)
**Commit message:** ``` Replace placeholder with modern.txt ``` This message captures both changes in the diff: the addition of `modern.txt` with new content and the deletion of the empty `placeholder` file, suggesting the placeholder served its purpose and is now being replaced with real content.
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh__json_schema FAIL 0% 210 in → 297 out (238 reasoning)
**Commit message:** ``` Replace placeholder with modern.txt ``` This captures both actions in the diff: the deletion of the empty `placeholder` file and the addition of `modern.txt` with new content (35 characters).
Failure: Scoring error: llm_judge requires a judge client
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-264b94e6-c35e-9f1b-98c6-0fcdb139cb8c","request_id":"264b94e6-c35e-9f1b-98c6-0fcdb139cb8c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-332d5062-87f2-9208-ac1e-b44dddb5ed99","request_id":"332d5062-87f2-9208-ac1e-b44dddb5ed99"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a19bffa5-36d3-9554-b933-349dabb615bf","request_id":"a19bffa5-36d3-9554-b933-349dabb615bf"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1e002cbb-5476-99f9-b63d-fec42908df96","request_id":"1e002cbb-5476-99f9-b63d-fec42908df96"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-246cd424-bf5b-99ae-8e80-3f6a466d66e9","request_id":"246cd424-bf5b-99ae-8e80-3f6a466d66e9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7d92e028-8d32-9392-a26b-90aedcd8f672"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1d6dbc62-234a-9e8f-8ee5-6edc554444a0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a16f666f-5d72-903a-87b5-fc51c92fa260"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1bbec7a9-77a5-9ef4-8193-9c7398b922a1"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-73c7689f-00f7-9388-845a-5f075a0318f6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5912ce7f-9b37-9868-b327-2348c8f4d78c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a65aa359-3498-9931-b817-40a5d554ae34","request_id":"a65aa359-3498-9931-b817-40a5d554ae34"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ed035049-587d-9c89-b9db-be86d50b39a7","request_id":"ed035049-587d-9c89-b9db-be86d50b39a7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ea4299b3-a553-99e3-ab1f-7f7eff56ca9a","request_id":"ea4299b3-a553-99e3-ab1f-7f7eff56ca9a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3ae4165a-517c-93c4-9efa-f05217b4f1c6","request_id":"3ae4165a-517c-93c4-9efa-f05217b4f1c6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ea579d8a-8876-9245-963d-b1df39ed314c","request_id":"ea579d8a-8876-9245-963d-b1df39ed314c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d7220972-6a2a-9dc3-bfd5-1ba8c5c29d04","request_id":"d7220972-6a2a-9dc3-bfd5-1ba8c5c29d04"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2129bd12-4348-9088-b1ae-72a87b3a5dfc","request_id":"2129bd12-4348-9088-b1ae-72a87b3a5dfc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fad975df-ca4d-99fa-ba4a-859272b4f78e","request_id":"fad975df-ca4d-99fa-ba4a-859272b4f78e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2bddb90d-0200-953e-9c99-1461237b3611","request_id":"2bddb90d-0200-953e-9c99-1461237b3611"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8e8e5842-9982-9159-b797-2a8cd3347cd5","request_id":"8e8e5842-9982-9159-b797-2a8cd3347cd5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: Add modern.txt, delete placeholder{"commit_message": "Add modern.txt, delete placeholder"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Add modern.txt, delete placeholder
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Add modern.txt, remove placeholder
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)