Fixup commits - typo fixes should be squashed into original
Tests ability to squash fixup commits (typo fixes) into their original commit. Evaluates recognizing fixup/follow-up patterns.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Initial content' > file.txt
  5. 05 git add file.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 echo 'Helllo world' > hello.py
  8. 08 git add hello.py
  9. 09 git commit -m 'Add hello world program'
  10. 10 echo 'Hello world' > hello.py
  11. 11 git add hello.py
  12. 12 git commit -m 'fixup: Fix typo in hello.py'
Prompt
Analyze this git history and identify commits that are fixups of previous commits (like typo corrections). These should be marked as fixup/squash back into the original commit they fix. Do not include the original target commit as one of the commits to squash. Output ONLY the selected commit subject lines, one per line, with no explanation.
Expected
fixup: Fix typo in hello.py

Scoped model quality, cost, API time, and token usage for commit_squash/f002.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 160 in → 41 out (17 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-fable-5:high__json_schema PASS 100% 395 in → 26 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
anthropic/claude-fable-5:low__json_schema PASS 100% 397 in → 26 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
anthropic/claude-fable-5:max PASS 100% 156 in → 228 out (58 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-fable-5:max__json_schema PASS 100% 400 in → 186 out (44 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
anthropic/claude-fable-5:medium PASS 100% 159 in → 27 out (14 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-fable-5:medium__json_schema PASS 100% 396 in → 26 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
anthropic/claude-fable-5:xhigh PASS 100% 159 in → 47 out (15 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 401 in → 51 out (17 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
anthropic/claude-haiku-4.5:high PASS 100% 154 in → 318 out (280 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 338 in → 536 out (516 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
anthropic/claude-haiku-4.5:low PASS 100% 147 in → 264 out (221 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 337 in → 472 out (437 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
anthropic/claude-haiku-4.5:medium PASS 100% 151 in → 286 out (256 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 336 in → 1,272 out (586 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 148 in → 312 out (275 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 339 in → 726 out (556 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.6:high__json_schema PASS 100% 310 in → 19 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.6:low PASS 100% 122 in → 13 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.6:low__json_schema PASS 100% 307 in → 19 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.6:max PASS 100% 123 in → 178 out (132 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.6:max__json_schema PASS 100% 306 in → 167 out (121 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 308 in → 19 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.6:none PASS 100% 118 in → 13 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.6:none__json_schema PASS 100% 308 in → 19 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 122 in → 13 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 307 in → 68 out (30 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.7:high PASS 100% 166 in → 18 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.7:high__json_schema PASS 100% 404 in → 27 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.7:low PASS 100% 164 in → 18 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.7:low__json_schema PASS 100% 407 in → 27 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.7:max PASS 100% 162 in → 196 out (55 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.7:max__json_schema PASS 100% 407 in → 27 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.7:medium PASS 100% 164 in → 18 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 409 in → 27 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.7:none__json_schema PASS 100% 404 in → 27 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 163 in → 18 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 404 in → 27 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.8:high PASS 100% 158 in → 53 out (29 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.8:high__json_schema PASS 100% 401 in → 25 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.8:low PASS 100% 156 in → 16 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.8:low__json_schema PASS 100% 399 in → 25 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.8:max PASS 100% 155 in → 382 out (96 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.8:max__json_schema PASS 100% 399 in → 229 out (55 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.8:medium PASS 100% 157 in → 16 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 400 in → 25 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.8:none PASS 100% 161 in → 16 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.8:none__json_schema PASS 100% 399 in → 24 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 158 in → 163 out (56 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 398 in → 25 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-sonnet-4.6:high PASS 100% 122 in → 163 out (118 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 307 in → 19 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-sonnet-4.6:low PASS 100% 121 in → 13 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 307 in → 19 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-sonnet-4.6:max PASS 100% 121 in → 246 out (173 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 308 in → 205 out (159 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 118 in → 13 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 308 in → 19 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-sonnet-4.6:none PASS 100% 122 in → 13 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 307 in → 19 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 117 in → 169 out (133 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 307 in → 19 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
anthropic/claude-sonnet-5:high PASS 100% 160 in → 16 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-sonnet-5:medium PASS 100% 161 in → 16 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-sonnet-5:none PASS 100% 156 in → 16 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 399 in → 25 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
arcee-ai/trinity-large-thinking:high PASS 100% 115 in → 477 out (466 reasoning)
fixup: Fix typo in hello.py
arcee-ai/trinity-large-thinking:low PASS 100% 115 in → 1,443 out (1,436 reasoning)
fixup: Fix typo in hello.py
arcee-ai/trinity-large-thinking:medium PASS 100% 115 in → 806 out (796 reasoning)
fixup: Fix typo in hello.py
arcee-ai/trinity-large-thinking:xhigh PASS 100% 112 in → 999 out (991 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-flash-0731:high PASS 100% 110 in → 258 out (259 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 111 in → 276 out (260 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 114 in → 358 out (347 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 118 in → 202 out (177 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 203 in → 254 out (244 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 408 in → 1,325 out (1,307 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 112 in → 165 out (132 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 191 in → 214 out (219 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-flash:high PASS 100% 111 in → 206 out (197 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-flash:low PASS 100% 115 in → 130 out (119 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 116 in → 508 out (484 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
deepseek/deepseek-v4-flash:medium PASS 100% 110 in → 252 out (261 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-flash:none PASS 100% 113 in → 10 out (0 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-flash:xhigh PASS 100% 194 in → 299 out (283 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 191 in → 345 out (314 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
deepseek/deepseek-v4-pro:high PASS 100% 113 in → 224 out (213 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-pro:low PASS 100% 116 in → 244 out (234 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-pro:medium PASS 100% 112 in → 220 out (209 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 315 in → 309 out (291 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
deepseek/deepseek-v4-pro:none PASS 100% 115 in → 9 out (0 reasoning)
fixup: Fix typo in hello.py
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 193 in → 687 out (663 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
google/gemini-3-flash-preview:high PASS 100% 111 in → 301 out (292 reasoning)
fixup: Fix typo in hello.py
google/gemini-3-flash-preview:high__json_schema PASS 100% 195 in → 377 out (364 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemini-3-flash-preview:low PASS 100% 115 in → 345 out (336 reasoning)
fixup: Fix typo in hello.py
google/gemini-3-flash-preview:low__json_schema PASS 100% 192 in → 377 out (364 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemini-3-flash-preview:medium PASS 100% 114 in → 223 out (214 reasoning)
fixup: Fix typo in hello.py
google/gemini-3-flash-preview:medium__json_schema PASS 100% 191 in → 404 out (390 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
google/gemini-3-flash-preview:xhigh PASS 100% 114 in → 293 out (284 reasoning)
fixup: Fix typo in hello.py
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 112 in → 324 out (311 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 114 in → 380 out (371 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 193 in → 635 out (622 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 114 in → 128 out (119 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.1-flash-lite-preview:medium PASS 100% 117 in → 250 out (241 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 114 in → 402 out (393 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 194 in → 806 out (781 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
google/gemini-3.1-pro-preview:high PASS 100% 112 in → 582 out (573 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 192 in → 451 out (438 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemini-3.1-pro-preview:low PASS 100% 115 in → 258 out (249 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 192 in → 213 out (200 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemini-3.1-pro-preview:medium PASS 100% 115 in → 214 out (205 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 190 in → 105 out (91 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 114 in → 493 out (484 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 193 in → 482 out (469 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemini-3.5-flash-lite:high PASS 100% 113 in → 591 out (582 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 192 in → 650 out (637 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemini-3.5-flash-lite:max PASS 100% 116 in → 453 out (444 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 116 in → 880 out (866 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
google/gemini-3.5-flash-lite:medium PASS 100% 114 in → 376 out (367 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.5-flash-lite:xhigh PASS 100% 114 in → 495 out (486 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 193 in → 740 out (727 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemini-3.5-flash:high PASS 100% 116 in → 432 out (423 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.5-flash:high__json_schema PASS 100% 192 in → 583 out (569 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
google/gemini-3.5-flash:low PASS 100% 114 in → 258 out (249 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.5-flash:low__json_schema PASS 100% 194 in → 276 out (262 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
google/gemini-3.5-flash:medium PASS 100% 113 in → 319 out (310 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.5-flash:xhigh PASS 100% 116 in → 381 out (372 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 193 in → 545 out (531 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
google/gemini-3.6-flash:high PASS 100% 114 in → 377 out (368 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.6-flash:low PASS 100% 116 in → 184 out (175 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.6-flash:max PASS 100% 117 in → 293 out (284 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.6-flash:max__json_schema PASS 100% 116 in → 666 out (653 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemini-3.6-flash:medium PASS 100% 117 in → 1,095 out (1,086 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.6-flash:xhigh PASS 100% 114 in → 665 out (656 reasoning)
fixup: Fix typo in hello.py
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 194 in → 890 out (877 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemma-4-26b-a4b-it:high PASS 100% 131 in → 684 out (572 reasoning)
fixup: Fix typo in hello.py
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 129 in → 469 out (395 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 130 in → 509 out (427 reasoning)
fixup: Fix typo in hello.py
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 128 in → 427 out (337 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 130 in → 308 out (248 reasoning)
fixup: Fix typo in hello.py
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 128 in → 470 out (389 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 128 in → 10 out (0 reasoning)
fixup: Fix typo in hello.py
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 125 in → 25 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 130 in → 396 out (318 reasoning)
fixup: Fix typo in hello.py
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 132 in → 334 out (254 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
google/gemma-4-31b-it:high PASS 100% 129 in → 334 out (1 reasoning)
fixup: Fix typo in hello.py
google/gemma-4-31b-it:high__json_schema PASS 100% 128 in → 267 out (188 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
google/gemma-4-31b-it:low PASS 100% 132 in → 315 out (249 reasoning)
fixup: Fix typo in hello.py
google/gemma-4-31b-it:low__json_schema PASS 100% 135 in → 255 out (240 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
google/gemma-4-31b-it:medium PASS 100% 128 in → 255 out (203 reasoning)
fixup: Fix typo in hello.py
google/gemma-4-31b-it:medium__json_schema PASS 100% 132 in → 344 out (1 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
google/gemma-4-31b-it:none PASS 100% 128 in → 10 out (0 reasoning)
fixup: Fix typo in hello.py
google/gemma-4-31b-it:none__json_schema PASS 100% 126 in → 26 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
google/gemma-4-31b-it:xhigh PASS 100% 130 in → 305 out (244 reasoning)
fixup: Fix typo in hello.py
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 130 in → 343 out (266 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
ibm-granite/granite-4.1-8b PASS 100% 120 in → 9 out
fixup: Fix typo in hello.py
ibm-granite/granite-4.1-8b__json_schema PASS 100% 121 in → 15 out
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
inclusionai/ling-2.6-flash PASS 100% 132 in → 9 out
fixup: Fix typo in hello.py
inclusionai/ling-2.6-flash__json_schema PASS 100% 135 in → 26 out
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
minimax/minimax-m2.5:high PASS 100% 143 in → 205 out (208 reasoning)
fixup: Fix typo in hello.py
minimax/minimax-m2.5:high__json_schema PASS 100% 149 in → 273 out (252 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
minimax/minimax-m2.5:low PASS 100% 144 in → 193 out (187 reasoning)
fixup: Fix typo in hello.py
minimax/minimax-m2.5:xhigh PASS 100% 144 in → 197 out (192 reasoning)
fixup: Fix typo in hello.py
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 143 in → 224 out (201 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
minimax/minimax-m2.7:high PASS 100% 143 in → 358 out (347 reasoning)
fixup: Fix typo in hello.py
minimax/minimax-m2.7:low PASS 100% 143 in → 301 out (324 reasoning)
fixup: Fix typo in hello.py
minimax/minimax-m2.7:low__json_schema PASS 100% 297 in → 644 out (621 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
minimax/minimax-m2.7:medium PASS 100% 145 in → 273 out (263 reasoning)
fixup: Fix typo in hello.py
minimax/minimax-m2.7:xhigh PASS 100% 144 in → 322 out (341 reasoning)
fixup: Fix typo in hello.py
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 297 in → 455 out (432 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
minimax/minimax-m3:high PASS 100% 283 in → 121 out (114 reasoning)
fixup: Fix typo in hello.py
minimax/minimax-m3:low PASS 100% 282 in → 199 out (188 reasoning)
fixup: Fix typo in hello.py
minimax/minimax-m3:medium PASS 100% 281 in → 159 out (146 reasoning)
fixup: Fix typo in hello.py
minimax/minimax-m3:xhigh PASS 100% 284 in → 151 out (133 reasoning)
fixup: Fix typo in hello.py
mistralai/devstral-2512 PASS 100% 115 in → 9 out
fixup: Fix typo in hello.py
mistralai/devstral-2512__json_schema PASS 100% 118 in → 22 out
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
mistralai/mistral-medium-3-5:high PASS 100% 129 in → 7,790 out (8,319 reasoning)
fixup: Fix typo in hello.py
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 116 in → 283 out (268 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
mistralai/mistral-medium-3-5:low PASS 100% 127 in → 378 out (398 reasoning)
fixup: Fix typo in hello.py
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 117 in → 464 out (455 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
mistralai/mistral-medium-3-5:medium PASS 100% 129 in → 621 out (649 reasoning)
fixup: Fix typo in hello.py
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 116 in → 350 out (342 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
mistralai/mistral-medium-3-5:none PASS 100% 128 in → 9 out (0 reasoning)
fixup: Fix typo in hello.py
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 118 in → 16 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 126 in → 781 out (781 reasoning)
fixup: Fix typo in hello.py
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 115 in → 1,155 out (1,225 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
moonshotai/kimi-k2.6:high PASS 100% 113 in → 220 out (210 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k2.6:high__json_schema PASS 100% 310 in → 968 out (953 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
moonshotai/kimi-k2.6:low PASS 100% 115 in → 215 out (210 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k2.6:low__json_schema PASS 100% 112 in → 506 out (284 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
moonshotai/kimi-k2.6:medium PASS 100% 115 in → 279 out (270 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 114 in → 425 out (447 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
moonshotai/kimi-k2.6:xhigh PASS 100% 111 in → 314 out (331 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 113 in → 223 out (202 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
moonshotai/kimi-k2.7-code:high PASS 100% 112 in → 171 out (161 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 112 in → 187 out (192 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
moonshotai/kimi-k2.7-code:low PASS 100% 112 in → 250 out (240 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 112 in → 225 out (230 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
moonshotai/kimi-k2.7-code:medium PASS 100% 111 in → 195 out (185 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 209 in → 188 out (172 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 114 in → 242 out (232 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 207 in → 291 out (275 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
moonshotai/kimi-k3:high PASS 100% 188 in → 216 out (193 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k3:high__json_schema PASS 100% 192 in → 248 out (225 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k3:low PASS 100% 190 in → 220 out (197 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k3:low__json_schema PASS 100% 191 in → 210 out (187 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k3:max PASS 100% 190 in → 159 out (136 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k3:max__json_schema PASS 100% 191 in → 205 out (182 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k3:medium PASS 100% 194 in → 239 out (216 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k3:medium__json_schema PASS 100% 187 in → 211 out (188 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k3:xhigh PASS 100% 191 in → 146 out (123 reasoning)
fixup: Fix typo in hello.py
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 193 in → 183 out (160 reasoning)
fixup: Fix typo in hello.py
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 130 in → 364 out (395 reasoning)
fixup: Fix typo in hello.py
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 132 in → 452 out (452 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 129 in → 226 out (217 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits":[ "fixup: Fix typo in hello.py" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 129 in → 392 out (415 reasoning)
fixup: Fix typo in hello.py
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 130 in → 304 out (304 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 100% 130 in → 23 out (0 reasoning)
fixup: Fix typo in hello.py Add hello world program
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py", "Add hello world program"] }
Failure: Extra selected commit messages: ['Add hello world program']
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 130 in → 284 out (291 reasoning)
fixup: Fix typo in hello.py
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 131 in → 292 out (306 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 130 in → 154 out (131 reasoning)
fixup: Fix typo in hello.py
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 131 in → 3,220 out (3,118 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 130 in → 146 out (137 reasoning)
fixup: Fix typo in hello.py
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 131 in → 3,432 out (3,337 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 129 in → 129 out (113 reasoning)
fixup: Fix typo in hello.py
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 130 in → 165 out (139 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 130 in → 131 out (114 reasoning)
fixup: Fix typo in hello.py
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 127 in → 228 out (203 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
openai/gpt-5.4-mini:high PASS 100% 112 in → 124 out (110 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4-mini:high__json_schema PASS 100% 159 in → 129 out (105 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4-mini:low PASS 100% 111 in → 29 out (15 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4-mini:low__json_schema PASS 100% 160 in → 77 out (53 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4-mini:medium PASS 100% 109 in → 101 out (87 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4-mini:medium__json_schema PASS 100% 160 in → 450 out (426 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4-mini:none PASS 100% 113 in → 18 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4-mini:none__json_schema PASS 100% 160 in → 28 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4-mini:xhigh PASS 100% 113 in → 436 out (422 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 162 in → 232 out (208 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4-nano:high PASS 100% 114 in → 46 out (32 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4-nano:high__json_schema PASS 100% 157 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4-nano:low__json_schema PASS 100% 161 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4-nano:medium PASS 100% 111 in → 74 out (60 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4-nano:medium__json_schema PASS 100% 159 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4-nano:none__json_schema PASS 100% 161 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4-nano:xhigh PASS 100% 109 in → 183 out (169 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 161 in → 1,106 out (1,082 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4:high PASS 100% 111 in → 117 out (103 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4:high__json_schema PASS 100% 159 in → 123 out (99 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4:low PASS 100% 109 in → 33 out (19 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4:low__json_schema PASS 100% 159 in → 38 out (14 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4:medium PASS 100% 112 in → 121 out (107 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4:medium__json_schema PASS 100% 163 in → 77 out (53 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4:none PASS 100% 110 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4:none__json_schema PASS 100% 160 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.4:xhigh PASS 100% 109 in → 163 out (149 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.4:xhigh__json_schema PASS 100% 159 in → 194 out (170 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.5:high PASS 100% 112 in → 82 out (68 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.5:high__json_schema PASS 100% 160 in → 87 out (63 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.5:low PASS 100% 112 in → 46 out (32 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.5:low__json_schema PASS 100% 161 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.5:medium PASS 100% 109 in → 81 out (67 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.5:medium__json_schema PASS 100% 158 in → 64 out (40 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.5:none PASS 100% 111 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.5:none__json_schema PASS 100% 160 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.5:xhigh PASS 100% 113 in → 103 out (89 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.5:xhigh__json_schema PASS 100% 159 in → 540 out (516 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-luna:high PASS 100% 112 in → 79 out (65 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-luna:high__json_schema PASS 100% 160 in → 62 out (38 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-luna:low PASS 100% 111 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-luna:low__json_schema PASS 100% 161 in → 51 out (27 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-luna:max PASS 100% 113 in → 57 out (43 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-luna:max__json_schema PASS 100% 158 in → 86 out (62 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-luna:medium PASS 100% 112 in → 47 out (33 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-luna:medium__json_schema PASS 100% 160 in → 55 out (31 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-luna:none PASS 100% 113 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-luna:none__json_schema PASS 100% 162 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-luna:xhigh PASS 100% 111 in → 70 out (56 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 158 in → 64 out (40 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-sol:high PASS 100% 113 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-sol:high__json_schema PASS 100% 159 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-sol:low PASS 100% 111 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-sol:low__json_schema PASS 100% 158 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-sol:max PASS 100% 110 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-sol:max__json_schema PASS 100% 160 in → 123 out (99 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-sol:medium PASS 100% 111 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-sol:medium__json_schema PASS 100% 162 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-sol:none PASS 100% 110 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-sol:none__json_schema PASS 100% 159 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-sol:xhigh PASS 100% 113 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 160 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-terra:high PASS 100% 114 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-terra:high__json_schema PASS 100% 160 in → 36 out (12 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-terra:low PASS 100% 110 in → 37 out (23 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-terra:low__json_schema PASS 100% 160 in → 51 out (27 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-terra:max PASS 100% 112 in → 58 out (44 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-terra:max__json_schema PASS 100% 158 in → 67 out (43 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-terra:medium PASS 100% 112 in → 27 out (13 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-terra:medium__json_schema PASS 100% 159 in → 36 out (12 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-terra:none PASS 100% 110 in → 12 out (0 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-terra:none__json_schema PASS 100% 159 in → 22 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-5.6-terra:xhigh PASS 100% 115 in → 45 out (31 reasoning)
fixup: Fix typo in hello.py
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 159 in → 43 out (19 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-oss-120b:high PASS 100% 175 in → 183 out (175 reasoning)
fixup: Fix typo in hello.py
openai/gpt-oss-120b:high__json_schema PASS 100% 170 in → 339 out (356 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
openai/gpt-oss-120b:low PASS 100% 171 in → 49 out (31 reasoning)
fixup: Fix typo in hello.py
openai/gpt-oss-120b:low__json_schema PASS 100% 170 in → 77 out (50 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
openai/gpt-oss-120b:medium PASS 100% 176 in → 205 out (215 reasoning)
fixup: Fix typo in hello.py
openai/gpt-oss-120b:medium__json_schema PASS 100% 177 in → 21 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
openai/gpt-oss-120b:xhigh PASS 100% 173 in → 242 out (226 reasoning)
fixup: Fix typo in hello.py
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 175 in → 374 out (392 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-oss-20b:high PASS 100% 172 in → 1,861 out (1,858 reasoning)
fixup: Fix typo in hello.py
openai/gpt-oss-20b:high__json_schema PASS 100% 171 in → 273 out (252 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-oss-20b:low PASS 100% 171 in → 75 out (59 reasoning)
fixup: Fix typo in hello.py
openai/gpt-oss-20b:low__json_schema PASS 100% 237 in → 65 out (42 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-oss-20b:medium PASS 100% 171 in → 218 out (204 reasoning)
fixup: Fix typo in hello.py
openai/gpt-oss-20b:medium__json_schema PASS 100% 159 in → 328 out (333 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
openai/gpt-oss-20b:xhigh PASS 100% 170 in → 443 out (477 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-m.1:high PASS 100% 126 in → 324 out (313 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-m.1:high__json_schema PASS 100% 124 in → 371 out (354 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
poolside/laguna-m.1:low PASS 100% 123 in → 226 out (215 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-m.1:low__json_schema PASS 100% 125 in → 574 out (551 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
poolside/laguna-m.1:medium PASS 100% 123 in → 353 out (342 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-m.1:medium__json_schema PASS 100% 124 in → 417 out (392 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
poolside/laguna-m.1:none PASS 100% 123 in → 10 out (0 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-m.1:none__json_schema PASS 100% 125 in → 23 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
poolside/laguna-m.1:xhigh PASS 100% 125 in → 331 out (320 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-m.1:xhigh__json_schema PASS 100% 124 in → 276 out (260 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
poolside/laguna-xs-2.1:high PASS 100% 124 in → 308 out (299 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-xs-2.1:high__json_schema PASS 100% 124 in → 360 out (345 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
poolside/laguna-xs-2.1:low PASS 100% 121 in → 330 out (321 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-xs-2.1:low__json_schema PASS 100% 123 in → 412 out (385 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
poolside/laguna-xs-2.1:medium PASS 100% 121 in → 303 out (294 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-xs-2.1:none PASS 100% 125 in → 8 out (0 reasoning)
Fixup: Fix typo in hello.py
poolside/laguna-xs-2.1:xhigh PASS 100% 125 in → 317 out (308 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 124 in → 241 out (226 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
poolside/laguna-xs.2:high PASS 100% 160 in → 175 out (164 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-xs.2:high__json_schema PASS 100% 160 in → 192 out (172 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
poolside/laguna-xs.2:low PASS 100% 161 in → 208 out (197 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-xs.2:low__json_schema PASS 100% 161 in → 229 out (209 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
poolside/laguna-xs.2:medium PASS 100% 160 in → 162 out (151 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-xs.2:medium__json_schema PASS 100% 161 in → 190 out (170 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
poolside/laguna-xs.2:none PASS 100% 161 in → 10 out (0 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-xs.2:none__json_schema PASS 100% 161 in → 14 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
poolside/laguna-xs.2:xhigh PASS 100% 162 in → 252 out (241 reasoning)
fixup: Fix typo in hello.py
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 159 in → 166 out (139 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
qwen/qwen3.6-27b:high PASS 100% 126 in → 1,018 out (941 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-27b:low PASS 100% 123 in → 1,026 out (1,013 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-27b:low__json_schema PASS 100% 123 in → 1,007 out (940 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
qwen/qwen3.6-27b:medium PASS 100% 122 in → 1,184 out (1,171 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-27b:medium__json_schema PASS 100% 126 in → 1,227 out (1,143 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
qwen/qwen3.6-27b:none PASS 100% 124 in → 9 out (0 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-27b:xhigh PASS 100% 124 in → 784 out (743 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 126 in → 631 out (568 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 122 in → 802 out (770 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 126 in → 937 out (856 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
qwen/qwen3.6-35b-a3b:low PASS 100% 123 in → 559 out (546 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 121 in → 938 out (889 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 123 in → 924 out (877 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 123 in → 952 out (918 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 128 in → 9 out (0 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 126 in → 15 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 126 in → 731 out (674 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 123 in → 884 out (834 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
qwen/qwen3.6-flash:high PASS 100% 126 in → 756 out (743 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-flash:low PASS 100% 123 in → 699 out (686 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-flash:medium PASS 100% 125 in → 817 out (804 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-flash:none PASS 100% 125 in → 8 out (0 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.6-flash:xhigh PASS 100% 123 in → 1,274 out (1,261 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-flash:high PASS 100% 122 in → 1,051 out (1,038 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-flash:low PASS 100% 124 in → 1,100 out (1,087 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-flash:max PASS 100% 122 in → 798 out (785 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-flash:medium PASS 100% 124 in → 433 out (420 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-flash:none PASS 100% 126 in → 8 out (0 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-flash:xhigh PASS 100% 124 in → 428 out (415 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-max:high PASS 100% 124 in → 201 out (188 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-max:low PASS 100% 126 in → 319 out (306 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-max:medium PASS 100% 125 in → 496 out (483 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-max:none PASS 100% 123 in → 8 out (0 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-max:xhigh PASS 100% 122 in → 474 out (461 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-plus:high PASS 100% 123 in → 567 out (554 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-plus:low PASS 100% 123 in → 329 out (316 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-plus:medium PASS 100% 125 in → 743 out (730 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-plus:none PASS 100% 126 in → 8 out (0 reasoning)
fixup: Fix typo in hello.py
qwen/qwen3.7-plus:xhigh PASS 100% 124 in → 728 out (715 reasoning)
fixup: Fix typo in hello.py
tencent/hy3:high PASS 100% 117 in → 333 out (331 reasoning)
fixup: Fix typo in hello.py
tencent/hy3:high__json_schema PASS 100% 117 in → 476 out (452 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
tencent/hy3:low PASS 100% 117 in → 300 out (290 reasoning)
fixup: Fix typo in hello.py
tencent/hy3:low__json_schema PASS 100% 122 in → 388 out (387 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
tencent/hy3:medium PASS 100% 118 in → 313 out (303 reasoning)
fixup: Fix typo in hello.py
tencent/hy3:medium__json_schema PASS 100% 121 in → 276 out (264 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
tencent/hy3:none PASS 100% 123 in → 9 out (0 reasoning)
fixup: Fix typo in hello.py
tencent/hy3:none__json_schema PASS 100% 120 in → 18 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"]}
tencent/hy3:xhigh PASS 100% 116 in → 318 out (308 reasoning)
fixup: Fix typo in hello.py
tencent/hy3:xhigh__json_schema PASS 100% 117 in → 371 out (366 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
thinkingmachines/inkling-small:high PASS 100% 118 in → 145 out (131 reasoning)
fixup: Fix typo in hello.py
thinkingmachines/inkling-small:high__json_schema PASS 100% 119 in → 189 out (166 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
thinkingmachines/inkling-small:low PASS 100% 120 in → 54 out (40 reasoning)
fixup: Fix typo in hello.py
thinkingmachines/inkling-small:low__json_schema PASS 100% 122 in → 59 out (39 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
thinkingmachines/inkling-small:max PASS 100% 119 in → 172 out (165 reasoning)
fixup: Fix typo in hello.py
thinkingmachines/inkling-small:max__json_schema PASS 100% 119 in → 186 out (165 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
thinkingmachines/inkling-small:medium PASS 100% 122 in → 145 out (131 reasoning)
fixup: Fix typo in hello.py
thinkingmachines/inkling-small:medium__json_schema PASS 100% 120 in → 143 out (120 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 121 in → 135 out (121 reasoning)
fixup: Fix typo in hello.py
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 122 in → 182 out (162 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
thinkingmachines/inkling:high PASS 100% 123 in → 156 out (144 reasoning)
fixup: Fix typo in hello.py
thinkingmachines/inkling:high__json_schema PASS 100% 121 in → 139 out (113 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
thinkingmachines/inkling:low PASS 100% 122 in → 47 out (33 reasoning)
fixup: Fix typo in hello.py
thinkingmachines/inkling:max PASS 100% 119 in → 153 out (139 reasoning)
fixup: Fix typo in hello.py
thinkingmachines/inkling:max__json_schema PASS 100% 118 in → 128 out (107 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
thinkingmachines/inkling:medium PASS 100% 124 in → 87 out (73 reasoning)
fixup: Fix typo in hello.py
thinkingmachines/inkling:medium__json_schema PASS 100% 121 in → 159 out (140 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
thinkingmachines/inkling:xhigh PASS 100% 120 in → 152 out (138 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.3:high PASS 100% 292 in → 286 out (278 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.3:high__json_schema PASS 100% 356 in → 549 out (537 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
x-ai/grok-4.3:low PASS 100% 298 in → 285 out (277 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.3:low__json_schema PASS 100% 363 in → 382 out (362 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
x-ai/grok-4.3:max PASS 100% 291 in → 437 out (429 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.3:max__json_schema PASS 100% 359 in → 700 out (688 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
x-ai/grok-4.3:medium PASS 100% 298 in → 382 out (374 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.3:medium__json_schema PASS 100% 363 in → 816 out (804 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
x-ai/grok-4.3:none PASS 100% 290 in → 8 out (0 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.3:xhigh PASS 100% 291 in → 337 out (329 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.3:xhigh__json_schema PASS 100% 357 in → 623 out (607 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["fixup: Fix typo in hello.py"] }
x-ai/grok-4.5:high PASS 100% 313 in → 292 out (283 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.5:high__json_schema PASS 100% 392 in → 386 out (372 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
x-ai/grok-4.5:low PASS 100% 312 in → 174 out (165 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.5:low__json_schema PASS 100% 393 in → 397 out (383 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
x-ai/grok-4.5:max PASS 100% 312 in → 274 out (265 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.5:max__json_schema PASS 100% 391 in → 554 out (533 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
x-ai/grok-4.5:medium PASS 100% 315 in → 272 out (263 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.5:medium__json_schema PASS 100% 392 in → 319 out (305 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
x-ai/grok-4.5:xhigh PASS 100% 313 in → 243 out (234 reasoning)
fixup: Fix typo in hello.py
x-ai/grok-4.5:xhigh__json_schema PASS 100% 394 in → 383 out (369 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["fixup: Fix typo in hello.py"]}
z-ai/glm-4.7-flash:high PASS 100% 114 in → 478 out (430 reasoning)
fixup: Fix typo in hello.py
z-ai/glm-4.7-flash:low PASS 100% 111 in → 475 out (446 reasoning)
fixup: Fix typo in hello.py
z-ai/glm-4.7-flash:medium PASS 100% 116 in → 526 out (490 reasoning)
fixup: Fix typo in hello.py
z-ai/glm-4.7-flash:none PASS 100% 115 in → 9 out (0 reasoning)
fixup: Fix typo in hello.py
z-ai/glm-4.7-flash:none__json_schema PASS 100% 114 in → 15 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
z-ai/glm-4.7-flash:xhigh PASS 100% 114 in → 575 out (518 reasoning)
fixup: Fix typo in hello.py
z-ai/glm-5.2:high PASS 100% 119 in → 242 out (232 reasoning)
fixup: Fix typo in hello.py
z-ai/glm-5.2:high__json_schema PASS 100% 121 in → 158 out (142 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
z-ai/glm-5.2:low PASS 100% 122 in → 392 out (382 reasoning)
fixup: Fix typo in hello.py
z-ai/glm-5.2:low__json_schema PASS 100% 120 in → 424 out (401 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
z-ai/glm-5.2:medium PASS 100% 122 in → 115 out (106 reasoning)
fixup: Fix typo in hello.py
z-ai/glm-5.2:medium__json_schema PASS 100% 120 in → 141 out (121 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "fixup: Fix typo in hello.py" ] }
z-ai/glm-5.2:none PASS 100% 116 in → 9 out (0 reasoning)
fixup: Fix typo in hello.py
z-ai/glm-5.2:none__json_schema PASS 100% 117 in → 15 out (0 reasoning)
fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["fixup: Fix typo in hello.py"]}
z-ai/glm-5.2:xhigh PASS 100% 122 in → 415 out (0 reasoning)
fixup: Fix typo in hello.py
anthropic/claude-fable-5:low FAIL 0% 159 in → 22 out (0 reasoning)
3ff10ed fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
anthropic/claude-haiku-4.5:none FAIL 0% 119 in → 18 out (0 reasoning)
493d137 fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 305 in → 22 out (0 reasoning)
abd23cc fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["abd23cc fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
anthropic/claude-opus-4.6:high FAIL 0% 123 in → 10 out (0 reasoning)
Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['Fix typo in hello.py']
anthropic/claude-opus-4.6:medium FAIL 0% 123 in → 10 out (0 reasoning)
Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['Fix typo in hello.py']
anthropic/claude-opus-4.7:none FAIL 0% 167 in → 24 out (0 reasoning)
cf1d54b fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
anthropic/claude-sonnet-5:high__json_schema FAIL 0% 398 in → 31 out (0 reasoning)
2d796a2 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["2d796a2 fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
anthropic/claude-sonnet-5:low FAIL 0% 158 in → 21 out (0 reasoning)
61c1955 fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 401 in → 30 out (0 reasoning)
430b4bd fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["430b4bd fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 399 in → 30 out (0 reasoning)
b04af19 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["b04af19 fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 398 in → 30 out (0 reasoning)
c8a2dda fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["c8a2dda fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
anthropic/claude-sonnet-5:xhigh FAIL 0% 161 in → 22 out (0 reasoning)
e8148e1 fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 118 in → 1,216 out (1,243 reasoning)
Fixtypo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['Fixtypo in hello.py']
arcee-ai/trinity-mini:high__json_schema FAIL 0% 119 in → 408 out (374 reasoning)
c37f3e0 fixup: Fix typo in hello.py e70f487 Add hello world program a04c7b1 Initial commit
JSON Schema Structured Output
(raw) { "commits": [ "c37f3e0 fixup: Fix typo in hello.py", "e70f487 Add hello world program", "a04c7b1 Initial commit" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py', 'Add hello world program', 'Initial commit']
arcee-ai/trinity-mini:low FAIL 0% 115 in → 432 out (465 reasoning)
Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['Fix typo in hello.py']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 115 in → 370 out (354 reasoning)
70da7ab fixup: Fix typo in hello.py f661e18 Add hello world program dcef76a Initial commit
JSON Schema Structured Output
(raw) { "commits":[ "70da7ab fixup: Fix typo in hello.py", "f661e18 Add hello world program", "dcef76a Initial commit" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py', 'Add hello world program', 'Initial commit']
arcee-ai/trinity-mini:medium FAIL 0% 115 in → 359 out (362 reasoning)
Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['Fix typo in hello.py']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 114 in → 694 out (721 reasoning)
31bcdbc fixup: Fix typo in hello.py 19236aa Add hello world program 1ac0c52 Initial commit
JSON Schema Structured Output
(raw) { "commits": [ "31bcdbc fixup: Fix typo in hello.py", "19236aa Add hello world program", "1ac0c52 Initial commit" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py', 'Add hello world program', 'Initial commit']
arcee-ai/trinity-mini:xhigh FAIL 0% 115 in → 340 out (362 reasoning)
Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['Fix typo in hello.py']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 117 in → 413 out (389 reasoning)
d0e378b fixup: Fix typo in hello.py d0d1a37 Add hello world program 934259f Initial commit
JSON Schema Structured Output
(raw) { "commits": [ "d0e378b fixup: Fix typo in hello.py", "d0d1a37 Add hello world program", "934259f Initial commit" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py', 'Add hello world program', 'Initial commit']
deepseek/deepseek-v4-flash-0731:medium FAIL 0% 114 in → 333 out (309 reasoning)
6b549ba fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 117 in → 13 out (0 reasoning)
cc4ff64 fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 111 in → 24 out (0 reasoning)
6cbcd34 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "6cbcd34 fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 113 in → 253 out (228 reasoning)
a490892
JSON Schema Structured Output
(raw) { "commits": ["a490892"] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
deepseek/deepseek-v4-flash:high__json_schema FAIL 0% 115 in → 144 out (116 reasoning)
e219ce2 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "e219ce2 fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
deepseek/deepseek-v4-flash:medium__json_schema FAIL 0% 114 in → 248 out (224 reasoning)
36f8d42 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["36f8d42 fixup: Fix typo in hello.py"] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 110 in → 21 out (0 reasoning)
0471381 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": ["0471381 fixup: Fix typo in hello.py"] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
Invalid JSON. Output: fixup: Fix typo in hello.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: fixup: Fix typo in hello.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 114 in → 28 out (0 reasoning)
7eccbfa fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "7eccbfa fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
deepseek/deepseek-v4-pro:xhigh FAIL 0% 112 in → 194 out (197 reasoning)
fixup: Fix typo inhello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo inhello.py']
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 0% 115 in → 112 out (102 reasoning)
ea225ba
JSON Schema Structured Output
(raw) {"commits": ["ea225ba"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 115 in → 270 out (239 reasoning)
61cb95a fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "61cb95a fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
google/gemini-3.5-flash-lite:low FAIL 0% 117 in → 16 out (0 reasoning)
c789719 fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 195 in → 32 out (0 reasoning)
2220c91 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "2220c91 fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
google/gemini-3.5-flash-lite:medium__json_schema FAIL 0% 116 in → 552 out (532 reasoning)
0e2e5e4 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["0e2e5e4 fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
google/gemini-3.5-flash:medium__json_schema FAIL 0% 194 in → 624 out (592 reasoning)
62f96b7 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "62f96b7 fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
google/gemini-3.6-flash:high__json_schema FAIL 0% 117 in → 918 out (898 reasoning)
420a627 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits":["420a627 fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
google/gemini-3.6-flash:low__json_schema FAIL 0% 194 in → 32 out (0 reasoning)
6b32128 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "6b32128 fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
google/gemini-3.6-flash:medium__json_schema FAIL 0% 193 in → 544 out (514 reasoning)
dbc41b2 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "dbc41b2 fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
liquid/lfm-2-24b-a2b FAIL 0% 123 in → 14 out
2402c42 fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
Invalid JSON. Output: f4165a5 fixup: Fix typo in hello.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: {
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 208 column 1 (char 1017)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 208 column 1 (char 1017)
minimax/minimax-m2.5:medium FAIL 0% 143 in → 173 out (159 reasoning)
00aa82b fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
minimax/minimax-m2.5:medium__json_schema FAIL 0% 144 in → 205 out (189 reasoning)
7c3515f fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "7c3515f fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
minimax/minimax-m2.7:high__json_schema FAIL 0% 144 in → 47 out (47 reasoning)
ab8d613 fixup: Fix typo in hello.py fbda647 Add hello world program 24c5a3f Initial commit
JSON Schema Structured Output
(raw) { "commits": [ "ab8d613 fixup: Fix typo in hello.py", "fbda647 Add hello world program", "24c5a3f Initial commit" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py', 'Add hello world program', 'Initial commit']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high__json_schema FAIL 0% 268 in → 6 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"commits":[]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']
minimax/minimax-m3:low__json_schema FAIL 0% 232 in → 27 out (26 reasoning)
64e65e3 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "64e65e3 fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
minimax/minimax-m3:medium__json_schema FAIL 0% 272 in → 23 out (0 reasoning)
c1f6d3b fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["c1f6d3b fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
minimax/minimax-m3:xhigh__json_schema FAIL 0% 269 in → 25 out (0 reasoning)
283bf83 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits": [ "283bf83 fixup: Fix typo in hello.py" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
nvidia/nemotron-3-nano-30b-a3b:low FAIL 0% 129 in → 399 out (429 reasoning)
Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['Fix typo in hello.py']
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 127 in → 24 out (0 reasoning)
bcbe492 fixup: Fix typo in hello.py e31eaeb Add hello world program
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py', 'Add hello world program']
nvidia/nemotron-3-super-120b-a12b:none FAIL 0% 132 in → 16 out (0 reasoning)
32e47d0 fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
nvidia/nemotron-3-super-120b-a12b:none__json_schema FAIL 0% 129 in → 27 out (0 reasoning)
a5abc68
JSON Schema Structured Output
(raw) { "commits": [ "a5abc68" ] }
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
openai/gpt-5.4-nano:low FAIL 0% 112 in → 15 out (0 reasoning)
89da846 fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
openai/gpt-5.4-nano:none FAIL 0% 111 in → 17 out (0 reasoning)
f2ad5af fixup: Fix typo in hello.py
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
openai/gpt-oss-20b:xhigh__json_schema FAIL 0% 172 in → 940 out (954 reasoning)
Fix typo in hello.py
JSON Schema Structured Output
(raw) { "commits" : [ "Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['Fix typo in hello.py']
poolside/laguna-xs-2.1:medium__json_schema FAIL 0% 125 in → 299 out (279 reasoning)
38e22da fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["38e22da fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 122 in → 20 out (0 reasoning)
d44c7e2 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["d44c7e2 fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
Invalid JSON. Output: fixup: Fix typo in hello.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:none__json_schema FAIL 0% 128 in → 14 out (0 reasoning)
f814b49
JSON Schema Structured Output
(raw) {"commits": ["f814b49"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ef26d957-1ef4-9b51-b437-99cd9330869d","request_id":"ef26d957-1ef4-9b51-b437-99cd9330869d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fbf78349-c8d9-9d66-96f5-2a057d960bc6","request_id":"fbf78349-c8d9-9d66-96f5-2a057d960bc6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0a410119-9bf0-9976-9b99-50f2930886f3","request_id":"0a410119-9bf0-9976-9b99-50f2930886f3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c5f8d264-d6ba-915e-8c1f-47dad9097a0a","request_id":"c5f8d264-d6ba-915e-8c1f-47dad9097a0a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bab41f66-cae6-9d78-bde9-3a539784e031","request_id":"bab41f66-cae6-9d78-bde9-3a539784e031"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-16390130-3d05-9fc8-8d78-49f00dacc103"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f1499af8-9077-9aab-b9ed-2a11fa568a8e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f407bf02-67fd-9675-a105-7bc42ba8dd85"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b38255c5-fca6-921f-8d4c-8f10f834bbae"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b8c7577b-b07b-90be-92b3-2aea60518ed3"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0219fcee-3535-96e5-b2de-0e1565a1eaa7"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a82f4cb2-ba66-9e09-bea7-8f406ddeda8e","request_id":"a82f4cb2-ba66-9e09-bea7-8f406ddeda8e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-887820f5-f0c7-9bd7-a71a-147e1010aab1","request_id":"887820f5-f0c7-9bd7-a71a-147e1010aab1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0885da68-441c-9a8f-9bff-3465ff1146cf","request_id":"0885da68-441c-9a8f-9bff-3465ff1146cf"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cf19e9da-a499-9ff9-968d-062dbc707324","request_id":"cf19e9da-a499-9ff9-968d-062dbc707324"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ab7c16ec-11d4-9072-98ea-cfc4ceaabd6e","request_id":"ab7c16ec-11d4-9072-98ea-cfc4ceaabd6e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6e451d1e-9c70-9192-bdd3-ef38c6d9e5e8","request_id":"6e451d1e-9c70-9192-bdd3-ef38c6d9e5e8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c3e5c2ce-04f5-94f9-9631-3d101e14ec82","request_id":"c3e5c2ce-04f5-94f9-9631-3d101e14ec82"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0718c93e-8f27-9720-90b6-7ce445b4f8e6","request_id":"0718c93e-8f27-9720-90b6-7ce445b4f8e6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c6bf8b4e-2e45-943a-828b-1358d8679076","request_id":"c6bf8b4e-2e45-943a-828b-1358d8679076"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-33c7bfb1-e9d9-9e55-9880-93d73585ee33","request_id":"33c7bfb1-e9d9-9e55-9880-93d73585ee33"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 121 in → 11 out (0 reasoning)
fixup: Fix typo in hello.py<|end_message|>
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
Invalid JSON. Output: fixup: Fix typo in hello.py{ "commits": [ "fixup: Fix typo in hello.py" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: fixup: Fix typo in hello.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 121 in → 11 out (0 reasoning)
fixup: Fix typo in hello.py<|end_message|>
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
thinkingmachines/inkling:none__json_schema FAIL 0% 118 in → 21 out (0 reasoning)
46a8313 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["46a8313 fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
Invalid JSON. Output: fixup: Fix typo in hello.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none__json_schema FAIL 0% 354 in → 16 out (0 reasoning)
8297635 fixup: Fix typo in hello.py
JSON Schema Structured Output
(raw) {"commits": ["8297635 fixup: Fix typo in hello.py"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:xhigh__json_schema FAIL 0% 120 in → 37 out (35 reasoning)
ed02939 fixup: Fix typo in hello.py 7845670 Add hello world program c343b47 Initial commit
JSON Schema Structured Output
(raw) {"commits":["ed02939 fixup: Fix typo in hello.py","7845670 Add hello world program","c343b47 Initial commit"]}
Failure: Missing expected commit messages: ['fixup: Fix typo in hello.py']; Extra selected commit messages: ['fixup: Fix typo in hello.py', 'Add hello world program', 'Initial commit']