WIP commits - two consecutive WIP commits should be squashed
Tests ability to identify that two consecutive WIP commits should be squashed into one meaningful commit. Evaluates recognizing work-in-progress patterns.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo '# Project' > README.md
  5. 05 git add README.md
  6. 06 git commit -m 'Initial commit'
  7. 07 echo 'print(1)' > main.py
  8. 08 git add main.py
  9. 09 git commit -m 'WIP: add main.py'
  10. 10 echo 'print(2)' >> main.py
  11. 11 git add main.py
  12. 12 git commit -m 'WIP: continue work'
  13. 13 echo 'print(3)' >> main.py
  14. 14 git add main.py
  15. 15 git commit -m 'Complete feature'
Prompt
Analyze this git history and identify which consecutive commits should be marked as squash/fixup in an interactive rebase to create a cleaner history. Focus on WIP (Work In Progress) commits that represent incomplete work. Do not include the base/anchor commit they would be squashed into. Output ONLY the selected commit subject lines, one per line, with no explanation and no base/anchor/final commits.
Expected
WIP: add main.py WIP: continue work

Scoped model quality, cost, API time, and token usage for commit_squash/f001.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5.1:high PASS 100% 198 in → 18 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-fable-5.1:high__json_schema PASS 100% 435 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-fable-5.1:low PASS 100% 194 in → 18 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-fable-5.1:low__json_schema PASS 100% 436 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-fable-5.1:max PASS 100% 199 in → 1,436 out (1,418 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-fable-5.1:max__json_schema PASS 100% 440 in → 1,124 out (1,097 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-fable-5.1:medium PASS 100% 200 in → 18 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-fable-5.1:medium__json_schema PASS 100% 439 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-fable-5.1:xhigh PASS 100% 194 in → 579 out (561 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-fable-5.1:xhigh__json_schema PASS 100% 437 in → 311 out (282 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
anthropic/claude-fable-5:high PASS 100% 195 in → 521 out (75 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-fable-5:high__json_schema PASS 100% 435 in → 558 out (78 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
anthropic/claude-fable-5:low PASS 100% 195 in → 37 out (12 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-fable-5:low__json_schema PASS 100% 434 in → 29 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
anthropic/claude-fable-5:max PASS 100% 196 in → 1,264 out (225 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-fable-5:max__json_schema PASS 100% 439 in → 1,680 out (278 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
anthropic/claude-fable-5:medium PASS 100% 194 in → 105 out (87 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-fable-5:medium__json_schema PASS 100% 439 in → 497 out (70 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
anthropic/claude-fable-5:xhigh FAIL 100% 196 in → 755 out (154 reasoning)
``` WIP: continue work WIP: add main.py ```
Failure: Extra selected commit messages: ['```', '```']
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 436 in → 735 out (102 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
anthropic/claude-haiku-4.5:high PASS 100% 171 in → 411 out (359 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 357 in → 930 out (599 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
anthropic/claude-haiku-4.5:low PASS 100% 170 in → 724 out (490 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 360 in → 1,420 out (665 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits": ["WIP: add main.py", "WIP: continue work"]}
anthropic/claude-haiku-4.5:medium PASS 100% 168 in → 741 out (474 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 357 in → 704 out (632 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits": ["WIP: add main.py", "WIP: continue work"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 169 in → 428 out (392 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 358 in → 4,949 out (1,625 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits": ["WIP: add main.py", "WIP: continue work"]}
anthropic/claude-opus-4.6:high PASS 100% 146 in → 105 out (82 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-opus-4.6:high__json_schema PASS 100% 329 in → 132 out (88 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.6:low PASS 100% 144 in → 16 out (0 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-opus-4.6:low__json_schema PASS 100% 324 in → 22 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.6:max PASS 100% 137 in → 228 out (167 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-opus-4.6:max__json_schema PASS 100% 329 in → 219 out (147 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.6:medium PASS 100% 142 in → 16 out (0 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 329 in → 22 out (0 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
anthropic/claude-opus-4.6:none PASS 100% 145 in → 16 out (0 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-opus-4.6:none__json_schema PASS 100% 326 in → 22 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 138 in → 123 out (82 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 326 in → 129 out (84 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
anthropic/claude-opus-4.7:high PASS 100% 200 in → 20 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-4.7:high__json_schema PASS 100% 437 in → 29 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.7:low PASS 100% 200 in → 20 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-4.7:low__json_schema PASS 100% 442 in → 29 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.7:max PASS 100% 199 in → 1,593 out (185 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-4.7:max__json_schema PASS 100% 440 in → 29 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.7:medium PASS 100% 200 in → 20 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 439 in → 29 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.7:none PASS 100% 197 in → 20 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-4.7:none__json_schema PASS 100% 444 in → 29 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 201 in → 20 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 440 in → 29 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 431 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.8:low PASS 100% 194 in → 18 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-4.8:low__json_schema PASS 100% 437 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.8:max PASS 100% 199 in → 2,675 out (398 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-4.8:max__json_schema PASS 100% 438 in → 3,046 out (266 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 436 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.8:none PASS 100% 193 in → 18 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-4.8:none__json_schema PASS 100% 436 in → 26 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 439 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-5.5:high PASS 100% 196 in → 279 out (261 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-5.5:high__json_schema PASS 100% 439 in → 293 out (266 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
anthropic/claude-opus-5.5:low PASS 100% 193 in → 179 out (161 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-5.5:max PASS 100% 197 in → 1,182 out (1,164 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-5.5:max__json_schema PASS 100% 437 in → 2,451 out (2,424 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-5.5:medium PASS 100% 196 in → 276 out (258 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-opus-5.5:medium__json_schema PASS 100% 441 in → 264 out (237 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-5.5:xhigh PASS 100% 199 in → 402 out (384 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-5.5:xhigh__json_schema PASS 100% 438 in → 469 out (442 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
anthropic/claude-opus-5:high PASS 100% 194 in → 820 out (85 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-5:high__json_schema PASS 100% 437 in → 665 out (105 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-5:low__json_schema PASS 100% 434 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-5:max PASS 100% 195 in → 724 out (93 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-opus-5:max__json_schema PASS 100% 434 in → 517 out (110 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits": ["WIP: add main.py", "WIP: continue work"]}
anthropic/claude-opus-5:medium PASS 100% 196 in → 388 out (108 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-opus-5:medium__json_schema PASS 100% 438 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-sonnet-4.6:high PASS 100% 139 in → 238 out (179 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 328 in → 22 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-sonnet-4.6:low PASS 100% 141 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-sonnet-4.6:max PASS 100% 141 in → 492 out (262 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 323 in → 261 out (175 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 140 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 328 in → 22 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-sonnet-4.6:none PASS 100% 143 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-sonnet-4.6:xhigh FAIL 100% 140 in → 357 out (172 reasoning)
``` WIP: add main.py WIP: continue work ```
Failure: Extra selected commit messages: ['```', '```']
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 328 in → 22 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-sonnet-5:high PASS 100% 194 in → 71 out (41 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-sonnet-5:high__json_schema PASS 100% 439 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-sonnet-5:low FAIL 100% 196 in → 22 out (0 reasoning)
``` WIP: continue work WIP: add main.py ```
Failure: Extra selected commit messages: ['```', '```']
anthropic/claude-sonnet-5:low__json_schema PASS 100% 436 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-sonnet-5:medium PASS 100% 195 in → 109 out (58 reasoning)
WIP: continue work WIP: add main.py
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 435 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-sonnet-5:none__json_schema PASS 100% 438 in → 26 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 195 in → 154 out (84 reasoning)
WIP: add main.py WIP: continue work
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 437 in → 480 out (102 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
arcee-ai/trinity-large-thinking:high PASS 100% 132 in → 2,074 out (2,165 reasoning)
WIP: continue work WIP: add main.py
arcee-ai/trinity-large-thinking:low PASS 100% 128 in → 1,807 out (1,887 reasoning)
WIP: add main.py WIP: continue work
arcee-ai/trinity-large-thinking:medium PASS 100% 134 in → 4,188 out (4,355 reasoning)
WIP: continue work WIP: add main.py
arcee-ai/trinity-large-thinking:xhigh PASS 100% 131 in → 3,416 out (3,581 reasoning)
WIP: continue work WIP: add main.py
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 134 in → 1,096 out (1,207 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
deepseek/deepseek-v4-flash-0731:high PASS 100% 213 in → 2,663 out (2,650 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 210 in → 3,816 out (3,809 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 131 in → 592 out (579 reasoning)
WIP: add main.py WIP: continue work
deepseek/deepseek-v4-flash-0731:max PASS 100% 221 in → 6,194 out (6,362 reasoning)
WIP: add main.py WIP: continue work
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 303 in → 389 out (370 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 134 in → 437 out (428 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 130 in → 532 out (506 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 211 in → 3,318 out (3,341 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 132 in → 532 out (458 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
deepseek/deepseek-v4-flash:high PASS 100% 129 in → 546 out (557 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 245 in → 21 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
deepseek/deepseek-v4-flash:low PASS 100% 134 in → 322 out (308 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 242 in → 21 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
deepseek/deepseek-v4-flash:medium PASS 100% 132 in → 665 out (651 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4-flash:none PASS 100% 132 in → 13 out (0 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 132 in → 21 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
deepseek/deepseek-v4-flash:xhigh PASS 100% 210 in → 830 out (816 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 213 in → 571 out (598 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
deepseek/deepseek-v4-pro:high PASS 100% 135 in → 537 out (523 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4-pro:low PASS 100% 129 in → 851 out (867 reasoning)
WIP: add main.py WIP: continue work
deepseek/deepseek-v4-pro:medium PASS 100% 128 in → 780 out (766 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 333 in → 372 out (350 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
deepseek/deepseek-v4.1-flash:high PASS 100% 162 in → 1,628 out (1,684 reasoning)
WIP: add main.py WIP: continue work
deepseek/deepseek-v4.1-flash:high__json_schema PASS 100% 158 in → 2,986 out (2,958 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
deepseek/deepseek-v4.1-flash:low PASS 100% 157 in → 1,898 out (1,884 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4.1-flash:low__json_schema PASS 100% 156 in → 3,937 out (3,909 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
deepseek/deepseek-v4.1-flash:max PASS 100% 155 in → 5,605 out (5,591 reasoning)
WIP: add main.py WIP: continue work
deepseek/deepseek-v4.1-flash:max__json_schema PASS 100% 159 in → 4,585 out (4,559 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
deepseek/deepseek-v4.1-flash:medium PASS 100% 161 in → 1,885 out (1,871 reasoning)
WIP: add main.py WIP: continue work
deepseek/deepseek-v4.1-flash:medium__json_schema PASS 100% 158 in → 1,625 out (1,594 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
deepseek/deepseek-v4.1-flash:xhigh PASS 100% 160 in → 1,522 out (1,508 reasoning)
WIP: continue work WIP: add main.py
deepseek/deepseek-v4.1-flash:xhigh__json_schema PASS 100% 253 in → 4,279 out (4,259 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
google/gemini-3-flash-preview:high PASS 100% 132 in → 1,636 out (1,623 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3-flash-preview:high__json_schema PASS 100% 212 in → 1,816 out (1,799 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3-flash-preview:low PASS 100% 133 in → 707 out (694 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3-flash-preview:low__json_schema PASS 100% 213 in → 555 out (538 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3-flash-preview:medium PASS 100% 132 in → 1,361 out (1,348 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3-flash-preview:medium__json_schema PASS 100% 212 in → 1,281 out (1,264 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3-flash-preview:xhigh PASS 100% 133 in → 1,024 out (1,011 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 211 in → 773 out (756 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 136 in → 1,165 out (1,152 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 135 in → 1,074 out (1,042 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemini-3.1-flash-lite-preview:low PASS 100% 135 in → 136 out (123 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 131 in → 124 out (105 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 132 in → 255 out (242 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 134 in → 438 out (406 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 133 in → 465 out (452 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 135 in → 891 out (874 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3.1-pro-preview:high PASS 100% 134 in → 851 out (838 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 211 in → 544 out (527 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3.1-pro-preview:low PASS 100% 131 in → 268 out (255 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 214 in → 120 out (88 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 215 in → 487 out (470 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 134 in → 6,689 out (6,676 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 212 in → 460 out (443 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3.5-flash-lite:high PASS 100% 135 in → 2,498 out (2,485 reasoning)
WIP: add main.py WIP: continue work
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 133 in → 1,404 out (1,387 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 136 in → 32 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemini-3.5-flash-lite:max PASS 100% 133 in → 1,424 out (1,411 reasoning)
WIP: add main.py WIP: continue work
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 135 in → 784 out (767 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3.5-flash-lite:medium PASS 100% 135 in → 885 out (872 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 213 in → 813 out (794 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits": ["WIP: add main.py", "WIP: continue work"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 135 in → 720 out (707 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 136 in → 1,664 out (1,647 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
google/gemini-3.5-flash:high PASS 100% 135 in → 1,710 out (1,697 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.5-flash:high__json_schema PASS 100% 213 in → 1,863 out (1,844 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
google/gemini-3.5-flash:low PASS 100% 132 in → 745 out (732 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.5-flash:low__json_schema PASS 100% 212 in → 366 out (334 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemini-3.5-flash:medium PASS 100% 133 in → 2,253 out (2,240 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.5-flash:medium__json_schema PASS 100% 214 in → 1,140 out (1,108 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemini-3.5-flash:xhigh PASS 100% 133 in → 1,971 out (1,958 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 212 in → 1,312 out (1,295 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3.6-flash:high PASS 100% 134 in → 1,306 out (1,293 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.6-flash:high__json_schema PASS 100% 134 in → 1,679 out (1,662 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3.6-flash:low PASS 100% 135 in → 695 out (682 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.6-flash:max__json_schema PASS 100% 210 in → 1,562 out (1,530 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemini-3.8-flash:high PASS 100% 135 in → 24,405 out (24,392 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.8-flash:high__json_schema PASS 100% 213 in → 18,914 out (18,897 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
google/gemini-3.8-flash:low PASS 100% 132 in → 13 out (0 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.8-flash:low__json_schema PASS 100% 212 in → 19 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
google/gemini-3.8-flash:max__json_schema PASS 100% 209 in → 1,013 out (994 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
google/gemini-3.8-flash:medium PASS 100% 134 in → 3,418 out (3,405 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.8-flash:medium__json_schema PASS 100% 215 in → 4,478 out (4,446 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemini-3.8-flash:xhigh PASS 100% 135 in → 5,390 out (5,377 reasoning)
WIP: continue work WIP: add main.py
google/gemini-3.8-flash:xhigh__json_schema PASS 100% 211 in → 51,570 out (51,538 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemma-4-26b-a4b-it:high PASS 100% 149 in → 2,871 out (2,857 reasoning)
WIP: continue work WIP: add main.py
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 149 in → 1,056 out (794 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 149 in → 2,389 out (1,900 reasoning)
WIP: continue work WIP: add main.py
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 150 in → 1,622 out (1,155 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 152 in → 4,430 out (3,719 reasoning)
WIP: continue work WIP: add main.py
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 146 in → 1,508 out (1,221 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 147 in → 14 out (0 reasoning)
WIP: continue work WIP: add main.py
google/gemma-4-26b-a4b-it:xhigh PASS 100% 151 in → 2,508 out (1,927 reasoning)
WIP: continue work WIP: add main.py
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 150 in → 2,147 out (1,732 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemma-4-31b-it:high PASS 100% 146 in → 739 out (652 reasoning)
WIP: continue work WIP: add main.py
google/gemma-4-31b-it:high__json_schema PASS 100% 150 in → 910 out (822 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
google/gemma-4-31b-it:low PASS 100% 150 in → 831 out (630 reasoning)
WIP: add main.py WIP: continue work
google/gemma-4-31b-it:low__json_schema PASS 100% 153 in → 1,240 out (1,221 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
google/gemma-4-31b-it:medium PASS 100% 150 in → 1,098 out (914 reasoning)
WIP: continue work WIP: add main.py
google/gemma-4-31b-it:medium__json_schema PASS 100% 147 in → 741 out (624 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
google/gemma-4-31b-it:none PASS 100% 145 in → 14 out (0 reasoning)
WIP: continue work WIP: add main.py
google/gemma-4-31b-it:xhigh PASS 100% 150 in → 717 out (616 reasoning)
WIP: continue work WIP: add main.py
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 150 in → 962 out (812 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
ibm-granite/granite-4.1-8b PASS 100% 138 in → 13 out
WIP: continue work WIP: add main.py
ibm-granite/granite-4.2-8b:high PASS 100% 146 in → 1,455 out (1,450 reasoning)
WIP: add main.py WIP: continue work
ibm-granite/granite-4.2-8b:low PASS 100% 156 in → 366 out (335 reasoning)
WIP: continue work WIP: add main.py
ibm-granite/granite-4.2-8b:max PASS 100% 145 in → 4,038 out (4,134 reasoning)
WIP: add main.py WIP: continue work
ibm-granite/granite-4.2-8b:medium PASS 100% 153 in → 621 out (539 reasoning)
WIP: continue work WIP: add main.py
ibm-granite/granite-4.2-8b:medium__json_schema PASS 100% 155 in → 25 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
ibm-granite/granite-4.2-8b:none PASS 100% 143 in → 14 out (0 reasoning)
WIP: continue work WIP: add main.py
ibm-granite/granite-4.2-8b:xhigh PASS 100% 143 in → 6,735 out (6,780 reasoning)
WIP: add main.py WIP: continue work
inception/mercury-2.5:high PASS 100% 130 in → 1,421 out (1,406 reasoning)
WIP: continue work WIP: add main.py
inception/mercury-2.5:max PASS 100% 130 in → 1,353 out (1,338 reasoning)
WIP: continue work WIP: add main.py
inception/mercury-2.5:max__json_schema PASS 100% 351 in → 1,834 out (1,740 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
inception/mercury-2.5:medium PASS 100% 132 in → 1,682 out (1,663 reasoning)
WIP: continue work WIP: add main.py
inception/mercury-2.5:medium__json_schema PASS 100% 364 in → 1,721 out (1,631 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
inception/mercury-2.5:none__json_schema PASS 100% 354 in → 202 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
inception/mercury-2.5:xhigh PASS 100% 132 in → 1,259 out (1,242 reasoning)
WIP: continue work WIP: add main.py
inception/mercury-2.5:xhigh__json_schema PASS 100% 360 in → 1,514 out (1,448 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
inclusionai/ling-2.6-flash__json_schema PASS 100% 156 in → 34 out
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
meta/muse-spark-1.1:high PASS 100% 136 in → 1,803 out (1,781 reasoning)
WIP: continue work WIP: add main.py
meta/muse-spark-1.1:high__json_schema PASS 100% 131 in → 1,026 out (995 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
meta/muse-spark-1.1:low PASS 100% 135 in → 508 out (486 reasoning)
WIP: continue work WIP: add main.py
meta/muse-spark-1.1:low__json_schema PASS 100% 135 in → 411 out (374 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
meta/muse-spark-1.1:max PASS 100% 132 in → 1,288 out (1,266 reasoning)
WIP: continue work WIP: add main.py
meta/muse-spark-1.1:max__json_schema PASS 100% 131 in → 1,434 out (1,403 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
meta/muse-spark-1.1:medium PASS 100% 132 in → 914 out (892 reasoning)
WIP: continue work WIP: add main.py
meta/muse-spark-1.1:medium__json_schema PASS 100% 133 in → 713 out (684 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
meta/muse-spark-1.1:xhigh PASS 100% 134 in → 1,086 out (1,064 reasoning)
WIP: continue work WIP: add main.py
meta/muse-spark-1.3-contributor:high PASS 100% 134 in → 1,069 out (1,006 reasoning)
WIP: continue work WIP: add main.py
meta/muse-spark-1.3-contributor:high__json_schema PASS 100% 132 in → 1,571 out (1,478 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
meta/muse-spark-1.3-contributor:low PASS 100% 134 in → 565 out (543 reasoning)
WIP: continue work WIP: add main.py
meta/muse-spark-1.3-contributor:low__json_schema PASS 100% 136 in → 566 out (519 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
meta/muse-spark-1.3-contributor:medium PASS 100% 135 in → 1,186 out (1,122 reasoning)
WIP: continue work WIP: add main.py
meta/muse-spark-1.3-contributor:medium__json_schema PASS 100% 132 in → 1,094 out (1,029 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
meta/muse-spark-1.3-contributor:xhigh PASS 100% 134 in → 2,736 out (2,649 reasoning)
WIP: continue work WIP: add main.py
meta/muse-spark-1.3-contributor:xhigh__json_schema PASS 100% 135 in → 1,214 out (1,147 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
minimax/minimax-m2.5:high PASS 100% 168 in → 261 out (248 reasoning)
WIP: continue work WIP: add main.py
minimax/minimax-m2.5:high__json_schema PASS 100% 165 in → 224 out (191 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
minimax/minimax-m2.5:low PASS 100% 165 in → 201 out (174 reasoning)
WIP: continue work WIP: add main.py
minimax/minimax-m2.5:low__json_schema PASS 100% 171 in → 427 out (398 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
minimax/minimax-m2.5:medium__json_schema PASS 100% 163 in → 325 out (303 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
minimax/minimax-m2.5:xhigh PASS 100% 164 in → 440 out (432 reasoning)
WIP: continue work WIP: add main.py
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 168 in → 330 out (303 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
minimax/minimax-m2.7:high PASS 100% 162 in → 1,087 out (1,072 reasoning)
WIP: continue work WIP: add main.py
minimax/minimax-m2.7:high__json_schema PASS 100% 318 in → 795 out (766 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
minimax/minimax-m2.7:low PASS 100% 163 in → 586 out (572 reasoning)
WIP: continue work WIP: add main.py
minimax/minimax-m2.7:low__json_schema PASS 100% 316 in → 460 out (432 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
minimax/minimax-m2.7:medium PASS 100% 163 in → 413 out (400 reasoning)
WIP: continue work WIP: add main.py
minimax/minimax-m2.7:xhigh PASS 100% 164 in → 655 out (640 reasoning)
WIP: continue work WIP: add main.py
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 316 in → 2,573 out (2,544 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
minimax/minimax-m3:high PASS 100% 309 in → 258 out (234 reasoning)
WIP: continue work WIP: add main.py
minimax/minimax-m3:high__json_schema PASS 100% 287 in → 21 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
minimax/minimax-m3:low PASS 100% 299 in → 321 out (317 reasoning)
WIP: continue work WIP: add main.py
minimax/minimax-m3:low__json_schema PASS 100% 288 in → 28 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
minimax/minimax-m3:medium PASS 100% 301 in → 287 out (263 reasoning)
WIP: continue work WIP: add main.py
minimax/minimax-m3:xhigh PASS 100% 301 in → 288 out (283 reasoning)
WIP: continue work WIP: add main.py
mistralai/devstral-2512__json_schema PASS 100% 141 in → 21 out
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
mistralai/mistral-medium-3-5:high PASS 100% 148 in → 346 out (336 reasoning)
WIP: continue work WIP: add main.py
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 137 in → 706 out (656 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
mistralai/mistral-medium-3-5:low PASS 100% 150 in → 1,626 out (1,660 reasoning)
WIP: continue work WIP: add main.py
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 141 in → 462 out (408 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
mistralai/mistral-medium-3-5:medium PASS 100% 153 in → 2,515 out (2,343 reasoning)
WIP: continue work WIP: add main.py
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 140 in → 427 out (372 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits": ["WIP: add main.py", "WIP: continue work"]}
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 139 in → 21 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 150 in → 596 out (548 reasoning)
WIP: continue work WIP: add main.py
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 140 in → 442 out (431 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
moonshotai/kimi-k2.6:high PASS 100% 134 in → 7,717 out (7,703 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k2.6:high__json_schema PASS 100% 131 in → 2,130 out (2,039 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
moonshotai/kimi-k2.6:low PASS 100% 133 in → 1,109 out (1,179 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k2.6:low__json_schema PASS 100% 131 in → 3,212 out (3,191 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
moonshotai/kimi-k2.6:medium PASS 100% 132 in → 3,931 out (3,919 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 133 in → 4,416 out (4,046 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
moonshotai/kimi-k2.6:xhigh PASS 100% 128 in → 4,595 out (4,581 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 133 in → 5,330 out (4,787 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
moonshotai/kimi-k2.7-code:high PASS 100% 134 in → 375 out (361 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 227 in → 719 out (698 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
moonshotai/kimi-k2.7-code:low PASS 100% 130 in → 2,036 out (2,022 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 132 in → 463 out (445 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
moonshotai/kimi-k2.7-code:medium PASS 100% 133 in → 316 out (302 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 133 in → 1,299 out (1,419 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 134 in → 1,582 out (1,568 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 135 in → 439 out (405 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
moonshotai/kimi-k3:high PASS 100% 210 in → 1,297 out (1,270 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k3:high__json_schema PASS 100% 315 in → 579 out (547 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
moonshotai/kimi-k3:low PASS 100% 209 in → 1,164 out (1,137 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k3:low__json_schema PASS 100% 208 in → 244 out (217 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k3:max PASS 100% 212 in → 1,070 out (1,043 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k3:max__json_schema PASS 100% 211 in → 937 out (910 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k3:medium PASS 100% 206 in → 433 out (406 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k3:medium__json_schema PASS 100% 210 in → 2,102 out (2,075 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k3:xhigh PASS 100% 207 in → 250 out (223 reasoning)
WIP: continue work WIP: add main.py
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 210 in → 1,358 out (1,331 reasoning)
WIP: continue work WIP: add main.py
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 150 in → 817 out (855 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 154 in → 528 out (513 reasoning)
WIP: continue work WIP: add main.py
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 154 in → 357 out (331 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 154 in → 490 out (467 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 151 in → 13 out (0 reasoning)
WIP: continue work WIP: add main.py
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 154 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 154 in → 612 out (578 reasoning)
WIP: continue work WIP: add main.py
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 151 in → 909 out (922 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 151 in → 366 out (343 reasoning)
WIP: continue work WIP: add main.py
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 154 in → 822 out (775 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 153 in → 264 out (235 reasoning)
WIP: continue work WIP: add main.py
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 154 in → 294 out (251 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 154 in → 499 out (452 reasoning)
WIP: continue work WIP: add main.py
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 153 in → 343 out (297 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 153 in → 13 out (0 reasoning)
WIP: add main.py WIP: continue work
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 151 in → 21 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 151 in → 313 out (289 reasoning)
WIP: continue work WIP: add main.py
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 153 in → 309 out (265 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
openai/gpt-5.4-mini:high PASS 100% 131 in → 335 out (317 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4-mini:high__json_schema PASS 100% 182 in → 310 out (282 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.4-mini:low PASS 100% 131 in → 88 out (70 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4-mini:low__json_schema PASS 100% 180 in → 104 out (76 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.4-mini:medium PASS 100% 133 in → 167 out (149 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4-mini:medium__json_schema PASS 100% 179 in → 308 out (280 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.4-mini:none PASS 100% 132 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4-mini:xhigh PASS 100% 134 in → 15,556 out (15,538 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 180 in → 11,940 out (11,912 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.4-nano:high PASS 100% 132 in → 165 out (147 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4-nano:high__json_schema PASS 100% 181 in → 158 out (130 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.4-nano:low PASS 100% 134 in → 79 out (61 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4-nano:low__json_schema PASS 100% 180 in → 62 out (34 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.4-nano:medium PASS 100% 133 in → 152 out (134 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4-nano:medium__json_schema PASS 100% 179 in → 76 out (48 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.4-nano:none__json_schema PASS 100% 182 in → 26 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.4-nano:xhigh PASS 100% 129 in → 225 out (207 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 180 in → 1,061 out (1,033 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
openai/gpt-5.4:high PASS 100% 134 in → 534 out (516 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4:high__json_schema PASS 100% 180 in → 3,180 out (3,152 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
openai/gpt-5.4:low PASS 100% 133 in → 96 out (78 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4:low__json_schema PASS 100% 179 in → 64 out (36 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.4:medium PASS 100% 132 in → 346 out (328 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4:medium__json_schema PASS 100% 175 in → 198 out (170 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.4:none PASS 100% 131 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.4:xhigh PASS 100% 134 in → 1,728 out (1,710 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.5:high PASS 100% 133 in → 534 out (516 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.5:high__json_schema PASS 100% 182 in → 544 out (516 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.5:low PASS 100% 129 in → 169 out (151 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.5:low__json_schema PASS 100% 181 in → 254 out (226 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.5:medium PASS 100% 132 in → 306 out (288 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.5:medium__json_schema PASS 100% 180 in → 544 out (516 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
openai/gpt-5.5:none PASS 100% 131 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.5:none__json_schema PASS 100% 177 in → 26 out (0 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
openai/gpt-5.5:xhigh PASS 100% 130 in → 534 out (516 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.5:xhigh__json_schema PASS 100% 181 in → 544 out (516 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-luna:high PASS 100% 134 in → 161 out (143 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-luna:high__json_schema PASS 100% 179 in → 181 out (153 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-luna:low PASS 100% 133 in → 174 out (156 reasoning)
WIP: add main.py WIP: continue work
openai/gpt-5.6-luna:low__json_schema PASS 100% 178 in → 121 out (93 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-luna:max PASS 100% 136 in → 240 out (222 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-luna:max__json_schema PASS 100% 178 in → 544 out (516 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-luna:medium PASS 100% 134 in → 128 out (110 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-luna:medium__json_schema PASS 100% 179 in → 120 out (92 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-luna:none PASS 100% 129 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-luna:none__json_schema PASS 100% 180 in → 26 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-luna:xhigh PASS 100% 131 in → 431 out (413 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 178 in → 263 out (235 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-sol:high PASS 100% 132 in → 16 out (0 reasoning)
WIP: add main.py WIP: continue work
openai/gpt-5.6-sol:high__json_schema PASS 100% 179 in → 26 out (0 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
openai/gpt-5.6-sol:low PASS 100% 132 in → 16 out (0 reasoning)
WIP: add main.py WIP: continue work
openai/gpt-5.6-sol:low__json_schema PASS 100% 180 in → 26 out (0 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
openai/gpt-5.6-sol:max PASS 100% 133 in → 194 out (176 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-sol:max__json_schema PASS 100% 182 in → 336 out (308 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-sol:medium PASS 100% 130 in → 16 out (0 reasoning)
WIP: add main.py WIP: continue work
openai/gpt-5.6-sol:medium__json_schema PASS 100% 178 in → 75 out (47 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
openai/gpt-5.6-sol:none PASS 100% 135 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-sol:none__json_schema PASS 100% 179 in → 26 out (0 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
openai/gpt-5.6-sol:xhigh PASS 100% 132 in → 16 out (0 reasoning)
WIP: add main.py WIP: continue work
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 181 in → 26 out (0 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
openai/gpt-5.6-terra:high PASS 100% 129 in → 167 out (149 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-terra:high__json_schema PASS 100% 180 in → 172 out (144 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-terra:low PASS 100% 134 in → 124 out (106 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-terra:low__json_schema PASS 100% 179 in → 91 out (63 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-terra:max PASS 100% 133 in → 383 out (365 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-terra:max__json_schema PASS 100% 179 in → 334 out (306 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-terra:medium PASS 100% 131 in → 75 out (57 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-terra:medium__json_schema PASS 100% 182 in → 75 out (47 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-terra:none PASS 100% 131 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-terra:none__json_schema PASS 100% 179 in → 26 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-5.6-terra:xhigh PASS 100% 130 in → 141 out (123 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 179 in → 117 out (89 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-astra:high PASS 100% 131 in → 60 out (42 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-astra:high__json_schema PASS 100% 179 in → 321 out (293 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-astra:low PASS 100% 131 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-astra:low__json_schema PASS 100% 179 in → 48 out (20 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-astra:max PASS 100% 134 in → 352 out (334 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-astra:max__json_schema PASS 100% 177 in → 411 out (383 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-astra:medium PASS 100% 132 in → 16 out (0 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-astra:medium__json_schema PASS 100% 176 in → 54 out (26 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-astra:xhigh PASS 100% 133 in → 100 out (82 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-astra:xhigh__json_schema PASS 100% 181 in → 167 out (139 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-luna:high PASS 100% 132 in → 136 out (118 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-luna:high__json_schema PASS 100% 180 in → 83 out (55 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-luna:low PASS 100% 130 in → 83 out (65 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-luna:low__json_schema PASS 100% 178 in → 77 out (49 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-luna:max PASS 100% 133 in → 154 out (136 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-luna:max__json_schema PASS 100% 181 in → 426 out (398 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-luna:medium PASS 100% 133 in → 150 out (132 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-luna:medium__json_schema PASS 100% 177 in → 112 out (84 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-luna:xhigh PASS 100% 135 in → 214 out (196 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-luna:xhigh__json_schema PASS 100% 184 in → 140 out (112 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-sol:high PASS 100% 132 in → 71 out (53 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-sol:high__json_schema PASS 100% 176 in → 273 out (245 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-sol:low PASS 100% 129 in → 35 out (17 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-sol:low__json_schema PASS 100% 179 in → 26 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-sol:max PASS 100% 134 in → 225 out (207 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-sol:max__json_schema PASS 100% 183 in → 895 out (867 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-sol:medium PASS 100% 133 in → 46 out (28 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-sol:medium__json_schema PASS 100% 182 in → 118 out (90 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-6-sol:xhigh PASS 100% 130 in → 179 out (161 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-6-sol:xhigh__json_schema PASS 100% 181 in → 99 out (71 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-oss-120b:high PASS 100% 192 in → 418 out (398 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-oss-120b:high__json_schema PASS 100% 196 in → 27 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
openai/gpt-oss-120b:low PASS 100% 191 in → 142 out (126 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-oss-120b:low__json_schema PASS 100% 176 in → 99 out (60 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
openai/gpt-oss-120b:medium PASS 100% 192 in → 344 out (355 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-oss-120b:medium__json_schema PASS 100% 189 in → 315 out (282 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
openai/gpt-oss-120b:xhigh PASS 100% 176 in → 467 out (503 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 194 in → 899 out (980 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
openai/gpt-oss-20b:high PASS 100% 191 in → 1,518 out (1,631 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-oss-20b:high__json_schema PASS 100% 196 in → 1,294 out (1,360 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-oss-20b:low PASS 100% 194 in → 92 out (66 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-oss-20b:low__json_schema PASS 100% 195 in → 150 out (95 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-oss-20b:medium PASS 100% 188 in → 345 out (347 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-oss-20b:medium__json_schema PASS 100% 194 in → 620 out (623 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
openai/gpt-oss-20b:xhigh PASS 100% 195 in → 533 out (544 reasoning)
WIP: continue work WIP: add main.py
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 192 in → 516 out (490 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
poolside/laguna-m.1:high PASS 100% 145 in → 308 out (295 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-m.1:high__json_schema PASS 100% 143 in → 510 out (479 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
poolside/laguna-m.1:low PASS 100% 141 in → 252 out (239 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-m.1:low__json_schema PASS 100% 147 in → 636 out (613 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
poolside/laguna-m.1:medium PASS 100% 144 in → 392 out (379 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-m.1:medium__json_schema PASS 100% 145 in → 402 out (384 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
poolside/laguna-m.1:none PASS 100% 142 in → 12 out (0 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-m.1:xhigh PASS 100% 143 in → 572 out (559 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-m.1:xhigh__json_schema PASS 100% 143 in → 869 out (838 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
poolside/laguna-xs-2.1:high__json_schema PASS 100% 145 in → 818 out (799 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
poolside/laguna-xs-2.1:low PASS 100% 143 in → 395 out (384 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-xs-2.1:low__json_schema PASS 100% 144 in → 839 out (807 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
poolside/laguna-xs-2.1:medium PASS 100% 142 in → 970 out (959 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-xs-2.1:none PASS 100% 145 in → 10 out (0 reasoning)
WIP: add main.py WIP: continue work
poolside/laguna-xs-2.1:none__json_schema PASS 100% 146 in → 30 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
poolside/laguna-xs-2.1:xhigh PASS 100% 145 in → 412 out (401 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 142 in → 4,202 out (4,184 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
poolside/laguna-xs.2:high PASS 100% 182 in → 269 out (256 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-xs.2:high__json_schema PASS 100% 183 in → 331 out (312 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
poolside/laguna-xs.2:low PASS 100% 184 in → 265 out (252 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-xs.2:low__json_schema PASS 100% 181 in → 343 out (324 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
poolside/laguna-xs.2:medium PASS 100% 180 in → 299 out (286 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-xs.2:medium__json_schema PASS 100% 182 in → 272 out (254 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
poolside/laguna-xs.2:none PASS 100% 184 in → 12 out (0 reasoning)
WIP: add main.py WIP: continue work
poolside/laguna-xs.2:none__json_schema PASS 100% 180 in → 18 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
poolside/laguna-xs.2:xhigh PASS 100% 179 in → 307 out (294 reasoning)
WIP: continue work WIP: add main.py
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 180 in → 260 out (241 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
prism-ml/ternary-bonsai-2-27b:high PASS 100% 145 in → 336 out (321 reasoning)
WIP: add main.py WIP: continue work
prism-ml/ternary-bonsai-2-27b:low PASS 100% 144 in → 334 out (319 reasoning)
WIP: continue work WIP: add main.py
prism-ml/ternary-bonsai-2-27b:low__json_schema PASS 100% 253 in → 2,067 out (2,047 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: add main.py","WIP: continue work"]}
prism-ml/ternary-bonsai-2-27b:max PASS 100% 190 in → 2,638 out (2,623 reasoning)
WIP: continue work WIP: add main.py
prism-ml/ternary-bonsai-2-27b:medium PASS 100% 145 in → 364 out (349 reasoning)
WIP: continue work WIP: add main.py
prism-ml/ternary-bonsai-2-27b:medium__json_schema PASS 100% 254 in → 658 out (638 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
prism-ml/ternary-bonsai-2-27b:none__json_schema PASS 100% 255 in → 33 out (0 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
prism-ml/ternary-bonsai-2-27b:xhigh PASS 100% 189 in → 454 out (438 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-27b:high PASS 100% 149 in → 951 out (870 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-27b:low PASS 100% 146 in → 1,244 out (1,228 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-27b:low__json_schema PASS 100% 147 in → 1,317 out (1,257 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
qwen/qwen3.6-27b:medium PASS 100% 147 in → 1,176 out (1,144 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-27b:medium__json_schema PASS 100% 147 in → 1,248 out (1,147 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 143 in → 1,324 out (1,307 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 147 in → 1,109 out (1,000 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 146 in → 1,337 out (1,248 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 147 in → 1,220 out (1,143 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 147 in → 974 out (957 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 146 in → 594 out (485 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 145 in → 1,233 out (1,110 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 146 in → 919 out (838 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 147 in → 13 out (0 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 148 in → 33 out (0 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 142 in → 1,201 out (1,158 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 146 in → 1,631 out (1,504 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
qwen/qwen3.6-flash:high PASS 100% 146 in → 1,226 out (1,209 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-flash:low PASS 100% 145 in → 1,094 out (1,077 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-flash:medium PASS 100% 146 in → 1,225 out (1,208 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-flash:none PASS 100% 150 in → 12 out (0 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.6-flash:xhigh PASS 100% 147 in → 1,155 out (1,138 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-flash:high PASS 100% 147 in → 1,115 out (1,098 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-flash:low PASS 100% 148 in → 1,113 out (1,096 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-flash:max PASS 100% 145 in → 1,209 out (1,192 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-flash:medium PASS 100% 147 in → 1,046 out (1,029 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-flash:none PASS 100% 149 in → 12 out (0 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-flash:none__json_schema PASS 100% 150 in → 12 out (0 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-flash:xhigh PASS 100% 145 in → 999 out (982 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-max:high PASS 100% 148 in → 321 out (304 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-max:low PASS 100% 145 in → 243 out (226 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-max:none PASS 100% 149 in → 12 out (0 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-max:xhigh PASS 100% 148 in → 377 out (360 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-plus:high PASS 100% 147 in → 1,559 out (1,542 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.7-plus:low PASS 100% 141 in → 1,497 out (1,480 reasoning)
WIP: add main.py WIP: continue work
qwen/qwen3.7-plus:xhigh PASS 100% 147 in → 321 out (304 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.8-flash:high PASS 100% 197 in → 6,193 out (6,178 reasoning)
WIP: add main.py WIP: continue work
qwen/qwen3.8-flash:low PASS 100% 195 in → 308 out (293 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.8-flash:low__json_schema PASS 100% 199 in → 436 out (416 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
qwen/qwen3.8-flash:max PASS 100% 199 in → 560 out (545 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.8-flash:medium PASS 100% 199 in → 3,398 out (3,383 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.8-flash:xhigh PASS 100% 199 in → 3,162 out (3,147 reasoning)
WIP: add main.py WIP: continue work
qwen/qwen3.8-max:high PASS 100% 182 in → 4,297 out (4,280 reasoning)
WIP: add main.py WIP: continue work
qwen/qwen3.8-max:high__json_schema PASS 100% 188 in → 7,104 out (7,068 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
qwen/qwen3.8-max:low PASS 100% 173 in → 253 out (236 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.8-max:low__json_schema PASS 100% 174 in → 270 out (249 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
qwen/qwen3.8-max:max PASS 100% 186 in → 8,661 out (8,644 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.8-max:medium PASS 100% 145 in → 298 out (281 reasoning)
WIP: continue work WIP: add main.py
qwen/qwen3.8-max:medium__json_schema PASS 100% 148 in → 793 out (759 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
qwen/qwen3.8-max:xhigh PASS 100% 183 in → 4,414 out (4,397 reasoning)
WIP: continue work WIP: add main.py
sakana/fugu-max:high PASS 100% 214 in → 67 out (53 reasoning)
WIP: add main.py WIP: continue work
sakana/fugu-max:high__json_schema PASS 100% 219 in → 49 out (23 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
sakana/fugu-max:low PASS 100% 215 in → 14 out (0 reasoning)
WIP: continue work WIP: add main.py
sakana/fugu-max:low__json_schema PASS 100% 218 in → 54 out (28 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
sakana/fugu-max:max PASS 100% 217 in → 3,657 out (3,643 reasoning)
WIP: continue work WIP: add main.py
sakana/fugu-max:max__json_schema PASS 100% 221 in → 961 out (936 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
sakana/fugu-max:medium PASS 100% 218 in → 169 out (155 reasoning)
WIP: continue work WIP: add main.py
sakana/fugu-max:xhigh PASS 100% 217 in → 435 out (421 reasoning)
WIP: continue work WIP: add main.py
sakana/fugu-max:xhigh__json_schema PASS 100% 218 in → 1,030 out (1,007 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
stealth/ox-alpha:low PASS 100% 218 in → 14 out (0 reasoning)
WIP: add main.py WIP: continue work
stealth/ox-alpha:max PASS 100% 218 in → 287 out (0 reasoning)
WIP: continue work WIP: add main.py
tencent/hy3:high PASS 100% 140 in → 1,133 out (1,119 reasoning)
WIP: continue work WIP: add main.py
tencent/hy3:high__json_schema PASS 100% 138 in → 1,148 out (1,196 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
tencent/hy3:low PASS 100% 141 in → 549 out (535 reasoning)
WIP: continue work WIP: add main.py
tencent/hy3:low__json_schema PASS 100% 137 in → 741 out (760 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
tencent/hy3:medium PASS 100% 140 in → 891 out (877 reasoning)
WIP: continue work WIP: add main.py
tencent/hy3:medium__json_schema PASS 100% 138 in → 807 out (791 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
tencent/hy3:none PASS 100% 139 in → 13 out (0 reasoning)
WIP: continue work WIP: add main.py
tencent/hy3:none__json_schema PASS 100% 141 in → 24 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
tencent/hy3:xhigh PASS 100% 136 in → 1,025 out (1,011 reasoning)
WIP: continue work WIP: add main.py
tencent/hy3:xhigh__json_schema PASS 100% 140 in → 1,274 out (1,191 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
thinkingmachines/inkling-small:high PASS 100% 141 in → 5,677 out (5,880 reasoning)
WIP: continue work WIP: add main.py
thinkingmachines/inkling-small:high__json_schema PASS 100% 143 in → 2,859 out (2,831 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
thinkingmachines/inkling-small:low PASS 100% 142 in → 207 out (145 reasoning)
WIP: continue work WIP: add main.py
thinkingmachines/inkling-small:low__json_schema PASS 100% 143 in → 190 out (165 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
thinkingmachines/inkling-small:max PASS 100% 143 in → 882 out (864 reasoning)
WIP: continue work WIP: add main.py
thinkingmachines/inkling-small:max__json_schema PASS 100% 140 in → 7,490 out (7,462 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
thinkingmachines/inkling-small:medium PASS 100% 142 in → 1,294 out (1,276 reasoning)
WIP: add main.py WIP: continue work
thinkingmachines/inkling-small:medium__json_schema PASS 100% 141 in → 565 out (540 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
thinkingmachines/inkling-small:none PASS 100% 141 in → 15 out (0 reasoning)
WIP: continue work WIP: add main.py
thinkingmachines/inkling-small:xhigh PASS 100% 144 in → 4,448 out (4,430 reasoning)
WIP: continue work WIP: add main.py
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 140 in → 5,472 out (5,446 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
thinkingmachines/inkling:high PASS 100% 139 in → 694 out (767 reasoning)
WIP: continue work WIP: add main.py
thinkingmachines/inkling:low PASS 100% 141 in → 55 out (36 reasoning)
WIP: continue work WIP: add main.py
thinkingmachines/inkling:max PASS 100% 141 in → 504 out (485 reasoning)
WIP: continue work WIP: add main.py
thinkingmachines/inkling:max__json_schema PASS 100% 142 in → 2,033 out (2,007 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
thinkingmachines/inkling:medium PASS 100% 142 in → 420 out (402 reasoning)
WIP: continue work WIP: add main.py
thinkingmachines/inkling:none PASS 100% 138 in → 16 out (0 reasoning)
WIP: add main.py WIP: continue work
thinkingmachines/inkling:xhigh PASS 100% 139 in → 441 out (423 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.3:high PASS 100% 307 in → 530 out (518 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.3:high__json_schema PASS 100% 375 in → 1,452 out (1,426 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
x-ai/grok-4.3:low PASS 100% 311 in → 747 out (735 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.3:low__json_schema PASS 100% 378 in → 544 out (518 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
x-ai/grok-4.3:max PASS 100% 306 in → 1,037 out (1,025 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.3:max__json_schema PASS 100% 377 in → 1,181 out (1,165 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
x-ai/grok-4.3:medium PASS 100% 311 in → 471 out (459 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.3:medium__json_schema PASS 100% 383 in → 847 out (821 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
x-ai/grok-4.3:xhigh PASS 100% 305 in → 495 out (483 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.3:xhigh__json_schema PASS 100% 372 in → 853 out (837 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
x-ai/grok-4.5:high PASS 100% 328 in → 1,171 out (1,161 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.5:high__json_schema PASS 100% 408 in → 828 out (803 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
x-ai/grok-4.5:low PASS 100% 329 in → 429 out (419 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.5:low__json_schema PASS 100% 408 in → 451 out (426 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
x-ai/grok-4.5:max PASS 100% 326 in → 555 out (545 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.5:max__json_schema PASS 100% 408 in → 1,117 out (1,092 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
x-ai/grok-4.5:medium PASS 100% 328 in → 1,431 out (1,421 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.5:medium__json_schema PASS 100% 411 in → 2,057 out (2,042 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
x-ai/grok-4.5:xhigh PASS 100% 324 in → 1,204 out (1,194 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.5:xhigh__json_schema PASS 100% 406 in → 634 out (609 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
x-ai/grok-4.7:high PASS 100% 1,362 in → 1,646 out (1,636 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.7:high__json_schema PASS 100% 1,445 in → 1,011 out (996 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
x-ai/grok-4.7:low PASS 100% 1,368 in → 699 out (689 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.7:low__json_schema PASS 100% 1,442 in → 501 out (486 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
x-ai/grok-4.7:max PASS 100% 1,363 in → 4,137 out (4,127 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.7:max__json_schema PASS 100% 1,446 in → 1,312 out (1,297 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
x-ai/grok-4.7:medium PASS 100% 1,366 in → 565 out (555 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.7:medium__json_schema PASS 100% 1,443 in → 1,036 out (1,021 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
x-ai/grok-4.7:xhigh PASS 100% 1,365 in → 563 out (553 reasoning)
WIP: continue work WIP: add main.py
x-ai/grok-4.7:xhigh__json_schema PASS 100% 1,445 in → 1,326 out (1,311 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
xiaomi/mimo-v2.6-flash:high PASS 100% 141 in → 40 out (25 reasoning)
WIP: continue work WIP: add main.py
xiaomi/mimo-v2.6-flash:high__json_schema PASS 100% 142 in → 149 out (118 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
xiaomi/mimo-v2.6-flash:low PASS 100% 142 in → 246 out (231 reasoning)
WIP: continue work WIP: add main.py
xiaomi/mimo-v2.6-flash:low__json_schema PASS 100% 136 in → 52 out (28 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
xiaomi/mimo-v2.6-flash:max PASS 100% 143 in → 84 out (70 reasoning)
WIP: add main.py WIP: continue work
xiaomi/mimo-v2.6-flash:max__json_schema PASS 100% 144 in → 78 out (47 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
xiaomi/mimo-v2.6-flash:medium PASS 100% 141 in → 767 out (752 reasoning)
WIP: continue work WIP: add main.py
xiaomi/mimo-v2.6-flash:medium__json_schema PASS 100% 143 in → 74 out (50 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
xiaomi/mimo-v2.6-flash:none PASS 100% 144 in → 13 out (0 reasoning)
WIP: add main.py WIP: continue work
xiaomi/mimo-v2.6-flash:xhigh PASS 100% 144 in → 93 out (79 reasoning)
WIP: add main.py WIP: continue work
xiaomi/mimo-v2.6-pro:high PASS 100% 142 in → 102 out (88 reasoning)
WIP: continue work WIP: add main.py
xiaomi/mimo-v2.6-pro:high__json_schema PASS 100% 141 in → 304 out (280 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": ["WIP: add main.py", "WIP: continue work"] }
xiaomi/mimo-v2.6-pro:low PASS 100% 140 in → 49 out (35 reasoning)
WIP: continue work WIP: add main.py
xiaomi/mimo-v2.6-pro:low__json_schema PASS 100% 142 in → 71 out (45 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
xiaomi/mimo-v2.6-pro:max PASS 100% 142 in → 61 out (47 reasoning)
WIP: continue work WIP: add main.py
xiaomi/mimo-v2.6-pro:max__json_schema PASS 100% 142 in → 144 out (118 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
xiaomi/mimo-v2.6-pro:medium PASS 100% 142 in → 437 out (423 reasoning)
WIP: continue work WIP: add main.py
xiaomi/mimo-v2.6-pro:medium__json_schema PASS 100% 143 in → 45 out (21 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue work", "WIP: add main.py"] }
xiaomi/mimo-v2.6-pro:xhigh PASS 100% 142 in → 99 out (85 reasoning)
WIP: continue work WIP: add main.py
xiaomi/mimo-v2.6-pro:xhigh__json_schema PASS 100% 142 in → 152 out (125 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add main.py", "WIP: continue work" ] }
z-ai/glm-4.7-flash:high PASS 100% 135 in → 683 out (589 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-4.7-flash:high__json_schema PASS 100% 136 in → 756 out (653 reasoning)
WIP: add main.py WIP: continue work
JSON Schema Structured Output
(raw) {"commits": ["WIP: add main.py", "WIP: continue work"]}
z-ai/glm-4.7-flash:low PASS 100% 135 in → 551 out (498 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-4.7-flash:medium PASS 100% 134 in → 1,049 out (1,008 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-4.7-flash:none PASS 100% 137 in → 13 out (0 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-4.7-flash:none__json_schema PASS 100% 136 in → 28 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
z-ai/glm-4.7-flash:xhigh PASS 100% 133 in → 487 out (460 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 136 in → 609 out (534 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work", "WIP: add main.py"]}
z-ai/glm-5.2:high PASS 100% 145 in → 587 out (546 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-5.2:high__json_schema PASS 100% 140 in → 610 out (581 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
z-ai/glm-5.2:low PASS 100% 138 in → 677 out (663 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-5.2:low__json_schema PASS 100% 330 in → 251 out (222 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
z-ai/glm-5.2:medium PASS 100% 144 in → 594 out (580 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-5.2:none PASS 100% 136 in → 12 out (0 reasoning)
WIP: add main.py WIP: continue work
z-ai/glm-5.2:none__json_schema PASS 100% 135 in → 20 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": [ "WIP: continue work", "WIP: add main.py" ]}
z-ai/glm-5.2:xhigh PASS 100% 136 in → 510 out (458 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-5.2:xhigh__json_schema PASS 100% 140 in → 790 out (723 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
z-ai/glm-5.3-flash:high PASS 100% 143 in → 79 out (65 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-5.3-flash:high__json_schema PASS 100% 141 in → 48 out (29 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
z-ai/glm-5.3-flash:max PASS 100% 141 in → 1,560 out (1,506 reasoning)
WIP: continue work WIP: add main.py
z-ai/glm-5.3-flash:max__json_schema PASS 100% 143 in → 297 out (226 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue work", "WIP: add main.py" ] }
z-ai/glm-5.3-flash:medium PASS 100% 142 in → 51 out (37 reasoning)
WIP: add main.py WIP: continue work
z-ai/glm-5.3-flash:medium__json_schema PASS 100% 144 in → 19 out (0 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
z-ai/glm-5.3-flash:xhigh__json_schema PASS 100% 143 in → 99 out (80 reasoning)
WIP: continue work WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","WIP: add main.py"]}
anthropic/claude-opus-4.8:medium FAIL 50% 196 in → 894 out (145 reasoning)
WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py']
anthropic/claude-opus-5.5:low__json_schema FAIL 50% 437 in → 24 out (0 reasoning)
WIP: continue work Complete feature
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work","Complete feature"]}
Failure: Missing expected commit messages: ['WIP: add main.py']; Extra selected commit messages: ['Complete feature']
anthropic/claude-opus-5:low FAIL 50% 194 in → 136 out (0 reasoning)
WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py']
anthropic/claude-opus-5:xhigh FAIL 50% 196 in → 505 out (110 reasoning)
WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py']
anthropic/claude-opus-5:xhigh__json_schema FAIL 50% 435 in → 905 out (108 reasoning)
WIP: continue work
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue work"]}
Failure: Missing expected commit messages: ['WIP: add main.py']
arcee-ai/trinity-mini:high FAIL 50% 133 in → 329 out (339 reasoning)
WIP: continuework WIP: add main.py
Failure: Missing expected commit messages: ['WIP: continue work']; Extra selected commit messages: ['WIP: continuework']
arcee-ai/trinity-mini:low FAIL 50% 129 in → 679 out (732 reasoning)
WIP: continuework WIP: add main.py
Failure: Missing expected commit messages: ['WIP: continue work']; Extra selected commit messages: ['WIP: continuework']
google/gemini-3.1-pro-preview:medium FAIL 50% 133 in → 3,718 out (3,713 reasoning)
WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py']
google/gemini-3.6-flash:max FAIL 50% 135 in → 4,890 out (4,885 reasoning)
WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py']
google/gemini-3.6-flash:medium FAIL 50% 133 in → 3,383 out (3,378 reasoning)
WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py']
google/gemini-3.6-flash:xhigh FAIL 50% 134 in → 6,000 out (5,995 reasoning)
WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py']
google/gemini-3.6-flash:xhigh__json_schema FAIL 50% 133 in → 6,256 out (6,247 reasoning)
WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work"]}
Failure: Missing expected commit messages: ['WIP: add main.py']
minimax/minimax-m2.5:medium FAIL 50% 167 in → 245 out (236 reasoning)
IP: continue work WIP: add main.py
Failure: Missing expected commit messages: ['WIP: continue work']; Extra selected commit messages: ['IP: continue work']
openai/gpt-5.4:xhigh__json_schema FAIL 50% 182 in → 8,364 out (8,343 reasoning)
WIP: continue work
JSON Schema Structured Output
(raw) {"commits":["WIP: continue work"]}
Failure: Missing expected commit messages: ['WIP: add main.py']
qwen/qwen3.7-max:medium FAIL 50% 146 in → 1,562 out (1,552 reasoning)
WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py']
anthropic/claude-haiku-4.5:none FAIL 0% 145 in → 28 out (0 reasoning)
0e7dda2 WIP: continue work 75fd37a WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 328 in → 19 out (0 reasoning)
6f19546 09562c9
JSON Schema Structured Output
(raw) {"commits": ["6f19546", "09562c9"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
anthropic/claude-opus-4.8:high FAIL 0% 196 in → 30 out (0 reasoning)
0b33ab2 WIP: add main.py 77c9eac WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']
anthropic/claude-opus-4.8:xhigh FAIL 0% 191 in → 27 out (0 reasoning)
f349a53 WIP: add main.py 751b869 WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']
anthropic/claude-sonnet-4.6:low__json_schema FAIL 0% 330 in → 32 out (0 reasoning)
07701b8 WIP: continue work a1fb56a WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["07701b8 WIP: continue work","a1fb56a WIP: add main.py"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
anthropic/claude-sonnet-4.6:none__json_schema FAIL 0% 327 in → 31 out (0 reasoning)
dd3d7ec WIP: continue work 650fc0c WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["dd3d7ec WIP: continue work","650fc0c WIP: add main.py"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
anthropic/claude-sonnet-5:none FAIL 0% 196 in → 29 out (0 reasoning)
b3a8c2f WIP: continue work e37749a WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high__json_schema FAIL 0% 130 in → 926 out (912 reasoning)
952f19f Complete feature 1d77d77 WIP: continue work 1045e1c WIP: add main.py 6397762 Initial commit
JSON Schema Structured Output
(raw) { "commits": [ "952f19f Complete feature", "1d77d77 WIP: continue work", "1045e1c WIP: add main.py", "6397762 Initial commit" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['Complete feature', 'WIP: continue work', 'WIP: add main.py', 'Initial commit']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 130 in → 660 out (632 reasoning)
b105b17 Complete feature a4e5d61 WIP: continue work a0757cc WIP: add main.py 588ee81 Initial commit
JSON Schema Structured Output
(raw) { "commits": [ "b105b17 Complete feature", "a4e5d61 WIP: continue work", "a0757cc WIP: add main.py", "588ee81 Initial commit" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['Complete feature', 'WIP: continue work', 'WIP: add main.py', 'Initial commit']
arcee-ai/trinity-mini:medium FAIL 0% 129 in → 508 out (535 reasoning)
WIP: continue workWIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 134 in → 432 out (403 reasoning)
05bc56f Complete feature 2b88d71 WIP: continue work f24e2e8 WIP: add main.py 7c3ed7f Initial commit
JSON Schema Structured Output
(raw) { "commits": [ "05bc56f Complete feature", "2b88d71 WIP: continue work", "f24e2e8 WIP: add main.py", "7c3ed7f Initial commit" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['Complete feature', 'WIP: continue work', 'WIP: add main.py', 'Initial commit']
arcee-ai/trinity-mini:xhigh FAIL 0% 128 in → 468 out (490 reasoning)
WIP: continue workWIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work']
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 0% 131 in → 210 out (169 reasoning)
eff031a WIP: add main.py 8c33da9 WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "eff031a WIP: add main.py", "8c33da9 WIP: continue work" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 132 in → 22 out (0 reasoning)
50336e2 WIP: continue work 704b9d4 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 132 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "commits": [] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:high__json_schema FAIL 0% 133 in → 37 out (0 reasoning)
1fa8dff WIP: add main.py 36b1511 WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "1fa8dff WIP: add main.py", "36b1511 WIP: continue work" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 135 in → 636 out (599 reasoning)
bacdb82 WIP: continue work 2a0b637 WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "bacdb82 WIP: continue work", "2a0b637 WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
deepseek/deepseek-v4-pro:none FAIL 0% 128 in → 19 out (0 reasoning)
c090918 WIP: add main.py 9cadfb7 WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 132 in → 35 out (0 reasoning)
eded0bf WIP: add main.py 7d6a928 WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "eded0bf WIP: add main.py", "7d6a928 WIP: continue work" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']
deepseek/deepseek-v4-pro:xhigh FAIL 0% 211 in → 2,744 out (2,721 reasoning)
b01f07c WIP: continue work 07fe06d WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 212 in → 750 out (713 reasoning)
e9c286b WIP: continue work 204b314 WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "e9c286b WIP: continue work", "204b314 WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
deepseek/deepseek-v4.1-flash:none FAIL 0% 130 in → 21 out (0 reasoning)
74b58cd WIP: continue work 6b58221 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
deepseek/deepseek-v4.1-flash:none__json_schema FAIL 0% 131 in → 29 out (0 reasoning)
2d048ff WIP: continue work f28352d WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["2d048ff WIP: continue work", "f28352d WIP: add main.py"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
google/gemini-3.5-flash-lite:low FAIL 0% 132 in → 20 out (0 reasoning)
fc3ae05 WIP: continue work cfbada1 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
google/gemini-3.6-flash:low__json_schema FAIL 0% 135 in → 43 out (0 reasoning)
2072904 WIP: continue work f3942ff WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "2072904 WIP: continue work", "f3942ff WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
google/gemini-3.6-flash:medium__json_schema FAIL 0% 213 in → 1,311 out (1,269 reasoning)
4ae8649 WIP: continue work 05468ae WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "4ae8649 WIP: continue work", "05468ae WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
google/gemini-3.8-flash:max FAIL 0% 138 in → 2,162 out (0 reasoning)
(empty output)
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']
google/gemma-4-26b-a4b-it:none__json_schema FAIL 0% 144 in → 32 out (0 reasoning)
480cfc2 WIP: continue work 43b1964 WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["480cfc2 WIP: continue work", "43b1964 WIP: add main.py"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
google/gemma-4-31b-it:none__json_schema FAIL 0% 144 in → 45 out (0 reasoning)
2779654 WIP: continue work 53857f6 WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "2779654 WIP: continue work", "53857f6 WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
d987785 WIP: continue work 9f831bf WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["d987785 WIP: continue work", "9f831bf WIP: add main.py"] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
ibm-granite/granite-4.2-8b:high__json_schema FAIL 0% 148 in → 65 out (0 reasoning)
e71c3e3 Complete feature b50251a WIP: continue work cbdbd16 WIP: add main.py 4d4f742 Initial commit
JSON Schema Structured Output
(raw) { "commits": [ "e71c3e3 Complete feature", "b50251a WIP: continue work", "cbdbd16 WIP: add main.py", "4d4f742 Initial commit" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['Complete feature', 'WIP: continue work', 'WIP: add main.py', 'Initial commit']
ibm-granite/granite-4.2-8b:low__json_schema FAIL 0% 156 in → 26 out (0 reasoning)
2f2dfe5 7c04c05
JSON Schema Structured Output
(raw) { "commits" : ["2f2dfe5", "7c04c05"] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
ibm-granite/granite-4.2-8b:max__json_schema FAIL 0% 145 in → 62 out (0 reasoning)
ba67f78 Complete feature 46897fa WIP: continue work fc8f3fa WIP: add main.py 9f214db Initial commit
JSON Schema Structured Output
(raw) { "commits": [ "ba67f78 Complete feature", "46897fa WIP: continue work", "fc8f3fa WIP: add main.py", "9f214db Initial commit" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['Complete feature', 'WIP: continue work', 'WIP: add main.py', 'Initial commit']
ibm-granite/granite-4.2-8b:none__json_schema FAIL 0% 149 in → 45 out (0 reasoning)
e53e8e6 WIP: add main.py e3562b9 WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "e53e8e6 WIP: add main.py", "e3562b9 WIP: continue work" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']
ibm-granite/granite-4.2-8b:xhigh__json_schema FAIL 0% 144 in → 61 out (0 reasoning)
92533ab Complete feature 3b16429 WIP: continue work c8584a6 WIP: add main.py 1974168 Initial commit
JSON Schema Structured Output
(raw) { "commits": [ "92533ab Complete feature", "3b16429 WIP: continue work", "c8584a6 WIP: add main.py", "1974168 Initial commit" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['Complete feature', 'WIP: continue work', 'WIP: add main.py', 'Initial commit']
inception/mercury-2.5:high__json_schema FAIL 0% 360 in → 1,957 out (1,818 reasoning)
32aff80 15ac586
JSON Schema Structured Output
(raw) { "commits": [ "32aff80", "15ac586" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
inception/mercury-2.5:low FAIL 0% 128 in → 283 out (258 reasoning)
e2d11ad WIP: continue work de825be WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
inception/mercury-2.5:low__json_schema FAIL 0% 353 in → 500 out (240 reasoning)
b7a6507 WIP: continue work 04b256e WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "b7a6507 WIP: continue work", "04b256e WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
inception/mercury-2.5:none FAIL 0% 129 in → 23 out (0 reasoning)
51bee8d WIP: continue work 1d915b6 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
inclusionai/ling-2.6-flash FAIL 0% 153 in → 23 out
dd89aa3 WIP: continue work 07ced5f WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
liquid/lfm-2-24b-a2b FAIL 0% 142 in → 17 out
cae8489 4aa522d c8adf8b
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['Complete feature', 'WIP: continue work', 'WIP: add main.py']
Invalid JSON. Output: 479831e 588b637
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 7 (char 6)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 7 (char 6)
meta/muse-spark-1.1:xhigh__json_schema FAIL 0% 134 in → 1,524 out (1,495 reasoning)
2f224b8 88a034c
JSON Schema Structured Output
(raw) { "commits": ["2f224b8", "88a034c"] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:medium__json_schema FAIL 0% 287 in → 35 out (0 reasoning)
7d16010 WIP: continue work c067651 WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "7d16010 WIP: continue work", "c067651 WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
minimax/minimax-m3:xhigh__json_schema FAIL 0% 291 in → 37 out (0 reasoning)
cc15d8a WIP: continue work de4e123 WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "cc15d8a WIP: continue work", "de4e123 WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
mistralai/devstral-2512 FAIL 0% 141 in → 27 out
c251895 WIP: continue work f90a9e0 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
mistralai/mistral-medium-3-5:none FAIL 0% 153 in → 27 out (0 reasoning)
e5a99b1 WIP: continue work 3478d64 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
nvidia/nemotron-3-nano-30b-a3b:high FAIL 0% 154 in → 32,768 out (67 reasoning)
(empty output)
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']
nvidia/nemotron-3-nano-30b-a3b:medium FAIL 0% 153 in → 697 out (666 reasoning)
WIP: continue workWIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work']
openai/gpt-5.4-mini:none__json_schema FAIL 0% 179 in → 33 out (0 reasoning)
4177b18 WIP: continue work 465efae WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["4177b18 WIP: continue work","465efae WIP: add main.py"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
openai/gpt-5.4-nano:none FAIL 0% 130 in → 23 out (0 reasoning)
e920244 WIP: continue work 254b22d WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
openai/gpt-5.4:none__json_schema FAIL 0% 180 in → 34 out (0 reasoning)
dbc07fe WIP: continue work 946a6e5 WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["dbc07fe WIP: continue work","946a6e5 WIP: add main.py"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
poolside/laguna-m.1:none__json_schema FAIL 0% 144 in → 41 out (0 reasoning)
0f8ff5f WIP: continue work 982d08b WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "0f8ff5f WIP: continue work", "982d08b WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
poolside/laguna-xs-2.1:high FAIL 0% 147 in → 554 out (529 reasoning)
d535896 WIP: continue work 826791a WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
poolside/laguna-xs-2.1:medium__json_schema FAIL 0% 143 in → 279 out (234 reasoning)
fe491b1 WIP: continue work 19a5d53 WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "fe491b1 WIP: continue work", "19a5d53 WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 3 column 1 (char 2)
Failure: Failed to parse structured JSON response: Expecting value: line 3 column 1 (char 2)
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 3 column 1 (char 2)
Failure: Failed to parse structured JSON response: Expecting value: line 3 column 1 (char 2)
prism-ml/ternary-bonsai-2-27b:none FAIL 0% 148 in → 27 out (0 reasoning)
99e6777 WIP: add main.py d47c11e WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 3 column 1 (char 2)
Failure: Failed to parse structured JSON response: Expecting value: line 3 column 1 (char 2)
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:none FAIL 0% 148 in → 25 out (0 reasoning)
1175c79 WIP: continue work 6ed3ae1 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 148 in → 46 out (0 reasoning)
a541dcd WIP: continue work 3ac73f3 WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "a541dcd WIP: continue work", "3ac73f3 WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-42152b88-0603-9bae-91a0-2b3cb72e894b","request_id":"42152b88-0603-9bae-91a0-2b3cb72e894b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-74421f0e-28fa-9e72-a4d3-9a226189236d","request_id":"74421f0e-28fa-9e72-a4d3-9a226189236d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9ee2d842-0342-9e8c-9be1-fb690aab4904","request_id":"9ee2d842-0342-9e8c-9be1-fb690aab4904"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bc425336-fe0e-9cd6-a61c-8ffeb403f4eb","request_id":"bc425336-fe0e-9cd6-a61c-8ffeb403f4eb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-86bd7269-ba7e-9d86-a3e8-d17db0fec1c4","request_id":"86bd7269-ba7e-9d86-a3e8-d17db0fec1c4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-87a6e26e-3fbd-902b-af43-4c2c7430c7ad"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9150ddfb-7eb0-9e68-b1de-bda654784f70"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-fbb6cfcc-3b51-9ab0-953e-c2057928dbde"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c0f79d16-cfa8-9ebd-8b52-eb372d40897a"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-31f691f5-f9fb-9509-89ef-e362deac96d5"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8771c4f0-9f7e-9865-90cf-2e5eed279a83","request_id":"8771c4f0-9f7e-9865-90cf-2e5eed279a83"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-12f59d99-707f-945c-9b3e-3fa2f65076f2","request_id":"12f59d99-707f-945c-9b3e-3fa2f65076f2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6553d392-23c0-978e-a0a8-e2cb74ba16a2","request_id":"6553d392-23c0-978e-a0a8-e2cb74ba16a2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-013b5679-2bfa-9a77-8010-72a6dad4a87b","request_id":"013b5679-2bfa-9a77-8010-72a6dad4a87b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-03fa0fe2-0663-9286-9d3e-b6f3a721a8af","request_id":"03fa0fe2-0663-9286-9d3e-b6f3a721a8af"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b5076fdc-7095-9951-81b8-8ed2ccaeb0d9","request_id":"b5076fdc-7095-9951-81b8-8ed2ccaeb0d9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0d2968d4-e812-966a-9686-a40bbe8fd2b2","request_id":"0d2968d4-e812-966a-9686-a40bbe8fd2b2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium FAIL 0% 148 in → 1,640 out (1,609 reasoning)
e54622b WIP: continue work c22a439 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d2f1e357-2b7c-9531-9017-7d5b25368602","request_id":"d2f1e357-2b7c-9531-9017-7d5b25368602"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none FAIL 0% 148 in → 25 out (0 reasoning)
4816f5f WIP: continue work 4db3a71 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-04010de1-041a-9c81-90fb-3457a65c9fe3","request_id":"04010de1-041a-9c81-90fb-3457a65c9fe3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d8dc5174-dc6d-9549-bab1-43a5e6bcb6f1","request_id":"d8dc5174-dc6d-9549-bab1-43a5e6bcb6f1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.8-flash:high__json_schema FAIL 0% 198 in → 466 out (432 reasoning)
3fcac25 WIP: continue work 4858f99 WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["3fcac25 WIP: continue work", "4858f99 WIP: add main.py"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
qwen/qwen3.8-flash:max__json_schema FAIL 0% 200 in → 5,393 out (5,368 reasoning)
9145667 5670c4c
JSON Schema Structured Output
(raw) {"commits": ["9145667", "5670c4c"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
qwen/qwen3.8-flash:medium__json_schema FAIL 0% 199 in → 434 out (399 reasoning)
2677950 WIP: continue work fb730c2 WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["2677950 WIP: continue work", "fb730c2 WIP: add main.py"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
qwen/qwen3.8-flash:xhigh__json_schema FAIL 0% 202 in → 313 out (290 reasoning)
a3f506d 9f78822
JSON Schema Structured Output
(raw) {"commits":["a3f506d","9f78822"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
qwen/qwen3.8-max:max__json_schema FAIL 0% 184 in → 3,699 out (3,666 reasoning)
a5eedda 07cdba7
JSON Schema Structured Output
(raw) { "commits": [ "a5eedda", "07cdba7" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
qwen/qwen3.8-max:xhigh__json_schema FAIL 0% 187 in → 5,859 out (5,839 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "commits": [ "" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']
sakana/fugu-max:medium__json_schema FAIL 0% 217 in → 95 out (60 reasoning)
6071753 WIP: add main.py 0b92815 WIP: continue work
JSON Schema Structured Output
(raw) { "commits": [ "6071753 WIP: add main.py", "0b92815 WIP: continue work" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']
stealth/ox-alpha:high FAIL 0% 216 in → 54 out (0 reasoning)
d1eaaa5 WIP: continue work eae74e5 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
Invalid JSON. Output: WIP: add main.py WIP: continue work
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue work WIP: add main.py{ "commits": [ "WIP: continue work", "WIP: add main.py" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0% 142 in → 48 out (0 reasoning)
51cb3e9 Complete feature 2ec6f89 WIP: continue work 0c064c5 WIP: add main.py d451d4f Initial commit
JSON Schema Structured Output
(raw) { "commits": ["51cb3e9 Complete feature","2ec6f89 WIP: continue work","0c064c5 WIP: add main.py","d451d4f Initial commit"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['Complete feature', 'WIP: continue work', 'WIP: add main.py', 'Initial commit']
Invalid JSON. Output: WIP: continue work WIP: add main.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none FAIL 0% 305 in → 17 out (0 reasoning)
1997bd6 WIP: continue work f391193 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
x-ai/grok-4.3:none__json_schema FAIL 0% 374 in → 24 out (0 reasoning)
23d8b6d WIP: continue work 279c71b WIP: add main.py
JSON Schema Structured Output
(raw) {"commits":["23d8b6d WIP: continue work","279c71b WIP: add main.py"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
xiaomi/mimo-v2.6-flash:none__json_schema FAIL 0% 143 in → 36 out (0 reasoning)
15f8874 WIP: continue work aa6282a WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": ["15f8874 WIP: continue work", "aa6282a WIP: add main.py"] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
xiaomi/mimo-v2.6-flash:xhigh__json_schema FAIL 0% 143 in → 105 out (77 reasoning)
f88cc3b 1f71c1e
JSON Schema Structured Output
(raw) { "commits": [ "f88cc3b", "1f71c1e" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:medium__json_schema FAIL 0% 331 in → 693 out (653 reasoning)
9bc8a8a WIP: continue work 3f203b4 WIP: add main.py
JSON Schema Structured Output
(raw) { "commits": [ "9bc8a8a WIP: continue work", "3f203b4 WIP: add main.py" ] }
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
z-ai/glm-5.3-flash:low FAIL 0% 140 in → 87 out (64 reasoning)
1a75044 WIP: continue work 3253eb2 WIP: add main.py
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
z-ai/glm-5.3-flash:low__json_schema FAIL 0% 143 in → 30 out (0 reasoning)
4cea3f6 WIP: continue work 83a0efe WIP: add main.py
JSON Schema Structured Output
(raw) {"commits": ["4cea3f6 WIP: continue work", "83a0efe WIP: add main.py"]}
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: continue work', 'WIP: add main.py']
z-ai/glm-5.3-flash:xhigh FAIL 0% 144 in → 47 out (24 reasoning)
9361f71 WIP: add main.py 6731b07 WIP: continue work
Failure: Missing expected commit messages: ['WIP: add main.py', 'WIP: continue work']; Extra selected commit messages: ['WIP: add main.py', 'WIP: continue work']