Rebase/Squash style - intermediate commits
Tests ability to identify intermediate/rebase-style commits that should be squashed. Evaluates recognizing incomplete or intermediate work.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 touch initial.txt
  5. 05 git add initial.txt
  6. 06 git commit -m 'Add initial file'
  7. 07 echo 'line1' > data.txt
  8. 08 git add data.txt
  9. 09 git commit -m 'WIP: adding data'
  10. 10 echo 'line1 line2' > data.txt
  11. 11 git add data.txt
  12. 12 git commit -m 'WIP: continue data'
  13. 13 echo 'line1 line2 line3' > data.txt
  14. 14 git add data.txt
  15. 15 git commit -m 'Add complete data'
Prompt
This repository has a series of commits that incrementally build up a feature. Identify the WIP commits that should be marked as squash/fixup into a single clean commit. Do not include the final completed-feature commit. Output ONLY the selected commit subject lines, one per line, with no explanation.
Expected
WIP: adding data WIP: continue data

Scoped model quality, cost, API time, and token usage for commit_squash/f004.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 155 in → 135 out (82 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-fable-5:high__json_schema PASS 100% 394 in → 126 out (67 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
anthropic/claude-fable-5:low PASS 100% 156 in → 34 out (29 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-fable-5:low__json_schema PASS 100% 395 in → 26 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-fable-5:max PASS 100% 156 in → 523 out (104 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-fable-5:max__json_schema PASS 100% 396 in → 450 out (115 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
anthropic/claude-fable-5:medium PASS 100% 154 in → 33 out (19 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-fable-5:medium__json_schema PASS 100% 395 in → 28 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
anthropic/claude-fable-5:xhigh PASS 100% 152 in → 138 out (60 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 391 in → 143 out (70 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
anthropic/claude-haiku-4.5:high PASS 100% 146 in → 666 out (511 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 334 in → 1,758 out (844 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) {"commits": ["WIP: adding data", "WIP: continue data"]}
anthropic/claude-haiku-4.5:low PASS 100% 149 in → 560 out (500 reasoning)
WIP: adding data WIP: continue data
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 337 in → 1,020 out (605 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) {"commits": ["WIP: adding data", "WIP: continue data"]}
anthropic/claude-haiku-4.5:medium PASS 100% 151 in → 368 out (313 reasoning)
WIP: adding data WIP: continue data
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 335 in → 1,080 out (653 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) {"commits": ["WIP: adding data", "WIP: continue data"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 150 in → 627 out (428 reasoning)
WIP: adding data WIP: continue data
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 338 in → 1,002 out (598 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) {"commits": ["WIP: adding data", "WIP: continue data"]}
anthropic/claude-opus-4.6:high PASS 100% 114 in → 130 out (89 reasoning)
WIP: adding data WIP: continue data
anthropic/claude-opus-4.6:low PASS 100% 116 in → 14 out (0 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.6:max PASS 100% 115 in → 227 out (183 reasoning)
WIP: adding data WIP: continue data
anthropic/claude-opus-4.6:max__json_schema PASS 100% 305 in → 209 out (167 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) {"commits":["WIP: adding data","WIP: continue data"]}
anthropic/claude-opus-4.6:medium PASS 100% 119 in → 14 out (0 reasoning)
WIP: adding data WIP: continue data
anthropic/claude-opus-4.6:none PASS 100% 121 in → 14 out (0 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.6:xhigh PASS 100% 121 in → 90 out (52 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.7:high__json_schema PASS 100% 398 in → 28 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-opus-4.7:low PASS 100% 162 in → 19 out (0 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.7:low__json_schema PASS 100% 400 in → 28 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-opus-4.7:max PASS 100% 165 in → 249 out (81 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.7:max__json_schema PASS 100% 405 in → 28 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-opus-4.7:medium PASS 100% 162 in → 19 out (0 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 402 in → 28 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-opus-4.7:none__json_schema PASS 100% 402 in → 28 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 401 in → 28 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-opus-4.8:high PASS 100% 159 in → 17 out (0 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.8:high__json_schema PASS 100% 399 in → 28 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
anthropic/claude-opus-4.8:low PASS 100% 156 in → 102 out (68 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.8:low__json_schema PASS 100% 399 in → 28 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
anthropic/claude-opus-4.8:max PASS 100% 154 in → 1,651 out (174 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.8:max__json_schema PASS 100% 392 in → 1,017 out (158 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
anthropic/claude-opus-4.8:medium PASS 100% 157 in → 177 out (82 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 396 in → 28 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
anthropic/claude-opus-4.8:none PASS 100% 155 in → 17 out (0 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.8:xhigh PASS 100% 155 in → 987 out (113 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 398 in → 26 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-sonnet-4.6:high PASS 100% 121 in → 181 out (124 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 305 in → 20 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-sonnet-4.6:low PASS 100% 122 in → 14 out (0 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 307 in → 20 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-sonnet-4.6:max PASS 100% 118 in → 230 out (167 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 303 in → 326 out (209 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 117 in → 14 out (0 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 307 in → 20 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-sonnet-4.6:none PASS 100% 120 in → 14 out (0 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 307 in → 20 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 117 in → 183 out (139 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 302 in → 151 out (99 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-sonnet-5:high__json_schema PASS 100% 394 in → 26 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
anthropic/claude-sonnet-5:low PASS 100% 156 in → 17 out (0 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-sonnet-5:medium PASS 100% 159 in → 30 out (19 reasoning)
WIP: continue data WIP: adding data
anthropic/claude-sonnet-5:xhigh PASS 100% 159 in → 206 out (81 reasoning)
WIP: adding data WIP: continue data
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 396 in → 275 out (74 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
arcee-ai/trinity-large-thinking:high PASS 100% 108 in → 1,535 out (1,610 reasoning)
WIP: continue data WIP: adding data
arcee-ai/trinity-large-thinking:low PASS 100% 110 in → 5,540 out (5,759 reasoning)
WIP: adding data WIP: continue data
arcee-ai/trinity-large-thinking:medium PASS 100% 114 in → 3,132 out (3,253 reasoning)
WIP: continue data WIP: adding data
arcee-ai/trinity-mini:high PASS 100% 109 in → 375 out (368 reasoning)
WIP: continue data WIP: adding data
arcee-ai/trinity-mini:medium PASS 100% 112 in → 290 out (275 reasoning)
WIP: continue data WIP: adding data
deepseek/deepseek-v4-flash-0731:high PASS 100% 191 in → 8,315 out (8,455 reasoning)
WIP: adding data WIP: continue data
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 191 in → 6,862 out (7,117 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 110 in → 60 out (60 reasoning)
WIP: continue data WIP: adding data
deepseek/deepseek-v4-flash-0731:max PASS 100% 186 in → 2,096 out (2,083 reasoning)
WIP: continue data WIP: adding data
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 189 in → 303 out (284 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ]}
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 111 in → 1,113 out (1,144 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 119 in → 21 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue data", "WIP: adding data"] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 189 in → 505 out (493 reasoning)
WIP: continue data WIP: adding data
deepseek/deepseek-v4-flash:high FAIL 100% 112 in → 149 out (132 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
deepseek/deepseek-v4-flash:low PASS 100% 111 in → 619 out (642 reasoning)
WIP: continue data WIP: adding data
deepseek/deepseek-v4-flash:medium FAIL 100% 110 in → 163 out (146 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
deepseek/deepseek-v4-flash:medium__json_schema FAIL 100% 109 in → 748 out (774 reasoning)
WIP: continue data WIP: adding data Add initial file
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data", "Add initial file" ] }
Failure: Extra selected commit messages: ['Add initial file']
deepseek/deepseek-v4-pro:high PASS 100% 111 in → 519 out (506 reasoning)
WIP: continue data WIP: adding data
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 312 in → 208 out (187 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
deepseek/deepseek-v4-pro:low PASS 100% 111 in → 285 out (272 reasoning)
WIP: continue data WIP: adding data
deepseek/deepseek-v4-pro:medium PASS 100% 110 in → 111 out (98 reasoning)
WIP: continue data WIP: adding data
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 313 in → 182 out (161 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
deepseek/deepseek-v4-pro:xhigh PASS 100% 190 in → 3,767 out (3,754 reasoning)
WIP: continue data WIP: adding data
google/gemini-3-flash-preview:high PASS 100% 111 in → 334 out (323 reasoning)
WIP: continue data WIP: adding data
google/gemini-3-flash-preview:high__json_schema PASS 100% 187 in → 570 out (555 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3-flash-preview:low PASS 100% 112 in → 309 out (298 reasoning)
WIP: continue data WIP: adding data
google/gemini-3-flash-preview:low__json_schema PASS 100% 192 in → 513 out (498 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3-flash-preview:medium PASS 100% 112 in → 333 out (322 reasoning)
WIP: continue data WIP: adding data
google/gemini-3-flash-preview:medium__json_schema PASS 100% 111 in → 590 out (575 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3-flash-preview:xhigh PASS 100% 109 in → 457 out (446 reasoning)
WIP: continue data WIP: adding data
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 191 in → 413 out (398 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 113 in → 612 out (601 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 112 in → 860 out (845 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 111 in → 222 out (211 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 108 in → 421 out (391 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 113 in → 813 out (802 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 111 in → 826 out (809 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
google/gemini-3.1-pro-preview:high PASS 100% 112 in → 541 out (530 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 188 in → 651 out (636 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.1-pro-preview:low PASS 100% 114 in → 184 out (173 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 188 in → 332 out (317 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.1-pro-preview:medium PASS 100% 114 in → 569 out (558 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 189 in → 282 out (267 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 113 in → 581 out (570 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 190 in → 619 out (604 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.5-flash-lite:high PASS 100% 113 in → 1,559 out (1,548 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 190 in → 714 out (699 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 112 in → 17 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
google/gemini-3.5-flash-lite:max PASS 100% 111 in → 1,853 out (1,842 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 192 in → 849 out (834 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.5-flash-lite:medium PASS 100% 114 in → 695 out (684 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 113 in → 668 out (653 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 113 in → 1,125 out (1,114 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 193 in → 1,018 out (1,003 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.5-flash:high PASS 100% 113 in → 757 out (746 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.5-flash:high__json_schema PASS 100% 193 in → 757 out (742 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.5-flash:low PASS 100% 113 in → 431 out (420 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.5-flash:low__json_schema PASS 100% 189 in → 283 out (253 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
google/gemini-3.5-flash:medium PASS 100% 112 in → 500 out (489 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.5-flash:medium__json_schema PASS 100% 193 in → 718 out (703 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.5-flash:xhigh PASS 100% 113 in → 478 out (467 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 193 in → 558 out (543 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.6-flash:high PASS 100% 111 in → 1,288 out (1,277 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.6-flash:high__json_schema PASS 100% 193 in → 504 out (489 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.6-flash:low PASS 100% 113 in → 412 out (401 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.6-flash:max PASS 100% 110 in → 522 out (511 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.6-flash:max__json_schema PASS 100% 112 in → 981 out (966 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemini-3.6-flash:medium PASS 100% 113 in → 682 out (671 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.6-flash:medium__json_schema PASS 100% 189 in → 1,146 out (1,116 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
google/gemini-3.6-flash:xhigh PASS 100% 111 in → 808 out (797 reasoning)
WIP: continue data WIP: adding data
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 192 in → 1,354 out (1,339 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemma-4-26b-a4b-it:high PASS 100% 124 in → 589 out (493 reasoning)
WIP: continue data WIP: adding data
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 129 in → 540 out (425 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 129 in → 557 out (459 reasoning)
WIP: continue data WIP: adding data
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 129 in → 605 out (491 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 125 in → 395 out (330 reasoning)
WIP: continue data WIP: adding data
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 127 in → 1,232 out (1,011 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 122 in → 12 out (0 reasoning)
WIP: continue data WIP: adding data
google/gemma-4-26b-a4b-it:xhigh PASS 100% 126 in → 626 out (614 reasoning)
WIP: continue data WIP: adding data
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 127 in → 808 out (681 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
google/gemma-4-31b-it:high PASS 100% 128 in → 598 out (535 reasoning)
WIP: continue data WIP: adding data
google/gemma-4-31b-it:high__json_schema FAIL 100% 128 in → 3,428 out (3,191 reasoning)
WIP: continue data WIP: adding data Add initial file
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data", "Add initial file" ] }
Failure: Extra selected commit messages: ['Add initial file']
google/gemma-4-31b-it:low PASS 100% 127 in → 908 out (804 reasoning)
WIP: continue data WIP: adding data
google/gemma-4-31b-it:low__json_schema PASS 100% 125 in → 681 out (587 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
google/gemma-4-31b-it:medium PASS 100% 131 in → 919 out (907 reasoning)
WIP: continue data WIP: adding data
google/gemma-4-31b-it:medium__json_schema PASS 100% 130 in → 816 out (799 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
google/gemma-4-31b-it:none PASS 100% 125 in → 12 out (0 reasoning)
WIP: continue data WIP: adding data
google/gemma-4-31b-it:xhigh PASS 100% 129 in → 542 out (458 reasoning)
WIP: continue data WIP: adding data
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 129 in → 371 out (280 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
ibm-granite/granite-4.1-8b PASS 100% 116 in → 12 out
WIP: adding data WIP: continue data
ibm-granite/granite-4.1-8b__json_schema PASS 100% 117 in → 19 out
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
inclusionai/ling-2.6-flash PASS 100% 135 in → 12 out
WIP: adding data WIP: continue data
inclusionai/ling-2.6-flash__json_schema PASS 100% 133 in → 32 out
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: adding data", "WIP: continue data" ] }
minimax/minimax-m2.5:high PASS 100% 146 in → 801 out (788 reasoning)
WIP: continue data WIP: adding data
minimax/minimax-m2.5:high__json_schema PASS 100% 142 in → 368 out (357 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
minimax/minimax-m2.5:low PASS 100% 142 in → 715 out (793 reasoning)
WIP: continue data WIP: adding data
minimax/minimax-m2.5:low__json_schema PASS 100% 141 in → 1,226 out (1,357 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
minimax/minimax-m2.5:medium PASS 100% 143 in → 215 out (202 reasoning)
WIP: continue data WIP: adding data
minimax/minimax-m2.5:medium__json_schema PASS 100% 148 in → 470 out (443 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
minimax/minimax-m2.5:xhigh FAIL 100% 145 in → 323 out (309 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
minimax/minimax-m2.7:high PASS 100% 142 in → 219 out (207 reasoning)
WIP: continue data WIP: adding data
minimax/minimax-m2.7:low PASS 100% 140 in → 673 out (722 reasoning)
WIP: continue data WIP: adding data
minimax/minimax-m2.7:low__json_schema FAIL 100% 294 in → 1,692 out (1,669 reasoning)
WIP: continue data WIP: adding data Add initial file
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data","Add initial file"]}
Failure: Extra selected commit messages: ['Add initial file']
minimax/minimax-m2.7:medium PASS 100% 143 in → 483 out (513 reasoning)
WIP: continue data WIP: adding data
minimax/minimax-m2.7:medium__json_schema FAIL 100% 292 in → 1,356 out (1,333 reasoning)
WIP: continue data WIP: adding data Add initial file
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data","Add initial file"]}
Failure: Extra selected commit messages: ['Add initial file']
minimax/minimax-m2.7:xhigh FAIL 100% 143 in → 576 out (560 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
minimax/minimax-m3:high PASS 100% 281 in → 135 out (119 reasoning)
WIP: continue data WIP: adding data
minimax/minimax-m3:high__json_schema PASS 100% 268 in → 27 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
minimax/minimax-m3:low PASS 100% 283 in → 301 out (288 reasoning)
WIP: continue data WIP: adding data
minimax/minimax-m3:medium PASS 100% 230 in → 510 out (497 reasoning)
WIP: continue data WIP: adding data
minimax/minimax-m3:xhigh PASS 100% 279 in → 296 out (296 reasoning)
WIP: continue data WIP: adding data
mistralai/devstral-2512 PASS 100% 119 in → 12 out
WIP: continue data WIP: adding data
mistralai/devstral-2512__json_schema PASS 100% 118 in → 20 out
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
mistralai/mistral-medium-3-5:high PASS 100% 132 in → 394 out (353 reasoning)
WIP: continue data WIP: adding data
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 118 in → 424 out (384 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
mistralai/mistral-medium-3-5:low PASS 100% 131 in → 432 out (386 reasoning)
WIP: continue data WIP: adding data
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 121 in → 1,023 out (1,011 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
mistralai/mistral-medium-3-5:medium PASS 100% 133 in → 253 out (224 reasoning)
WIP: continue data WIP: adding data
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 121 in → 224 out (179 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 117 in → 20 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 128 in → 664 out (653 reasoning)
WIP: continue data WIP: adding data
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 121 in → 319 out (267 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
moonshotai/kimi-k2.6:high PASS 100% 113 in → 1,115 out (1,104 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k2.6:high__json_schema PASS 100% 111 in → 1,601 out (1,579 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue data", "WIP: adding data"] }
moonshotai/kimi-k2.6:low PASS 100% 115 in → 715 out (686 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k2.6:low__json_schema PASS 100% 109 in → 1,062 out (1,070 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
moonshotai/kimi-k2.6:medium PASS 100% 111 in → 1,057 out (1,174 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 112 in → 690 out (662 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
moonshotai/kimi-k2.6:xhigh PASS 100% 109 in → 1,122 out (1,109 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k2.7-code:high PASS 100% 110 in → 749 out (818 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 205 in → 236 out (216 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) {"commits": ["WIP: adding data", "WIP: continue data"]}
moonshotai/kimi-k2.7-code:low PASS 100% 112 in → 354 out (341 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 206 in → 330 out (312 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
moonshotai/kimi-k2.7-code:medium PASS 100% 109 in → 281 out (268 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k2.7-code:xhigh PASS 100% 114 in → 195 out (182 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 204 in → 210 out (192 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
moonshotai/kimi-k3:high PASS 100% 189 in → 175 out (149 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k3:high__json_schema PASS 100% 188 in → 510 out (484 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k3:low PASS 100% 190 in → 368 out (342 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k3:low__json_schema PASS 100% 189 in → 325 out (299 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k3:max PASS 100% 187 in → 198 out (172 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k3:max__json_schema PASS 100% 188 in → 425 out (399 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k3:medium PASS 100% 187 in → 289 out (263 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k3:medium__json_schema PASS 100% 193 in → 370 out (344 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k3:xhigh PASS 100% 191 in → 418 out (392 reasoning)
WIP: continue data WIP: adding data
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 186 in → 224 out (198 reasoning)
WIP: continue data WIP: adding data
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 133 in → 299 out (296 reasoning)
WIP: continue data WIP: adding data
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 130 in → 317 out (309 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 131 in → 217 out (197 reasoning)
WIP: continue data WIP: adding data
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 132 in → 349 out (336 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: adding data", "WIP: continue data" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 133 in → 304 out (277 reasoning)
WIP: continue data WIP: adding data
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 132 in → 389 out (365 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
nvidia/nemotron-3-nano-30b-a3b:none FAIL 100% 131 in → 16 out (0 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 100% 134 in → 27 out (0 reasoning)
WIP: adding data WIP: continue data Add complete data
JSON Schema Structured Output
(raw) {"commits": [ "WIP: adding data", "WIP: continue data", "Add complete data" ]}
Failure: Extra selected commit messages: ['Add complete data']
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 133 in → 282 out (258 reasoning)
WIP: continue data WIP: adding data
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 132 in → 218 out (191 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 134 in → 206 out (174 reasoning)
WIP: continue data WIP: adding data
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 134 in → 246 out (201 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 134 in → 243 out (214 reasoning)
WIP: continue data WIP: adding data
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 133 in → 205 out (167 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 134 in → 271 out (243 reasoning)
WIP: continue data WIP: adding data
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 131 in → 311 out (264 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 131 in → 27 out (0 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: adding data", "WIP: continue data" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 131 in → 188 out (165 reasoning)
WIP: continue data WIP: adding data
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 132 in → 218 out (175 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
openai/gpt-5.4-mini:high PASS 100% 112 in → 2,369 out (2,352 reasoning)
WIP: adding data WIP: continue data
openai/gpt-5.4-mini:high__json_schema PASS 100% 163 in → 294 out (267 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4-mini:low PASS 100% 114 in → 73 out (56 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4-mini:low__json_schema PASS 100% 160 in → 93 out (66 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4-mini:medium PASS 100% 107 in → 271 out (254 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4-mini:medium__json_schema PASS 100% 160 in → 103 out (76 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4-mini:none PASS 100% 108 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4-mini:none__json_schema PASS 100% 159 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4-mini:xhigh PASS 100% 110 in → 2,605 out (2,588 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 158 in → 543 out (516 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4-nano:high PASS 100% 112 in → 76 out (59 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4-nano:high__json_schema PASS 100% 159 in → 151 out (124 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4-nano:low PASS 100% 110 in → 115 out (98 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4-nano:low__json_schema PASS 100% 160 in → 80 out (53 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4-nano:medium PASS 100% 111 in → 119 out (102 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4-nano:medium__json_schema PASS 100% 160 in → 131 out (104 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4-nano:none__json_schema PASS 100% 160 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4-nano:xhigh PASS 100% 111 in → 488 out (471 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 159 in → 221 out (194 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4:high PASS 100% 113 in → 253 out (236 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4:high__json_schema PASS 100% 158 in → 207 out (180 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4:low PASS 100% 110 in → 82 out (65 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4:low__json_schema PASS 100% 159 in → 78 out (51 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4:medium PASS 100% 113 in → 210 out (193 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4:medium__json_schema PASS 100% 161 in → 183 out (156 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4:none PASS 100% 112 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4:none__json_schema PASS 100% 159 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.4:xhigh PASS 100% 112 in → 1,569 out (1,552 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.4:xhigh__json_schema PASS 100% 162 in → 543 out (516 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.5:high PASS 100% 109 in → 143 out (126 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.5:high__json_schema PASS 100% 160 in → 193 out (166 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.5:low PASS 100% 113 in → 72 out (55 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.5:low__json_schema PASS 100% 160 in → 60 out (33 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.5:medium PASS 100% 110 in → 83 out (66 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.5:medium__json_schema PASS 100% 159 in → 94 out (67 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.5:none PASS 100% 113 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.5:none__json_schema PASS 100% 158 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.5:xhigh PASS 100% 113 in → 238 out (221 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.5:xhigh__json_schema PASS 100% 159 in → 121 out (94 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-luna:high PASS 100% 114 in → 318 out (301 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-luna:high__json_schema PASS 100% 160 in → 377 out (350 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-luna:low PASS 100% 109 in → 94 out (77 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-luna:low__json_schema PASS 100% 160 in → 84 out (57 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-luna:max PASS 100% 111 in → 276 out (259 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-luna:max__json_schema PASS 100% 159 in → 221 out (194 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-luna:medium PASS 100% 113 in → 169 out (152 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-luna:medium__json_schema PASS 100% 160 in → 82 out (55 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-luna:none PASS 100% 111 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-luna:none__json_schema PASS 100% 160 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-luna:xhigh PASS 100% 110 in → 140 out (123 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 159 in → 127 out (100 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-sol:high PASS 100% 113 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-sol:high__json_schema PASS 100% 160 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-sol:low PASS 100% 109 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-sol:low__json_schema PASS 100% 160 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-sol:max PASS 100% 111 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-sol:max__json_schema PASS 100% 159 in → 388 out (361 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-sol:medium PASS 100% 111 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-sol:medium__json_schema PASS 100% 157 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-sol:none PASS 100% 110 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-sol:none__json_schema PASS 100% 159 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-sol:xhigh PASS 100% 111 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 159 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-terra:high PASS 100% 110 in → 62 out (45 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-terra:high__json_schema PASS 100% 158 in → 52 out (25 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-terra:low PASS 100% 111 in → 60 out (43 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-terra:low__json_schema PASS 100% 160 in → 45 out (18 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-terra:max PASS 100% 112 in → 165 out (148 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-terra:max__json_schema PASS 100% 157 in → 111 out (84 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-terra:medium PASS 100% 110 in → 45 out (28 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-terra:medium__json_schema PASS 100% 158 in → 45 out (18 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-terra:none PASS 100% 114 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-terra:none__json_schema PASS 100% 157 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-5.6-terra:xhigh PASS 100% 109 in → 40 out (23 reasoning)
WIP: continue data WIP: adding data
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 158 in → 47 out (20 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
openai/gpt-oss-120b:high PASS 100% 172 in → 470 out (490 reasoning)
WIP: continue data WIP: adding data
openai/gpt-oss-120b:high__json_schema PASS 100% 167 in → 408 out (430 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
openai/gpt-oss-120b:low PASS 100% 159 in → 115 out (96 reasoning)
WIP: continue data WIP: adding data
openai/gpt-oss-120b:low__json_schema PASS 100% 158 in → 91 out (55 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
openai/gpt-oss-120b:medium PASS 100% 172 in → 163 out (155 reasoning)
WIP: continue data WIP: adding data
openai/gpt-oss-120b:xhigh PASS 100% 171 in → 769 out (819 reasoning)
WIP: continue data WIP: adding data
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 173 in → 343 out (336 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
openai/gpt-oss-20b:high PASS 100% 174 in → 216 out (205 reasoning)
WIP: continue data WIP: adding data
openai/gpt-oss-20b:high__json_schema PASS 100% 161 in → 497 out (470 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":[ "WIP: continue data", "WIP: adding data" ] }
openai/gpt-oss-20b:low PASS 100% 172 in → 43 out (22 reasoning)
WIP: continue data WIP: adding data
openai/gpt-oss-20b:low__json_schema PASS 100% 168 in → 91 out (61 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
openai/gpt-oss-20b:medium PASS 100% 156 in → 227 out (210 reasoning)
WIP: continue data WIP: adding data
openai/gpt-oss-20b:medium__json_schema PASS 100% 174 in → 320 out (297 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) {"commits":["WIP: adding data","WIP: continue data"]}
openai/gpt-oss-20b:xhigh PASS 100% 172 in → 538 out (535 reasoning)
WIP: continue data WIP: adding data
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 169 in → 386 out (360 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":[ "WIP: continue data", "WIP: adding data" ]}
poolside/laguna-m.1:high__json_schema PASS 100% 125 in → 368 out (351 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
poolside/laguna-m.1:low PASS 100% 125 in → 348 out (336 reasoning)
WIP: continue data WIP: adding data
poolside/laguna-m.1:low__json_schema PASS 100% 124 in → 379 out (349 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
poolside/laguna-m.1:medium PASS 100% 123 in → 813 out (801 reasoning)
WIP: continue data WIP: adding data
poolside/laguna-m.1:medium__json_schema PASS 100% 122 in → 1,957 out (1,940 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
poolside/laguna-m.1:none FAIL 100% 124 in → 15 out (0 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
poolside/laguna-m.1:none__json_schema PASS 100% 124 in → 16 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
poolside/laguna-m.1:xhigh__json_schema PASS 100% 125 in → 995 out (965 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
poolside/laguna-xs-2.1:high PASS 100% 125 in → 250 out (240 reasoning)
WIP: continue data WIP: adding data
poolside/laguna-xs-2.1:high__json_schema PASS 100% 123 in → 524 out (506 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
poolside/laguna-xs-2.1:low PASS 100% 121 in → 369 out (359 reasoning)
WIP: continue data WIP: adding data
poolside/laguna-xs-2.1:low__json_schema PASS 100% 122 in → 1,029 out (1,007 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue data", "WIP: adding data"] }
poolside/laguna-xs-2.1:medium PASS 100% 121 in → 928 out (918 reasoning)
WIP: continue data WIP: adding data
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 121 in → 289 out (259 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
poolside/laguna-xs-2.1:none PASS 100% 118 in → 9 out (0 reasoning)
WIP: continue data WIP: adding data
poolside/laguna-xs-2.1:xhigh FAIL 100% 125 in → 794 out (780 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 125 in → 397 out (370 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
poolside/laguna-xs.2:high PASS 100% 156 in → 236 out (224 reasoning)
WIP: continue data WIP: adding data
poolside/laguna-xs.2:high__json_schema PASS 100% 159 in → 185 out (168 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
poolside/laguna-xs.2:low PASS 100% 159 in → 249 out (237 reasoning)
WIP: continue data WIP: adding data
poolside/laguna-xs.2:low__json_schema PASS 100% 162 in → 205 out (187 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
poolside/laguna-xs.2:medium PASS 100% 159 in → 329 out (317 reasoning)
WIP: continue data WIP: adding data
poolside/laguna-xs.2:medium__json_schema PASS 100% 163 in → 243 out (225 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
poolside/laguna-xs.2:none PASS 100% 159 in → 11 out (0 reasoning)
WIP: adding data WIP: continue data
poolside/laguna-xs.2:none__json_schema PASS 100% 157 in → 30 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
poolside/laguna-xs.2:xhigh PASS 100% 157 in → 229 out (217 reasoning)
WIP: continue data WIP: adding data
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 158 in → 288 out (270 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
qwen/qwen3.6-27b:high PASS 100% 123 in → 1,502 out (1,469 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-27b:high__json_schema PASS 100% 126 in → 1,745 out (1,639 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
qwen/qwen3.6-27b:low PASS 100% 125 in → 1,075 out (1,010 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-27b:low__json_schema PASS 100% 125 in → 1,146 out (1 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
qwen/qwen3.6-27b:medium PASS 100% 122 in → 1,255 out (1,215 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-27b:medium__json_schema PASS 100% 125 in → 1,594 out (1 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
qwen/qwen3.6-27b:none PASS 100% 121 in → 12 out (0 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-27b:xhigh PASS 100% 125 in → 1,319 out (1,297 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 127 in → 1,129 out (1,082 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 123 in → 1,164 out (1,148 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 126 in → 1,319 out (1,238 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 121 in → 1,420 out (1,404 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 124 in → 1,342 out (1,257 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 124 in → 1,249 out (1,189 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 125 in → 1,485 out (1,401 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
qwen/qwen3.6-35b-a3b:none FAIL 100% 124 in → 16 out (0 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 100% 126 in → 39 out (0 reasoning)
Add initial file WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) { "commits": [ "Add initial file", "WIP: adding data", "WIP: continue data" ] }
Failure: Extra selected commit messages: ['Add initial file']
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 127 in → 1,386 out (1,322 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 124 in → 2,579 out (2,524 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
qwen/qwen3.6-flash:high FAIL 100% 124 in → 2,172 out (2,152 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
qwen/qwen3.6-flash:low PASS 100% 125 in → 1,479 out (1,463 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-flash:medium PASS 100% 124 in → 1,582 out (1,566 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-flash:none PASS 100% 125 in → 11 out (0 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.6-flash:xhigh FAIL 100% 125 in → 2,164 out (2,144 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
qwen/qwen3.7-flash:high FAIL 100% 124 in → 376 out (356 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
qwen/qwen3.7-flash:low FAIL 100% 124 in → 1,137 out (1,117 reasoning)
WIP: continue data WIP: adding data Add initial file
Failure: Extra selected commit messages: ['Add initial file']
qwen/qwen3.7-flash:max PASS 100% 124 in → 1,183 out (1,167 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-flash:medium PASS 100% 121 in → 1,417 out (1,401 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-flash:none PASS 100% 126 in → 11 out (0 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-flash:xhigh PASS 100% 123 in → 2,376 out (2,360 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-max:high PASS 100% 123 in → 371 out (355 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-max:low PASS 100% 125 in → 355 out (339 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-max:medium PASS 100% 122 in → 303 out (287 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-max:none PASS 100% 126 in → 11 out (0 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-max:xhigh PASS 100% 121 in → 268 out (252 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-plus:high PASS 100% 123 in → 2,703 out (2,687 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-plus:low PASS 100% 125 in → 1,564 out (1,548 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-plus:medium PASS 100% 126 in → 1,074 out (1,058 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-plus:none PASS 100% 123 in → 11 out (0 reasoning)
WIP: continue data WIP: adding data
qwen/qwen3.7-plus:xhigh PASS 100% 125 in → 1,027 out (1,011 reasoning)
WIP: adding data WIP: continue data
tencent/hy3:high PASS 100% 117 in → 859 out (846 reasoning)
WIP: continue data WIP: adding data
tencent/hy3:high__json_schema PASS 100% 119 in → 942 out (914 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
tencent/hy3:low PASS 100% 119 in → 552 out (539 reasoning)
WIP: continue data WIP: adding data
tencent/hy3:low__json_schema PASS 100% 119 in → 566 out (551 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue data", "WIP: adding data"] }
tencent/hy3:medium PASS 100% 119 in → 625 out (612 reasoning)
WIP: continue data WIP: adding data
tencent/hy3:medium__json_schema PASS 100% 117 in → 737 out (683 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
tencent/hy3:none PASS 100% 125 in → 12 out (0 reasoning)
WIP: continue data WIP: adding data
tencent/hy3:none__json_schema PASS 100% 123 in → 25 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
tencent/hy3:xhigh PASS 100% 120 in → 877 out (864 reasoning)
WIP: continue data WIP: adding data
tencent/hy3:xhigh__json_schema PASS 100% 118 in → 1,378 out (1,363 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
thinkingmachines/inkling-small:high PASS 100% 121 in → 215 out (205 reasoning)
WIP: continue data WIP: adding data
thinkingmachines/inkling-small:high__json_schema PASS 100% 121 in → 318 out (294 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) {"commits": ["WIP: adding data", "WIP: continue data"]}
thinkingmachines/inkling-small:low PASS 100% 119 in → 62 out (33 reasoning)
WIP: continue data WIP: adding data
thinkingmachines/inkling-small:low__json_schema PASS 100% 118 in → 68 out (37 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
thinkingmachines/inkling-small:max PASS 100% 119 in → 282 out (265 reasoning)
WIP: adding data WIP: continue data
thinkingmachines/inkling-small:max__json_schema PASS 100% 118 in → 416 out (391 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue data", "WIP: adding data"] }
thinkingmachines/inkling-small:medium PASS 100% 119 in → 178 out (161 reasoning)
WIP: continue data WIP: adding data
thinkingmachines/inkling-small:medium__json_schema PASS 100% 120 in → 269 out (238 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 120 in → 403 out (386 reasoning)
WIP: continue data WIP: adding data
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 121 in → 239 out (208 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
thinkingmachines/inkling:high PASS 100% 125 in → 274 out (261 reasoning)
WIP: adding data WIP: continue data
thinkingmachines/inkling:low PASS 100% 119 in → 38 out (21 reasoning)
WIP: continue data WIP: adding data
thinkingmachines/inkling:low__json_schema PASS 100% 120 in → 45 out (21 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
thinkingmachines/inkling:max PASS 100% 117 in → 200 out (182 reasoning)
WIP: continue data WIP: adding data
thinkingmachines/inkling:max__json_schema PASS 100% 117 in → 244 out (221 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":[ "WIP: continue data", "WIP: adding data" ]}
thinkingmachines/inkling:medium PASS 100% 121 in → 176 out (159 reasoning)
WIP: continue data WIP: adding data
thinkingmachines/inkling:medium__json_schema PASS 100% 121 in → 143 out (112 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
thinkingmachines/inkling:xhigh PASS 100% 117 in → 312 out (294 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.3:high PASS 100% 284 in → 462 out (451 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.3:high__json_schema PASS 100% 350 in → 892 out (867 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
x-ai/grok-4.3:low PASS 100% 291 in → 352 out (341 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.3:low__json_schema PASS 100% 361 in → 440 out (415 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
x-ai/grok-4.3:max PASS 100% 288 in → 337 out (326 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.3:max__json_schema PASS 100% 351 in → 690 out (665 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
x-ai/grok-4.3:medium PASS 100% 290 in → 543 out (532 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.3:medium__json_schema PASS 100% 359 in → 691 out (666 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
x-ai/grok-4.3:xhigh PASS 100% 285 in → 235 out (224 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.3:xhigh__json_schema PASS 100% 350 in → 1,120 out (1,105 reasoning)
WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) {"commits":["WIP: adding data","WIP: continue data"]}
x-ai/grok-4.5:high PASS 100% 305 in → 275 out (266 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.5:high__json_schema PASS 100% 393 in → 497 out (473 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
x-ai/grok-4.5:low PASS 100% 309 in → 212 out (203 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.5:low__json_schema PASS 100% 386 in → 316 out (292 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
x-ai/grok-4.5:max PASS 100% 309 in → 341 out (332 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.5:max__json_schema PASS 100% 390 in → 689 out (675 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
x-ai/grok-4.5:medium PASS 100% 308 in → 423 out (414 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.5:medium__json_schema PASS 100% 391 in → 421 out (407 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
x-ai/grok-4.5:xhigh PASS 100% 307 in → 393 out (384 reasoning)
WIP: continue data WIP: adding data
x-ai/grok-4.5:xhigh__json_schema PASS 100% 387 in → 529 out (515 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["WIP: continue data","WIP: adding data"]}
z-ai/glm-4.7-flash:high PASS 100% 115 in → 457 out (415 reasoning)
WIP: continue data WIP: adding data
z-ai/glm-4.7-flash:low__json_schema PASS 100% 114 in → 1,139 out (1,043 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
z-ai/glm-4.7-flash:medium PASS 100% 113 in → 675 out (584 reasoning)
WIP: continue data WIP: adding data
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 113 in → 522 out (440 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
z-ai/glm-4.7-flash:none PASS 100% 113 in → 12 out (0 reasoning)
WIP: continue data WIP: adding data
z-ai/glm-4.7-flash:none__json_schema PASS 100% 114 in → 27 out (0 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue data", "WIP: adding data" ] }
z-ai/glm-4.7-flash:xhigh PASS 100% 113 in → 429 out (394 reasoning)
WIP: continue data WIP: adding data
z-ai/glm-5.2:high PASS 100% 121 in → 639 out (626 reasoning)
WIP: continue data WIP: adding data
z-ai/glm-5.2:low PASS 100% 122 in → 453 out (427 reasoning)
WIP: continue data WIP: adding data
z-ai/glm-5.2:low__json_schema PASS 100% 116 in → 2,719 out (2,699 reasoning)
WIP: continue data WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue data", "WIP: adding data"]}
z-ai/glm-5.2:medium PASS 100% 119 in → 597 out (575 reasoning)
WIP: continue data WIP: adding data
z-ai/glm-5.2:none PASS 100% 111 in → 12 out (0 reasoning)
WIP: adding data WIP: continue data
z-ai/glm-5.2:xhigh PASS 100% 119 in → 1,225 out (1,165 reasoning)
WIP: continue data WIP: adding data
z-ai/glm-5.2:xhigh__json_schema FAIL 100% 118 in → 3,265 out (3,259 reasoning)
Add initial file WIP: adding data WIP: continue data
JSON Schema Structured Output
(raw) { "commits": [ "Add initial file", "WIP: adding data", "WIP: continue data" ] }
Failure: Extra selected commit messages: ['Add initial file']
arcee-ai/trinity-mini:low FAIL 50% 111 in → 338 out (359 reasoning)
WIP: continuedata WIP: adding data
Failure: Missing expected commit messages: ['WIP: continue data']; Extra selected commit messages: ['WIP: continuedata']
arcee-ai/trinity-mini:xhigh FAIL 50% 107 in → 314 out (325 reasoning)
WIP: continue data WIP:adding data
Failure: Missing expected commit messages: ['WIP: adding data']; Extra selected commit messages: ['WIP:adding data']
thinkingmachines/inkling-small:none FAIL 50% 120 in → 14 out (0 reasoning)
WIP: adding data WIP: continue data<|end_message|>
Failure: Missing expected commit messages: ['WIP: continue data']; Extra selected commit messages: ['WIP: continue data']
thinkingmachines/inkling:none FAIL 50% 122 in → 14 out (0 reasoning)
WIP: continue data WIP: adding data<|end_message|>
Failure: Missing expected commit messages: ['WIP: adding data']; Extra selected commit messages: ['WIP: adding data']
anthropic/claude-haiku-4.5:none FAIL 0% 115 in → 22 out (0 reasoning)
eb49bea WIP: continue data 1f821ec WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 306 in → 33 out (0 reasoning)
9cb41b0 WIP: continue data 82d4b93 WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["9cb41b0 WIP: continue data", "82d4b93 WIP: adding data"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 309 in → 20 out (0 reasoning)
8d5725d 2b99bff
JSON Schema Structured Output
(raw) {"commits":["8d5725d","2b99bff"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-opus-4.6:low__json_schema FAIL 0% 306 in → 21 out (0 reasoning)
c75b3c5 31d97f3
JSON Schema Structured Output
(raw) {"commits":["c75b3c5","31d97f3"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: adding data', 'WIP: continue data']
anthropic/claude-opus-4.6:medium__json_schema FAIL 0% 307 in → 20 out (0 reasoning)
b3beb03 c53dcb1
JSON Schema Structured Output
(raw) {"commits":["b3beb03","c53dcb1"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: adding data', 'WIP: continue data']
anthropic/claude-opus-4.6:none__json_schema FAIL 0% 307 in → 19 out (0 reasoning)
3cf29f5 f58fc23
JSON Schema Structured Output
(raw) {"commits":["3cf29f5","f58fc23"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 307 in → 21 out (0 reasoning)
5a1cfe9 4282f3b
JSON Schema Structured Output
(raw) {"commits":["5a1cfe9","4282f3b"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-opus-4.7:high FAIL 0% 160 in → 32 out (0 reasoning)
0f6a944 WIP: continue data 798c1b2 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-opus-4.7:none FAIL 0% 160 in → 30 out (0 reasoning)
aa32be0 WIP: continue data 99453f1 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-opus-4.7:xhigh FAIL 0% 158 in → 29 out (0 reasoning)
ab123fd WIP: continue data 18a9434 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-opus-4.8:none__json_schema FAIL 0% 400 in → 41 out (0 reasoning)
9c398f6 WIP: continue data 6b4ccb6 WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["9c398f6 WIP: continue data", "6b4ccb6 WIP: adding data"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-sonnet-5:high FAIL 0% 154 in → 54 out (25 reasoning)
7686aab WIP: continue data 8ee54af WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 400 in → 40 out (0 reasoning)
1d5b3b2 WIP: continue data b8b7057 WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["1d5b3b2 WIP: continue data","b8b7057 WIP: adding data"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 396 in → 38 out (0 reasoning)
cbb8e75 WIP: continue data f77a915 WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["cbb8e75 WIP: continue data","f77a915 WIP: adding data"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
anthropic/claude-sonnet-5:none FAIL 0% 157 in → 31 out (0 reasoning)
``` 80fec42 WIP: continue data 63412a4 WIP: adding data ```
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['```', 'WIP: continue data', 'WIP: adding data', '```']
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 393 in → 36 out (0 reasoning)
ecdef87 WIP: continue data 76a2385 WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["ecdef87 WIP: continue data","76a2385 WIP: adding data"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh FAIL 0% 110 in → 2,272 out (2,364 reasoning)
532b3ec WIP: continue data 4073ed4 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high__json_schema FAIL 0% 110 in → 369 out (330 reasoning)
1952032 Add complete data a5b24ea WIP: continue data 06a68e5 WIP: adding data 6981edf Add initial file
JSON Schema Structured Output
(raw) { "commits": [ "1952032 Add complete data", "a5b24ea WIP: continue data", "06a68e5 WIP: adding data", "6981edf Add initial file" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['Add complete data', 'WIP: continue data', 'WIP: adding data', 'Add initial file']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 108 in → 378 out (357 reasoning)
41e1798 Add complete data 590806a WIP: continue data 24821cc WIP: adding data 9fdbd14 Add initial file
JSON Schema Structured Output
(raw) { "commits": [ "41e1798 Add complete data", "590806a WIP: continue data", "24821cc WIP: adding data", "9fdbd14 Add initial file" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['Add complete data', 'WIP: continue data', 'WIP: adding data', 'Add initial file']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 111 in → 357 out (329 reasoning)
34afe4b Add complete data 146e2d5 WIP: continue data 0139c44 WIP: adding data ed90e52 Add initial file
JSON Schema Structured Output
(raw) { "commits": [ "34afe4b Add complete data", "146e2d5 WIP: continue data", "0139c44 WIP: adding data", "ed90e52 Add initial file" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['Add complete data', 'WIP: continue data', 'WIP: adding data', 'Add initial file']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 110 in → 754 out (780 reasoning)
1b1ca12 Add complete data ca070a8 WIP: continue data 5cc970f WIP: adding data 7149890 Add initial file
JSON Schema Structured Output
(raw) { "commits": [ "1b1ca12 Add complete data", "ca070a8 WIP: continue data", "5cc970f WIP: adding data", "7149890 Add initial file" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['Add complete data', 'WIP: continue data', 'WIP: adding data', 'Add initial file']
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 0% 108 in → 68 out (42 reasoning)
6d158c0 WIP: continue data ef52a62 WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "6d158c0 WIP: continue data", "ef52a62 WIP: adding data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
deepseek/deepseek-v4-flash-0731:medium FAIL 0% 189 in → 271 out (251 reasoning)
8deb613 WIP: continue data 14fe0b2 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 109 in → 19 out (0 reasoning)
4fe1346 WIP: continue data 3900de0 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 109 in → 77 out (66 reasoning)
06704c5 WIP: continue data 090d575 WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["06704c5 WIP: continue data", "090d575 WIP: adding data"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
Invalid JSON. Output: 7c4a63e WIP: continue data 53e247e WIP: adding data 3bc658e Add initial file
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
deepseek/deepseek-v4-flash:low__json_schema FAIL 0% 110 in → 257 out (221 reasoning)
dbfaa05 WIP: adding data e64ec13 WIP: continue data
JSON Schema Structured Output
(raw) { "commits": [ "dbfaa05 WIP: adding data", "e64ec13 WIP: continue data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: adding data', 'WIP: continue data']
deepseek/deepseek-v4-flash:none FAIL 0% 108 in → 19 out (0 reasoning)
85ac316 WIP: adding data 1084b53 WIP: continue data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: adding data', 'WIP: continue data']
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 111 in → 40 out (0 reasoning)
0e6ab4a Add initial file 6853f07 WIP: adding data d63dd30 WIP: continue data
JSON Schema Structured Output
(raw) { "commits": ["0e6ab4a Add initial file", "6853f07 WIP: adding data", "d63dd30 WIP: continue data"] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['Add initial file', 'WIP: adding data', 'WIP: continue data']
deepseek/deepseek-v4-flash:xhigh FAIL 0% 192 in → 766 out (793 reasoning)
8f9a6ca WIP: continue data ca2f514 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
Invalid JSON. Output: WIP: continue data WIP: adding data
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 111 in → 386 out (349 reasoning)
8306a7c WIP: adding data 1c796ba WIP: continue data
JSON Schema Structured Output
(raw) { "commits": [ "8306a7c WIP: adding data", "1c796ba WIP: continue data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: adding data', 'WIP: continue data']
deepseek/deepseek-v4-pro:none FAIL 0% 109 in → 19 out (0 reasoning)
4eacefa WIP: continue data e214f97 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 313 in → 36 out (0 reasoning)
0e5acef WIP: continue data 2a035ba WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "0e5acef WIP: continue data", "2a035ba WIP: adding data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 388 in → 1,165 out (1,136 reasoning)
e93ec6d WIP: continue data 7354597 WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["e93ec6d WIP: continue data", "7354597 WIP: adding data"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
google/gemini-3.1-flash-lite-preview:low FAIL 0% 111 in → 139 out (117 reasoning)
18633dd WIP: continue data 1827b40 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 0% 190 in → 160 out (119 reasoning)
db77874 WIP: continue data d327c24 WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "db77874 WIP: continue data", "d327c24 WIP: adding data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
google/gemini-3.5-flash-lite:low FAIL 0% 113 in → 22 out (0 reasoning)
f7ff17e WIP: continue data 81a5f0e WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
google/gemini-3.6-flash:low__json_schema FAIL 0% 192 in → 41 out (0 reasoning)
c2b284f WIP: continue data 9b0ca0d WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "c2b284f WIP: continue data", "9b0ca0d WIP: adding data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
google/gemma-4-26b-a4b-it:none__json_schema FAIL 0% 125 in → 30 out (0 reasoning)
d01f103 WIP: continue data f95ab1a WIP: adding data
JSON Schema Structured Output
(raw) {"commits": ["d01f103 WIP: continue data", "f95ab1a WIP: adding data"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
google/gemma-4-31b-it:none__json_schema FAIL 0% 125 in → 42 out (0 reasoning)
37e45b1 WIP: continue data bc3c404 WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "37e45b1 WIP: continue data", "bc3c404 WIP: adding data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
liquid/lfm-2-24b-a2b FAIL 0% 121 in → 22 out
6d5d569 WIP: continue data 3b8b288 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
Invalid JSON. Output: 614bdb9 Add complete data ef3b6e5 WIP: continuing data 5452ec8 WIP: adding data
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 4 (char 3)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 4 (char 3)
minimax/minimax-m2.5:xhigh__json_schema FAIL 0% 150 in → 217 out (173 reasoning)
1f5bc42 WIP: continue data 1c8bf3e WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "1f5bc42 WIP: continue data", "1c8bf3e WIP: adding data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
minimax/minimax-m2.7:high__json_schema FAIL 0% 142 in → 56 out (56 reasoning)
afa8a82 Add complete data 37c1260 WIP: continue data ba97a73 WIP: adding data 9d9430c Add initial file
JSON Schema Structured Output
(raw) { "commits": [ "afa8a82 Add complete data", "37c1260 WIP: continue data", "ba97a73 WIP: adding data", "9d9430c Add initial file" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['Add complete data', 'WIP: continue data', 'WIP: adding data', 'Add initial file']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:low__json_schema FAIL 0% 269 in → 19 out (0 reasoning)
45ebb70 82c5a0a
JSON Schema Structured Output
(raw) {"commits": ["45ebb70", "82c5a0a"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
minimax/minimax-m3:medium__json_schema FAIL 0% 269 in → 20 out (0 reasoning)
4acfece 48e7b5c
JSON Schema Structured Output
(raw) {"commits": ["4acfece", "48e7b5c"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
minimax/minimax-m3:xhigh__json_schema FAIL 0% 229 in → 48 out (47 reasoning)
1a0c13c Add complete data 9ecccad WIP: continue data c7733ed WIP: adding data 6820e52 Add initial file
JSON Schema Structured Output
(raw) {"commits": ["1a0c13c Add complete data", "9ecccad WIP: continue data", "c7733ed WIP: adding data", "6820e52 Add initial file"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['Add complete data', 'WIP: continue data', 'WIP: adding data', 'Add initial file']
mistralai/mistral-medium-3-5:none FAIL 0% 129 in → 25 out (0 reasoning)
06fda32 WIP: adding data 4c57d28 WIP: continue data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: adding data', 'WIP: continue data']
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0% 114 in → 58 out (0 reasoning)
1d88712 Add complete data 5eeaf1c WIP: continue data 5c1cef7 WIP: adding data 20cc9b0 Add initial file
JSON Schema Structured Output
(raw) { "commits": [ "1d88712 Add complete data", "5eeaf1c WIP: continue data", "5c1cef7 WIP: adding data", "20cc9b0 Add initial file" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['Add complete data', 'WIP: continue data', 'WIP: adding data', 'Add initial file']
moonshotai/kimi-k2.7-code:medium__json_schema FAIL 0% 112 in → 171 out (165 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "commits": [] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']
nvidia/nemotron-3-super-120b-a12b:none FAIL 0% 133 in → 24 out (0 reasoning)
bac37d0 WIP: continue data 8013da3 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
openai/gpt-5.4-nano:none FAIL 0% 113 in → 35 out (0 reasoning)
860a8b3 WIP: continue data 40a4ce9 WIP: adding data e9b34e8 Add initial file
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data', 'Add initial file']
openai/gpt-oss-120b:medium__json_schema FAIL 0% 168 in → 32 out (0 reasoning)
fdad278 WIP: continue data 4852990 WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "fdad278 WIP: continue data", "4852990 WIP: adding data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
poolside/laguna-m.1:high FAIL 0% 120 in → 736 out (704 reasoning)
ed23acf WIP: continue data 4488fd9 WIP: adding data d56d5dc Add initial file
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data', 'Add initial file']
poolside/laguna-m.1:xhigh FAIL 0% 126 in → 629 out (603 reasoning)
d10d339 WIP: continue data f0419b8 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 123 in → 57 out (0 reasoning)
be1851e Add complete data ac967ea WIP: continue data c39523d WIP: adding data f278967 Add initial file
JSON Schema Structured Output
(raw) { "commits": ["be1851e Add complete data", "ac967ea WIP: continue data", "c39523d WIP: adding data", "f278967 Add initial file"] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['Add complete data', 'WIP: continue data', 'WIP: adding data', 'Add initial file']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 128 in → 31 out (0 reasoning)
5c8d6a3 2581cb8
JSON Schema Structured Output
(raw) { "commits": [ "5c8d6a3", "2581cb8" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-aa541130-fb0f-9aed-ad02-08d3d9f85685","request_id":"aa541130-fb0f-9aed-ad02-08d3d9f85685"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-29cb9334-a359-9772-8b6b-e330bc502b6a","request_id":"29cb9334-a359-9772-8b6b-e330bc502b6a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f144f1f3-d123-9b9d-82c5-8ecef2f29903","request_id":"f144f1f3-d123-9b9d-82c5-8ecef2f29903"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-719915d0-2853-9421-ad1c-9fbbde317bef","request_id":"719915d0-2853-9421-ad1c-9fbbde317bef"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-415f05dc-0b30-963c-bbab-b7bee090f0ce","request_id":"415f05dc-0b30-963c-bbab-b7bee090f0ce"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-eac53272-6518-9208-b568-3fe993f734ec"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0ce9b1de-31c2-9b53-9ee4-9c713ed19ee3"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-133afaee-ec3d-986b-a72b-cffa248ad8d1"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-16ef5aa9-ac41-9b7e-84ef-697046bcdbe3"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ef5df4d9-8ca5-945d-b9db-a59aa7adf854"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-600f415c-ad94-94d2-991e-dff54103e889"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4c200e8e-20a9-90f3-8d97-d6b9caae237a","request_id":"4c200e8e-20a9-90f3-8d97-d6b9caae237a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c6c782fd-e639-9b2d-9bb7-038c0cb43cd6","request_id":"c6c782fd-e639-9b2d-9bb7-038c0cb43cd6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4ccbb0fb-8390-9aa5-825a-b6ce8fdc1d65","request_id":"4ccbb0fb-8390-9aa5-825a-b6ce8fdc1d65"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7c385a7d-e557-9dc4-8807-a0a068ba384d","request_id":"7c385a7d-e557-9dc4-8807-a0a068ba384d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fb2e128c-cc4e-9904-a3c3-eb0f81be1cb5","request_id":"fb2e128c-cc4e-9904-a3c3-eb0f81be1cb5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0c989a6f-27a8-91ae-93e2-b90ce56923b6","request_id":"0c989a6f-27a8-91ae-93e2-b90ce56923b6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bd8386e8-ccaa-9422-8b75-d2d2cd36fd49","request_id":"bd8386e8-ccaa-9422-8b75-d2d2cd36fd49"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-73469e98-a72d-9e0e-b45d-831cfb627895","request_id":"73469e98-a72d-9e0e-b45d-831cfb627895"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9aedf0cf-d9a2-9bfe-8e82-e7f7998d24fe","request_id":"9aedf0cf-d9a2-9bfe-8e82-e7f7998d24fe"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ae62835b-b4d0-9d31-aacd-a531266e6391","request_id":"ae62835b-b4d0-9d31-aacd-a531266e6391"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: WIP: continue data WIP: adding data{"commits": ["WIP: continue data", "WIP: adding data"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue data WIP: adding data
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue data WIP: adding data{ "commits": [ "WIP: continue data", "WIP: adding data" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue data WIP: adding data
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none FAIL 0% 285 in → 17 out (0 reasoning)
8c13142 WIP: continue data e33008f WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
x-ai/grok-4.3:none__json_schema FAIL 0% 352 in → 23 out (0 reasoning)
750fe9b WIP: continue data e2db4a0 WIP: adding data
JSON Schema Structured Output
(raw) {"commits":["750fe9b WIP: continue data","e2db4a0 WIP: adding data"]}
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low FAIL 0% 109 in → 2,311 out (2,262 reasoning)
f06fafa WIP: continue data 3704986 WIP: adding data
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue data WIP: adding data
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:medium__json_schema FAIL 0% 120 in → 714 out (637 reasoning)
1995112 WIP: continue data 5333cd0 WIP: adding data
JSON Schema Structured Output
(raw) { "commits": [ "1995112 WIP: continue data", "5333cd0 WIP: adding data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['WIP: continue data', 'WIP: adding data']
z-ai/glm-5.2:none__json_schema FAIL 0% 114 in → 41 out (0 reasoning)
e580732 Add initial file a362fea WIP: adding data 9cf5977 WIP: continue data
JSON Schema Structured Output
(raw) { "commits": [ "e580732 Add initial file", "a362fea WIP: adding data", "9cf5977 WIP: continue data" ] }
Failure: Missing expected commit messages: ['WIP: adding data', 'WIP: continue data']; Extra selected commit messages: ['Add initial file', 'WIP: adding data', 'WIP: continue data']