Related changes split across commits
Tests ability to identify related changes split across multiple commits that should be combined. Evaluates recognizing atomic-commit boundaries.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo '// v1' > app.js
  5. 05 git add app.js
  6. 06 git commit -m 'Add app.js v1'
  7. 07 echo '// v2' > app.js
  8. 08 git add app.js
  9. 09 git commit -m 'WIP: update app.js'
  10. 10 echo '// v3' > app.js
  11. 11 git add app.js
  12. 12 git commit -m 'WIP: more updates'
  13. 13 echo '// final' > app.js
  14. 14 git add app.js
  15. 15 git commit -m 'Finalize app.js'
Prompt
Analyze this git history. Multiple consecutive WIP commits updating the same file suggest the changes should be combined. Identify the WIP commits that should be marked as squash/fixup in an interactive rebase. Do not include the initial or final anchor commit. Output ONLY the selected commit subject lines, one per line, with no explanation.
Expected
WIP: update app.js WIP: more updates

Scoped model quality, cost, API time, and token usage for commit_squash/f003.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 178 in → 50 out (27 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-fable-5:high__json_schema PASS 100% 420 in → 59 out (33 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
anthropic/claude-fable-5:low PASS 100% 180 in → 18 out (0 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-fable-5:low__json_schema PASS 100% 422 in → 29 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
anthropic/claude-fable-5:max PASS 100% 176 in → 279 out (80 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-fable-5:max__json_schema PASS 100% 419 in → 412 out (72 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
anthropic/claude-fable-5:medium PASS 100% 179 in → 29 out (12 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-fable-5:medium__json_schema PASS 100% 418 in → 29 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
anthropic/claude-fable-5:xhigh PASS 100% 183 in → 110 out (35 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 420 in → 87 out (25 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 353 in → 884 out (627 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits": ["WIP: update app.js", "WIP: more updates"]}
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 352 in → 2,801 out (1,072 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits": ["WIP: update app.js", "WIP: more updates"]}
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 352 in → 869 out (538 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits": ["WIP: update app.js", "WIP: more updates"]}
anthropic/claude-haiku-4.5:none PASS 100% 134 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-haiku-4.5:xhigh PASS 100% 164 in → 708 out (570 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 351 in → 942 out (539 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits": ["WIP: update app.js", "WIP: more updates"]}
anthropic/claude-opus-4.6:high PASS 100% 134 in → 130 out (70 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-opus-4.6:high__json_schema PASS 100% 320 in → 160 out (101 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
anthropic/claude-opus-4.6:low PASS 100% 135 in → 16 out (0 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-opus-4.6:low__json_schema PASS 100% 320 in → 22 out (0 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
anthropic/claude-opus-4.6:max PASS 100% 134 in → 191 out (122 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-opus-4.6:max__json_schema PASS 100% 322 in → 217 out (138 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
anthropic/claude-opus-4.6:medium PASS 100% 138 in → 16 out (0 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 323 in → 22 out (0 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
anthropic/claude-opus-4.6:none PASS 100% 133 in → 16 out (0 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-opus-4.6:none__json_schema PASS 100% 321 in → 22 out (0 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 133 in → 174 out (113 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 322 in → 149 out (88 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
anthropic/claude-opus-4.7:high PASS 100% 187 in → 20 out (0 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.7:high__json_schema PASS 100% 425 in → 29 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-opus-4.7:low PASS 100% 183 in → 20 out (0 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.7:low__json_schema PASS 100% 423 in → 29 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-opus-4.7:max PASS 100% 184 in → 249 out (49 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.7:max__json_schema PASS 100% 426 in → 29 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-opus-4.7:medium PASS 100% 185 in → 20 out (0 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 427 in → 29 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-opus-4.7:none PASS 100% 185 in → 20 out (0 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.7:none__json_schema PASS 100% 422 in → 29 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 185 in → 20 out (0 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 427 in → 29 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-opus-4.8:high PASS 100% 182 in → 89 out (30 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.8:high__json_schema PASS 100% 421 in → 27 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-opus-4.8:low PASS 100% 177 in → 71 out (45 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.8:low__json_schema PASS 100% 418 in → 27 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-opus-4.8:max PASS 100% 175 in → 791 out (112 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-opus-4.8:max__json_schema PASS 100% 418 in → 365 out (83 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
anthropic/claude-opus-4.8:medium PASS 100% 180 in → 74 out (38 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 419 in → 27 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-opus-4.8:none PASS 100% 180 in → 18 out (0 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.8:none__json_schema PASS 100% 418 in → 26 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 181 in → 194 out (61 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 420 in → 27 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-sonnet-4.6:high PASS 100% 131 in → 249 out (157 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 324 in → 180 out (110 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 320 in → 22 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-sonnet-4.6:max PASS 100% 134 in → 280 out (177 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 323 in → 263 out (173 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 138 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 319 in → 22 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-sonnet-4.6:none PASS 100% 135 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 323 in → 22 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 134 in → 252 out (174 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 319 in → 303 out (297 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-sonnet-5:high PASS 100% 181 in → 50 out (25 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-sonnet-5:low PASS 100% 183 in → 18 out (0 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-sonnet-5:low__json_schema PASS 100% 417 in → 27 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-sonnet-5:medium PASS 100% 180 in → 38 out (20 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 420 in → 27 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
anthropic/claude-sonnet-5:none PASS 100% 185 in → 18 out (0 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-sonnet-5:xhigh PASS 100% 182 in → 139 out (48 reasoning)
WIP: update app.js WIP: more updates
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 418 in → 72 out (32 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
arcee-ai/trinity-large-thinking:high PASS 100% 118 in → 1,568 out (1,643 reasoning)
WIP: more updates WIP: update app.js
arcee-ai/trinity-large-thinking:low PASS 100% 122 in → 1,143 out (1,197 reasoning)
WIP: more updates WIP: update app.js
arcee-ai/trinity-large-thinking:medium PASS 100% 119 in → 2,535 out (2,633 reasoning)
WIP: more updates WIP: update app.js
arcee-ai/trinity-mini:low PASS 100% 122 in → 467 out (499 reasoning)
WIP: more updates WIP: update app.js
arcee-ai/trinity-mini:xhigh PASS 100% 120 in → 423 out (432 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-flash-0731:high PASS 100% 120 in → 432 out (425 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-flash-0731:low PASS 100% 120 in → 359 out (346 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-flash-0731:max PASS 100% 202 in → 6,030 out (5,919 reasoning)
WIP: update app.js WIP: more updates
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 198 in → 211 out (195 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits":["WIP: more updates","WIP: update app.js"] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 121 in → 351 out (320 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 202 in → 21 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 203 in → 7,758 out (7,745 reasoning)
WIP: update app.js WIP: more updates
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 125 in → 640 out (624 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
deepseek/deepseek-v4-flash:high PASS 100% 124 in → 443 out (429 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-flash:low PASS 100% 121 in → 396 out (382 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-flash:medium PASS 100% 119 in → 570 out (556 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-flash:none PASS 100% 119 in → 12 out (0 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 119 in → 22 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": ["WIP: more updates", "WIP: update app.js"] }
deepseek/deepseek-v4-flash:xhigh PASS 100% 199 in → 602 out (588 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-pro:high PASS 100% 120 in → 275 out (261 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-pro:low PASS 100% 123 in → 255 out (231 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-pro:medium PASS 100% 121 in → 255 out (241 reasoning)
WIP: more updates WIP: update app.js
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 398 in → 1,882 out (1,861 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
google/gemini-3-flash-preview:high PASS 100% 126 in → 564 out (551 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3-flash-preview:high__json_schema PASS 100% 126 in → 490 out (473 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3-flash-preview:low PASS 100% 124 in → 407 out (394 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3-flash-preview:low__json_schema PASS 100% 200 in → 443 out (426 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3-flash-preview:medium PASS 100% 124 in → 493 out (480 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3-flash-preview:medium__json_schema PASS 100% 126 in → 388 out (369 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
google/gemini-3-flash-preview:xhigh PASS 100% 126 in → 633 out (620 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 205 in → 1,274 out (1,257 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 125 in → 670 out (657 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 125 in → 904 out (887 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 128 in → 139 out (126 reasoning)
WIP: update app.js WIP: more updates
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 203 in → 148 out (116 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 125 in → 432 out (419 reasoning)
WIP: update app.js WIP: more updates
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 126 in → 307 out (288 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 122 in → 1,472 out (1,459 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 204 in → 595 out (578 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3.1-pro-preview:high PASS 100% 123 in → 703 out (690 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 201 in → 529 out (512 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3.1-pro-preview:low PASS 100% 123 in → 519 out (506 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 204 in → 162 out (130 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemini-3.1-pro-preview:medium PASS 100% 124 in → 630 out (617 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 206 in → 318 out (299 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 121 in → 985 out (972 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 204 in → 601 out (584 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3.5-flash-lite:high PASS 100% 124 in → 1,081 out (1,068 reasoning)
WIP: update app.js WIP: more updates
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 204 in → 781 out (764 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3.5-flash-lite:low PASS 100% 126 in → 13 out (0 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.5-flash-lite:max PASS 100% 123 in → 4,151 out (4,138 reasoning)
WIP: update app.js WIP: more updates
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 127 in → 704 out (687 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3.5-flash-lite:medium PASS 100% 125 in → 747 out (734 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 127 in → 530 out (511 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 126 in → 1,281 out (1,268 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 206 in → 1,190 out (1,173 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3.5-flash:high PASS 100% 125 in → 667 out (654 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.5-flash:low PASS 100% 125 in → 432 out (419 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.5-flash:medium PASS 100% 123 in → 643 out (630 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.5-flash:medium__json_schema PASS 100% 202 in → 748 out (716 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemini-3.5-flash:xhigh PASS 100% 126 in → 1,133 out (1,120 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.6-flash:high PASS 100% 126 in → 3,111 out (3,098 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.6-flash:high__json_schema PASS 100% 125 in → 1,743 out (1,726 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemini-3.6-flash:max PASS 100% 123 in → 3,521 out (3,508 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.6-flash:max__json_schema PASS 100% 205 in → 1,458 out (1,426 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemini-3.6-flash:medium PASS 100% 126 in → 4,527 out (4,514 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.6-flash:medium__json_schema PASS 100% 205 in → 1,334 out (1,302 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemini-3.6-flash:xhigh PASS 100% 125 in → 646 out (633 reasoning)
WIP: more updates WIP: update app.js
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 204 in → 957 out (940 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemma-4-26b-a4b-it:high PASS 100% 137 in → 1,105 out (865 reasoning)
WIP: more updates WIP: update app.js
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 139 in → 691 out (540 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 136 in → 696 out (682 reasoning)
WIP: more updates WIP: update app.js
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 139 in → 1,142 out (914 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 140 in → 1,291 out (1,060 reasoning)
WIP: more updates WIP: update app.js
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 140 in → 1,474 out (1,175 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 136 in → 14 out (0 reasoning)
WIP: more updates WIP: update app.js
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 137 in → 32 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 141 in → 957 out (943 reasoning)
WIP: more updates WIP: update app.js
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 140 in → 1,928 out (1,547 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemma-4-31b-it:high PASS 100% 141 in → 1,016 out (1 reasoning)
WIP: update app.js WIP: more updates
google/gemma-4-31b-it:high__json_schema PASS 100% 138 in → 590 out (1 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemma-4-31b-it:low PASS 100% 140 in → 799 out (660 reasoning)
WIP: update app.js WIP: more updates
google/gemma-4-31b-it:low__json_schema PASS 100% 142 in → 935 out (916 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
google/gemma-4-31b-it:medium PASS 100% 139 in → 346 out (253 reasoning)
WIP: more updates WIP: update app.js
google/gemma-4-31b-it:medium__json_schema PASS 100% 135 in → 404 out (302 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
google/gemma-4-31b-it:none PASS 100% 137 in → 14 out (0 reasoning)
WIP: more updates WIP: update app.js
google/gemma-4-31b-it:xhigh PASS 100% 140 in → 340 out (237 reasoning)
WIP: more updates WIP: update app.js
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 140 in → 675 out (508 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
ibm-granite/granite-4.1-8b PASS 100% 131 in → 13 out
WIP: more updates WIP: update app.js
ibm-granite/granite-4.1-8b__json_schema PASS 100% 133 in → 22 out
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": ["WIP: more updates", "WIP: update app.js"] }
inclusionai/ling-2.6-flash__json_schema PASS 100% 147 in → 34 out
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update app.js", "WIP: more updates" ] }
minimax/minimax-m2.5:high PASS 100% 154 in → 206 out (186 reasoning)
WIP: more updates WIP: update app.js
minimax/minimax-m2.5:high__json_schema PASS 100% 154 in → 331 out (301 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
minimax/minimax-m2.5:low__json_schema PASS 100% 153 in → 308 out (285 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": ["WIP: more updates", "WIP: update app.js"] }
minimax/minimax-m2.5:xhigh PASS 100% 151 in → 1,822 out (1,808 reasoning)
WIP: more updates WIP: update app.js
minimax/minimax-m2.7:high PASS 100% 156 in → 224 out (211 reasoning)
WIP: more updates WIP: update app.js
minimax/minimax-m2.7:high__json_schema PASS 100% 307 in → 636 out (616 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
minimax/minimax-m2.7:low PASS 100% 154 in → 444 out (430 reasoning)
WIP: more updates WIP: update app.js
minimax/minimax-m2.7:low__json_schema PASS 100% 305 in → 707 out (674 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
minimax/minimax-m2.7:medium PASS 100% 153 in → 397 out (419 reasoning)
WIP: more updates WIP: update app.js
minimax/minimax-m2.7:xhigh PASS 100% 154 in → 310 out (297 reasoning)
WIP: more updates WIP: update app.js
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 308 in → 767 out (738 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
minimax/minimax-m3:high PASS 100% 293 in → 170 out (143 reasoning)
WIP: more updates WIP: update app.js
minimax/minimax-m3:low PASS 100% 289 in → 125 out (104 reasoning)
WIP: more updates WIP: update app.js
minimax/minimax-m3:medium PASS 100% 292 in → 141 out (114 reasoning)
WIP: more updates WIP: update app.js
minimax/minimax-m3:xhigh PASS 100% 292 in → 169 out (145 reasoning)
WIP: more updates WIP: update app.js
minimax/minimax-m3:xhigh__json_schema PASS 100% 277 in → 21 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
mistralai/devstral-2512__json_schema PASS 100% 132 in → 28 out
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
mistralai/mistral-medium-3-5:high PASS 100% 142 in → 400 out (332 reasoning)
WIP: more updates WIP: update app.js
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 129 in → 4,585 out (4,283 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
mistralai/mistral-medium-3-5:low PASS 100% 144 in → 3,949 out (3,546 reasoning)
WIP: update app.js WIP: more updates
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 129 in → 445 out (418 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
mistralai/mistral-medium-3-5:medium PASS 100% 140 in → 341 out (284 reasoning)
WIP: more updates WIP: update app.js
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 129 in → 1,988 out (1,907 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
mistralai/mistral-medium-3-5:none PASS 100% 143 in → 13 out (0 reasoning)
WIP: more updates WIP: update app.js
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 132 in → 21 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 145 in → 3,104 out (3,003 reasoning)
WIP: more updates WIP: update app.js
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 132 in → 351 out (279 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits": ["WIP: update app.js", "WIP: more updates"]}
moonshotai/kimi-k2.6:high PASS 100% 122 in → 2,308 out (2,540 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k2.6:high__json_schema PASS 100% 123 in → 4,899 out (4,874 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
moonshotai/kimi-k2.6:low PASS 100% 124 in → 2,081 out (2,067 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k2.6:low__json_schema PASS 100% 122 in → 3,263 out (3,513 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update app.js", "WIP: more updates" ] }
moonshotai/kimi-k2.6:medium PASS 100% 123 in → 898 out (884 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k2.6:xhigh PASS 100% 125 in → 1,199 out (1,237 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k2.7-code:high PASS 100% 126 in → 241 out (227 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k2.7-code:low PASS 100% 122 in → 437 out (423 reasoning)
WIP: update app.js WIP: more updates
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 125 in → 909 out (887 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
moonshotai/kimi-k2.7-code:medium PASS 100% 125 in → 218 out (204 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 216 in → 257 out (238 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 123 in → 349 out (335 reasoning)
WIP: update app.js WIP: more updates
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 219 in → 199 out (180 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
moonshotai/kimi-k3:high PASS 100% 199 in → 366 out (339 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k3:low PASS 100% 201 in → 735 out (708 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k3:low__json_schema PASS 100% 201 in → 707 out (680 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k3:max PASS 100% 198 in → 334 out (307 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k3:max__json_schema PASS 100% 202 in → 2,493 out (2,466 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k3:medium PASS 100% 198 in → 318 out (291 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k3:medium__json_schema PASS 100% 202 in → 191 out (164 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k3:xhigh PASS 100% 201 in → 1,076 out (1,049 reasoning)
WIP: more updates WIP: update app.js
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 202 in → 270 out (243 reasoning)
WIP: more updates WIP: update app.js
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 143 in → 713 out (678 reasoning)
WIP: more updates WIP: update app.js
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 140 in → 309 out (266 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 143 in → 815 out (752 reasoning)
WIP: more updates WIP: update app.js
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 144 in → 505 out (491 reasoning)
WIP: more updates WIP: update app.js
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 141 in → 459 out (445 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 143 in → 604 out (596 reasoning)
WIP: more updates WIP: update app.js
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 143 in → 447 out (414 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 144 in → 221 out (196 reasoning)
WIP: more updates WIP: update app.js
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 144 in → 204 out (165 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 146 in → 264 out (198 reasoning)
WIP: more updates WIP: update app.js
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 144 in → 450 out (388 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update app.js", "WIP: more updates" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 144 in → 174 out (143 reasoning)
WIP: more updates WIP: update app.js
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 144 in → 714 out (654 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update app.js", "WIP: more updates" ] }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 144 in → 28 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
openai/gpt-5.4-mini:high PASS 100% 121 in → 202 out (184 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4-mini:high__json_schema PASS 100% 172 in → 165 out (137 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.4-mini:low PASS 100% 120 in → 63 out (45 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4-mini:low__json_schema PASS 100% 167 in → 61 out (33 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.4-mini:medium PASS 100% 124 in → 232 out (214 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4-mini:medium__json_schema PASS 100% 173 in → 108 out (80 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.4-mini:none PASS 100% 125 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4-mini:none__json_schema PASS 100% 171 in → 26 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.4-mini:xhigh PASS 100% 124 in → 21,441 out (21,423 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 172 in → 20,344 out (20,307 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update app.js", "WIP: more updates" ] }
openai/gpt-5.4-nano:high PASS 100% 117 in → 114 out (96 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4-nano:high__json_schema PASS 100% 169 in → 72 out (44 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.4-nano:low PASS 100% 122 in → 81 out (63 reasoning)
WIP: update app.js WIP: more updates
openai/gpt-5.4-nano:low__json_schema PASS 100% 172 in → 74 out (46 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
openai/gpt-5.4-nano:medium PASS 100% 127 in → 71 out (53 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4-nano:medium__json_schema PASS 100% 172 in → 91 out (63 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
openai/gpt-5.4-nano:none__json_schema PASS 100% 171 in → 26 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.4-nano:xhigh PASS 100% 122 in → 312 out (294 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4:high PASS 100% 120 in → 161 out (143 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4:high__json_schema PASS 100% 169 in → 182 out (154 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.4:low PASS 100% 122 in → 49 out (31 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4:low__json_schema PASS 100% 169 in → 53 out (25 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.4:medium PASS 100% 123 in → 133 out (115 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4:medium__json_schema PASS 100% 166 in → 81 out (53 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.4:none PASS 100% 120 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4:none__json_schema PASS 100% 172 in → 26 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.4:xhigh PASS 100% 122 in → 3,534 out (3,516 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.4:xhigh__json_schema PASS 100% 168 in → 6,760 out (6,732 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.5:high PASS 100% 123 in → 137 out (119 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.5:high__json_schema PASS 100% 168 in → 544 out (516 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
openai/gpt-5.5:low PASS 100% 118 in → 81 out (63 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.5:low__json_schema PASS 100% 167 in → 26 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.5:medium PASS 100% 120 in → 149 out (131 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.5:medium__json_schema PASS 100% 171 in → 239 out (211 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.5:none PASS 100% 121 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.5:none__json_schema PASS 100% 169 in → 26 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.5:xhigh PASS 100% 125 in → 534 out (516 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.5:xhigh__json_schema PASS 100% 166 in → 988 out (960 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-luna:high PASS 100% 119 in → 112 out (94 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-luna:high__json_schema PASS 100% 173 in → 79 out (51 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-luna:low PASS 100% 123 in → 56 out (38 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-luna:low__json_schema PASS 100% 171 in → 49 out (21 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-luna:max PASS 100% 124 in → 216 out (198 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-luna:max__json_schema PASS 100% 170 in → 183 out (155 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-luna:medium PASS 100% 124 in → 54 out (36 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-luna:medium__json_schema PASS 100% 170 in → 66 out (38 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-luna:none PASS 100% 121 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-luna:none__json_schema PASS 100% 170 in → 26 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-luna:xhigh PASS 100% 124 in → 210 out (192 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 169 in → 131 out (103 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-sol:high PASS 100% 125 in → 16 out (0 reasoning)
WIP: update app.js WIP: more updates
openai/gpt-5.6-sol:high__json_schema PASS 100% 167 in → 26 out (0 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
openai/gpt-5.6-sol:low PASS 100% 123 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-sol:low__json_schema PASS 100% 171 in → 26 out (0 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
openai/gpt-5.6-sol:max PASS 100% 124 in → 16 out (0 reasoning)
WIP: update app.js WIP: more updates
openai/gpt-5.6-sol:max__json_schema PASS 100% 171 in → 224 out (196 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-sol:medium PASS 100% 122 in → 16 out (0 reasoning)
WIP: update app.js WIP: more updates
openai/gpt-5.6-sol:medium__json_schema PASS 100% 170 in → 26 out (0 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
openai/gpt-5.6-sol:none PASS 100% 124 in → 16 out (0 reasoning)
WIP: update app.js WIP: more updates
openai/gpt-5.6-sol:none__json_schema PASS 100% 171 in → 26 out (0 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
openai/gpt-5.6-sol:xhigh PASS 100% 120 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 169 in → 26 out (0 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
openai/gpt-5.6-terra:high PASS 100% 125 in → 40 out (22 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-terra:high__json_schema PASS 100% 171 in → 53 out (25 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-terra:low PASS 100% 125 in → 39 out (21 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-terra:low__json_schema PASS 100% 169 in → 46 out (18 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-terra:max PASS 100% 122 in → 120 out (102 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-terra:max__json_schema PASS 100% 173 in → 158 out (130 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-terra:medium PASS 100% 122 in → 39 out (21 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-terra:medium__json_schema PASS 100% 166 in → 40 out (12 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-terra:none PASS 100% 119 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-terra:none__json_schema PASS 100% 171 in → 26 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-5.6-terra:xhigh PASS 100% 121 in → 50 out (32 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 170 in → 57 out (29 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-oss-120b:high PASS 100% 185 in → 477 out (500 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-oss-120b:low PASS 100% 186 in → 99 out (79 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-oss-120b:low__json_schema PASS 100% 183 in → 107 out (80 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-oss-120b:medium PASS 100% 193 in → 213 out (0 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-oss-120b:medium__json_schema PASS 100% 186 in → 310 out (304 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 268 in → 1,180 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
openai/gpt-oss-20b:high PASS 100% 184 in → 308 out (296 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-oss-20b:high__json_schema PASS 100% 184 in → 10,850 out (10,944 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
openai/gpt-oss-20b:low PASS 100% 173 in → 148 out (100 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-oss-20b:low__json_schema PASS 100% 170 in → 245 out (194 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
openai/gpt-oss-20b:medium__json_schema PASS 100% 171 in → 198 out (159 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: update app.js","WIP: more updates"]}
openai/gpt-oss-20b:xhigh PASS 100% 185 in → 800 out (854 reasoning)
WIP: more updates WIP: update app.js
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 185 in → 861 out (896 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
poolside/laguna-m.1:high PASS 100% 133 in → 343 out (330 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-m.1:high__json_schema PASS 100% 130 in → 570 out (542 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update app.js", "WIP: more updates" ] }
poolside/laguna-m.1:low PASS 100% 132 in → 345 out (332 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-m.1:low__json_schema PASS 100% 128 in → 291 out (261 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
poolside/laguna-m.1:medium PASS 100% 130 in → 436 out (423 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-m.1:none PASS 100% 132 in → 12 out (0 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-m.1:none__json_schema PASS 100% 135 in → 26 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
poolside/laguna-m.1:xhigh PASS 100% 131 in → 407 out (394 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-xs-2.1:high PASS 100% 134 in → 631 out (620 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-xs-2.1:high__json_schema PASS 100% 130 in → 1,699 out (1,681 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
poolside/laguna-xs-2.1:low PASS 100% 133 in → 262 out (251 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-xs-2.1:low__json_schema PASS 100% 132 in → 330 out (298 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update app.js", "WIP: more updates" ] }
poolside/laguna-xs-2.1:medium PASS 100% 134 in → 618 out (607 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-xs-2.1:none PASS 100% 131 in → 10 out (0 reasoning)
WIP: update app.js WIP: more updates
poolside/laguna-xs-2.1:none__json_schema FAIL 100% 131 in → 46 out (0 reasoning)
Finalize app.js WIP: more updates WIP: update app.js Add app.js v1
JSON Schema Structured Output
(raw) { "commits": [ "Finalize app.js", "WIP: more updates", "WIP: update app.js", "Add app.js v1" ] }
Failure: Extra selected commit messages: ['Finalize app.js', 'Add app.js v1']
poolside/laguna-xs-2.1:xhigh PASS 100% 132 in → 661 out (650 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-xs.2:high PASS 100% 172 in → 274 out (261 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-xs.2:high__json_schema PASS 100% 165 in → 288 out (269 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
poolside/laguna-xs.2:low__json_schema PASS 100% 170 in → 328 out (296 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
poolside/laguna-xs.2:medium PASS 100% 170 in → 463 out (450 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-xs.2:medium__json_schema PASS 100% 171 in → 265 out (233 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
poolside/laguna-xs.2:none PASS 100% 171 in → 12 out (0 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-xs.2:none__json_schema PASS 100% 167 in → 18 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
poolside/laguna-xs.2:xhigh PASS 100% 168 in → 201 out (188 reasoning)
WIP: more updates WIP: update app.js
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 166 in → 195 out (177 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
qwen/qwen3.6-27b:high PASS 100% 137 in → 1,079 out (991 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-27b:high__json_schema PASS 100% 138 in → 1,437 out (1,308 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
qwen/qwen3.6-27b:low PASS 100% 136 in → 1,171 out (1,088 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-27b:low__json_schema PASS 100% 136 in → 1,456 out (1 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
qwen/qwen3.6-27b:medium PASS 100% 137 in → 1,026 out (952 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-27b:medium__json_schema PASS 100% 135 in → 1,182 out (1,069 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
qwen/qwen3.6-27b:none PASS 100% 137 in → 12 out (0 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-27b:xhigh PASS 100% 135 in → 1,112 out (1,084 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 137 in → 1,165 out (1 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 137 in → 1,267 out (1,250 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 137 in → 1,724 out (1,596 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 136 in → 2,292 out (2,200 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 135 in → 1,375 out (1,219 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 135 in → 1,296 out (1,203 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 133 in → 2,814 out (2,627 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 141 in → 13 out (0 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 135 in → 33 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 137 in → 1,322 out (1,234 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 137 in → 1,508 out (1,362 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
qwen/qwen3.6-flash:high PASS 100% 135 in → 1,603 out (1,586 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-flash:low PASS 100% 138 in → 1,181 out (1,164 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-flash:medium PASS 100% 134 in → 1,704 out (1,687 reasoning)
WIP: update app.js WIP: more updates
qwen/qwen3.6-flash:none PASS 100% 138 in → 12 out (0 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.6-flash:xhigh PASS 100% 137 in → 1,898 out (1,881 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-flash:high PASS 100% 136 in → 1,476 out (1,459 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-flash:low PASS 100% 136 in → 1,240 out (1,223 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-flash:max PASS 100% 136 in → 1,244 out (1,227 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-flash:medium PASS 100% 136 in → 1,409 out (1,392 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-flash:xhigh PASS 100% 136 in → 1,331 out (1,314 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-max:high PASS 100% 133 in → 463 out (446 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-max:low PASS 100% 137 in → 403 out (386 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-max:none PASS 100% 137 in → 12 out (0 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-max:xhigh PASS 100% 136 in → 1,274 out (1,257 reasoning)
WIP: update app.js WIP: more updates
qwen/qwen3.7-plus:high PASS 100% 136 in → 288 out (271 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-plus:low PASS 100% 138 in → 700 out (683 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-plus:medium PASS 100% 138 in → 1,405 out (1,388 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-plus:none PASS 100% 138 in → 12 out (0 reasoning)
WIP: more updates WIP: update app.js
qwen/qwen3.7-plus:xhigh PASS 100% 137 in → 1,262 out (1,245 reasoning)
WIP: more updates WIP: update app.js
tencent/hy3:high PASS 100% 129 in → 517 out (503 reasoning)
WIP: more updates WIP: update app.js
tencent/hy3:high__json_schema PASS 100% 130 in → 1,407 out (1,326 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update app.js", "WIP: more updates" ] }
tencent/hy3:low PASS 100% 126 in → 482 out (468 reasoning)
WIP: more updates WIP: update app.js
tencent/hy3:low__json_schema PASS 100% 124 in → 405 out (375 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
tencent/hy3:medium PASS 100% 131 in → 813 out (799 reasoning)
WIP: more updates WIP: update app.js
tencent/hy3:medium__json_schema PASS 100% 131 in → 893 out (814 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": ["WIP: more updates", "WIP: update app.js"] }
tencent/hy3:none PASS 100% 131 in → 13 out (0 reasoning)
WIP: more updates WIP: update app.js
tencent/hy3:none__json_schema PASS 100% 134 in → 25 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
tencent/hy3:xhigh PASS 100% 126 in → 418 out (404 reasoning)
WIP: more updates WIP: update app.js
tencent/hy3:xhigh__json_schema PASS 100% 128 in → 4,860 out (4,899 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
thinkingmachines/inkling-small:high PASS 100% 129 in → 414 out (396 reasoning)
WIP: update app.js WIP: more updates
thinkingmachines/inkling-small:high__json_schema PASS 100% 132 in → 860 out (834 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) { "commits": ["WIP: update app.js", "WIP: more updates"] }
thinkingmachines/inkling-small:low PASS 100% 133 in → 87 out (47 reasoning)
WIP: update app.js WIP: more updates
thinkingmachines/inkling-small:low__json_schema PASS 100% 130 in → 117 out (85 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
thinkingmachines/inkling-small:max PASS 100% 128 in → 1,058 out (1,086 reasoning)
WIP: more updates WIP: update app.js
thinkingmachines/inkling-small:max__json_schema PASS 100% 130 in → 675 out (650 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
thinkingmachines/inkling-small:medium PASS 100% 131 in → 366 out (349 reasoning)
WIP: more updates WIP: update app.js
thinkingmachines/inkling-small:medium__json_schema PASS 100% 132 in → 247 out (222 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates", "WIP: update app.js"]}
thinkingmachines/inkling-small:xhigh PASS 100% 128 in → 314 out (296 reasoning)
WIP: more updates WIP: update app.js
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 131 in → 2,169 out (2,143 reasoning)
WIP: update app.js WIP: more updates
JSON Schema Structured Output
(raw) { "commits": ["WIP: update app.js", "WIP: more updates"] }
thinkingmachines/inkling:high PASS 100% 131 in → 303 out (284 reasoning)
WIP: more updates WIP: update app.js
thinkingmachines/inkling:high__json_schema PASS 100% 133 in → 346 out (314 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
thinkingmachines/inkling:low PASS 100% 132 in → 59 out (41 reasoning)
WIP: more updates WIP: update app.js
thinkingmachines/inkling:low__json_schema PASS 100% 134 in → 48 out (25 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
thinkingmachines/inkling:max PASS 100% 130 in → 357 out (339 reasoning)
WIP: update app.js WIP: more updates
thinkingmachines/inkling:max__json_schema PASS 100% 132 in → 339 out (307 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
thinkingmachines/inkling:medium PASS 100% 132 in → 175 out (157 reasoning)
WIP: more updates WIP: update app.js
thinkingmachines/inkling:none PASS 100% 130 in → 16 out (0 reasoning)
WIP: update app.js WIP: more updates
thinkingmachines/inkling:none__json_schema PASS 100% 128 in → 27 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":[ "WIP: more updates", "WIP: update app.js" ]}
thinkingmachines/inkling:xhigh PASS 100% 131 in → 279 out (261 reasoning)
WIP: more updates WIP: update app.js
thinkingmachines/inkling:xhigh__json_schema PASS 100% 134 in → 377 out (345 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
x-ai/grok-4.3:high PASS 100% 295 in → 729 out (717 reasoning)
WIP: more updates WIP: update app.js
x-ai/grok-4.3:high__json_schema PASS 100% 363 in → 605 out (579 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
x-ai/grok-4.3:low PASS 100% 302 in → 340 out (328 reasoning)
WIP: more updates WIP: update app.js
x-ai/grok-4.3:low__json_schema PASS 100% 371 in → 460 out (444 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
x-ai/grok-4.3:max PASS 100% 296 in → 376 out (364 reasoning)
WIP: more updates WIP: update app.js
x-ai/grok-4.3:max__json_schema PASS 100% 366 in → 816 out (800 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
x-ai/grok-4.3:medium PASS 100% 304 in → 1,679 out (1,667 reasoning)
WIP: more updates WIP: update app.js
x-ai/grok-4.3:medium__json_schema PASS 100% 370 in → 954 out (928 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
x-ai/grok-4.3:none__json_schema PASS 100% 364 in → 16 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
x-ai/grok-4.3:xhigh PASS 100% 296 in → 713 out (701 reasoning)
WIP: more updates WIP: update app.js
x-ai/grok-4.3:xhigh__json_schema PASS 100% 365 in → 965 out (939 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
x-ai/grok-4.5:high PASS 100% 315 in → 455 out (445 reasoning)
WIP: more updates WIP: update app.js
x-ai/grok-4.5:high__json_schema PASS 100% 398 in → 469 out (454 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
x-ai/grok-4.5:low PASS 100% 315 in → 367 out (357 reasoning)
WIP: more updates WIP: update app.js
x-ai/grok-4.5:low__json_schema PASS 100% 395 in → 366 out (341 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
x-ai/grok-4.5:max PASS 100% 315 in → 703 out (693 reasoning)
WIP: more updates WIP: update app.js
x-ai/grok-4.5:max__json_schema PASS 100% 396 in → 441 out (416 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
x-ai/grok-4.5:medium PASS 100% 316 in → 965 out (955 reasoning)
WIP: more updates WIP: update app.js
x-ai/grok-4.5:medium__json_schema PASS 100% 396 in → 592 out (577 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
x-ai/grok-4.5:xhigh PASS 100% 319 in → 1,028 out (1,018 reasoning)
WIP: more updates WIP: update app.js
x-ai/grok-4.5:xhigh__json_schema PASS 100% 397 in → 435 out (420 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates","WIP: update app.js"]}
z-ai/glm-4.7-flash:high PASS 100% 126 in → 1,327 out (1,092 reasoning)
WIP: more updates WIP: update app.js
z-ai/glm-4.7-flash:low PASS 100% 122 in → 678 out (609 reasoning)
WIP: update app.js WIP: more updates
z-ai/glm-4.7-flash:medium PASS 100% 122 in → 467 out (408 reasoning)
WIP: update app.js WIP: more updates
z-ai/glm-4.7-flash:none PASS 100% 122 in → 13 out (0 reasoning)
WIP: more updates WIP: update app.js
z-ai/glm-4.7-flash:none__json_schema PASS 100% 123 in → 28 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
z-ai/glm-4.7-flash:xhigh PASS 100% 127 in → 1,690 out (1,553 reasoning)
WIP: more updates WIP: update app.js
z-ai/glm-5.2:high PASS 100% 129 in → 261 out (247 reasoning)
WIP: more updates WIP: update app.js
z-ai/glm-5.2:low PASS 100% 131 in → 446 out (432 reasoning)
WIP: more updates WIP: update app.js
z-ai/glm-5.2:low__json_schema PASS 100% 129 in → 376 out (329 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
z-ai/glm-5.2:medium PASS 100% 131 in → 405 out (363 reasoning)
WIP: more updates WIP: update app.js
z-ai/glm-5.2:none PASS 100% 125 in → 13 out (0 reasoning)
WIP: more updates WIP: update app.js
z-ai/glm-5.2:none__json_schema PASS 100% 124 in → 22 out (0 reasoning)
WIP: more updates WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": ["WIP: more updates", "WIP: update app.js"] }
z-ai/glm-5.2:xhigh PASS 100% 133 in → 672 out (632 reasoning)
WIP: more updates WIP: update app.js
anthropic/claude-sonnet-4.6:low FAIL 50% 131 in → 8 out (0 reasoning)
WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js']
arcee-ai/trinity-large-thinking:xhigh FAIL 50% 118 in → 3,958 out (4,098 reasoning)
WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js']
arcee-ai/trinity-mini:high FAIL 50% 120 in → 460 out (461 reasoning)
WIP:more updates WIP: update app.js
Failure: Missing expected commit messages: ['WIP: more updates']; Extra selected commit messages: ['WIP:more updates']
arcee-ai/trinity-mini:medium FAIL 50% 119 in → 535 out (531 reasoning)
WIP: more updates WIP:update app.js
Failure: Missing expected commit messages: ['WIP: update app.js']; Extra selected commit messages: ['WIP:update app.js']
deepseek/deepseek-v4-flash-0731:medium__json_schema FAIL 50% 122 in → 2,556 out (2,523 reasoning)
WIP: more updates
JSON Schema Structured Output
(raw) { "commits": ["WIP: more updates"] }
Failure: Missing expected commit messages: ['WIP: update app.js']
google/gemini-3.5-flash:high__json_schema FAIL 50% 202 in → 3,294 out (3,284 reasoning)
WIP: more updates
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates"]}
Failure: Missing expected commit messages: ['WIP: update app.js']
google/gemini-3.5-flash:xhigh__json_schema FAIL 50% 204 in → 3,186 out (3,165 reasoning)
WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates" ] }
Failure: Missing expected commit messages: ['WIP: update app.js']
google/gemini-3.6-flash:low FAIL 50% 125 in → 1,305 out (1,300 reasoning)
WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js']
moonshotai/kimi-k3:high__json_schema FAIL 50% 198 in → 666 out (646 reasoning)
WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js']
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 50% 143 in → 2,831 out (2,656 reasoning)
WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update app.js" ] }
Failure: Missing expected commit messages: ['WIP: more updates']
nvidia/nemotron-3-super-120b-a12b:xhigh FAIL 50% 146 in → 3,783 out (3,487 reasoning)
WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js']
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema FAIL 50% 144 in → 3,298 out (3,169 reasoning)
WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates" ] }
Failure: Missing expected commit messages: ['WIP: update app.js']
openai/gpt-5.4-nano:xhigh__json_schema FAIL 50% 169 in → 14,565 out (14,544 reasoning)
WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["WIP: more updates"]}
Failure: Missing expected commit messages: ['WIP: update app.js']
openai/gpt-oss-120b:high__json_schema FAIL 50% 267 in → 2,184 out (0 reasoning)
WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates" ] }
Failure: Missing expected commit messages: ['WIP: update app.js']
openai/gpt-oss-120b:xhigh FAIL 50% 182 in → 2,784 out (2,981 reasoning)
WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js']
openai/gpt-oss-20b:medium FAIL 50% 178 in → 1,693 out (1,797 reasoning)
WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js']
poolside/laguna-xs-2.1:medium__json_schema FAIL 50% 134 in → 1,855 out (1,844 reasoning)
WIP: more updates
JSON Schema Structured Output
(raw) {"commits": ["WIP: more updates"]}
Failure: Missing expected commit messages: ['WIP: update app.js']
poolside/laguna-xs-2.1:xhigh__json_schema FAIL 50% 133 in → 361 out (338 reasoning)
WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "WIP: more updates" ] }
Failure: Missing expected commit messages: ['WIP: update app.js']
qwen/qwen3.7-max:medium FAIL 50% 135 in → 2,618 out (2,608 reasoning)
WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js']
thinkingmachines/inkling-small:none FAIL 50% 130 in → 15 out (0 reasoning)
WIP: update app.js WIP: more updates<|end_message|>
Failure: Missing expected commit messages: ['WIP: more updates']; Extra selected commit messages: ['WIP: more updates']
anthropic/claude-haiku-4.5:high FAIL 0% 160 in → 662 out (529 reasoning)
f31f575 WIP: update app.js 784972b WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: update app.js', 'WIP: more updates']
anthropic/claude-haiku-4.5:low FAIL 0% 165 in → 2,838 out (1,411 reasoning)
9b7ae6f WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates']
anthropic/claude-haiku-4.5:medium FAIL 0% 160 in → 5,194 out (1,822 reasoning)
802e84c WIP: more updates
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 321 in → 20 out (0 reasoning)
b6c37d3 081a357
JSON Schema Structured Output
(raw) {"commits": ["b6c37d3", "081a357"]}
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
anthropic/claude-sonnet-5:high__json_schema FAIL 0% 416 in → 36 out (0 reasoning)
7add25b WIP: update app.js c1a412b WIP: more updates
JSON Schema Structured Output
(raw) {"commits":["7add25b WIP: update app.js","c1a412b WIP: more updates"]}
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: update app.js', 'WIP: more updates']
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 418 in → 36 out (0 reasoning)
7735378 WIP: more updates 91b8f33 WIP: update app.js
JSON Schema Structured Output
(raw) {"commits":["7735378 WIP: more updates","91b8f33 WIP: update app.js"]}
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high__json_schema FAIL 0% 120 in → 587 out (587 reasoning)
c51a989 Finalize app.js f5a309f WIP: more updates 1744516 WIP: update app.js 4f3aa24 Add app.js v1
JSON Schema Structured Output
(raw) { "commits": [ "c51a989 Finalize app.js", "f5a309f WIP: more updates", "1744516 WIP: update app.js", "4f3aa24 Add app.js v1" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['Finalize app.js', 'WIP: more updates', 'WIP: update app.js', 'Add app.js v1']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 121 in → 498 out (467 reasoning)
165e87f Finalize app.js 1e5faab WIP: more updates 8332b5d WIP: update app.js 1ec2762 Add app.js v1
JSON Schema Structured Output
(raw) { "commits": [ "165e87f Finalize app.js", "1e5faab WIP: more updates", "8332b5d WIP: update app.js", "1ec2762 Add app.js v1" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['Finalize app.js', 'WIP: more updates', 'WIP: update app.js', 'Add app.js v1']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 120 in → 572 out (547 reasoning)
a559dd7 Finalize app.js d80d12d WIP: more updates 63de88e WIP: update app.js 181e10d Add app.js v1
JSON Schema Structured Output
(raw) { "commits": [ "a559dd7 Finalize app.js", "d80d12d WIP: more updates", "63de88e WIP: update app.js", "181e10d Add app.js v1" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['Finalize app.js', 'WIP: more updates', 'WIP: update app.js', 'Add app.js v1']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 117 in → 354 out (329 reasoning)
48fe677 Finalize app.js d6341fd WIP: more updates 801373d WIP: update app.js 71dc3ca Add app.js v1
JSON Schema Structured Output
(raw) {"commits": ["48fe677 Finalize app.js", "d6341fd WIP: more updates", "801373d WIP: update app.js", "71dc3ca Add app.js v1"]}
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['Finalize app.js', 'WIP: more updates', 'WIP: update app.js', 'Add app.js v1']
deepseek/deepseek-v4-flash-0731:high__json_schema FAIL 0% 203 in → 1,346 out (1,274 reasoning)
0731325 WIP: update app.js 02ec63a WIP: more updates
JSON Schema Structured Output
(raw) { "commits": [ "0731325 WIP: update app.js", "02ec63a WIP: more updates" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: update app.js', 'WIP: more updates']
Invalid JSON. Output: { { "commits": [ "WIP: more updates", "WIP: update app.js" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)
deepseek/deepseek-v4-flash-0731:none FAIL 0% 120 in → 21 out (0 reasoning)
307b86a WIP: more updates 65baaa4 WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
Invalid JSON. Output: WIP: more updates
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:low__json_schema FAIL 0% 125 in → 667 out (629 reasoning)
83971c2 WIP: more updates 2ee7bb3 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "83971c2 WIP: more updates", "2ee7bb3 WIP: update app.js" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
Invalid JSON. Output: WIP: more updates WIP: update app.js
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: more updates WIP: update app.js
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:high__json_schema FAIL 0% 121 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "commits": [] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 124 in → 222 out (173 reasoning)
c2287ca WIP: more updates b2a6957 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "c2287ca WIP: more updates", "b2a6957 WIP: update app.js" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
Invalid JSON. Output: WIP: update app.js WIP: more updates
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:none FAIL 0% 124 in → 22 out (0 reasoning)
10ca6a3 WIP: more updates c3aff1e WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 120 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "commits": [] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']
deepseek/deepseek-v4-pro:xhigh FAIL 0% 125 in → 468 out (448 reasoning)
WIP: more updatesWIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 205 in → 31 out (0 reasoning)
a8d114b WIP: more updates a3d685a WIP: update app.js
JSON Schema Structured Output
(raw) {"commits": ["a8d114b WIP: more updates", "a3d685a WIP: update app.js"]}
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
google/gemini-3.5-flash:low__json_schema FAIL 0% 206 in → 654 out (610 reasoning)
7c723d9 WIP: more updates 7b1c974 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "7c723d9 WIP: more updates", "7b1c974 WIP: update app.js" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
google/gemini-3.6-flash:low__json_schema FAIL 0% 126 in → 44 out (0 reasoning)
33c4380 WIP: more updates 5895862 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "33c4380 WIP: more updates", "5895862 WIP: update app.js" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
google/gemma-4-31b-it:none__json_schema FAIL 0% 139 in → 45 out (0 reasoning)
525e191 WIP: more updates 93a4396 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "525e191 WIP: more updates", "93a4396 WIP: update app.js" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
inclusionai/ling-2.6-flash FAIL 0% 148 in → 27 out
e0077c2 WIP: more updates c42a634 WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
liquid/lfm-2-24b-a2b FAIL 0% 129 in → 7 out
more updates update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['more updates', 'update app.js']
Invalid JSON. Output: 95f4bc2 2c17afa
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
minimax/minimax-m2.5:low FAIL 0% 158 in → 283 out (225 reasoning)
3b0a948 WIP: more updates 5d08d44 WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
minimax/minimax-m2.5:medium FAIL 0% 160 in → 209 out (186 reasoning)
57507e6 WIP: more updates d4f5287 WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
minimax/minimax-m2.5:medium__json_schema FAIL 0% 153 in → 443 out (393 reasoning)
fa9b199 WIP: more updates c845102 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "fa9b199 WIP: more updates", "c845102 WIP: update app.js" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
minimax/minimax-m2.5:xhigh__json_schema FAIL 0% 153 in → 533 out (497 reasoning)
ed9d153 WIP: more updates 65934b8 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "ed9d153 WIP: more updates", "65934b8 WIP: update app.js" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high__json_schema FAIL 0% 242 in → 62 out (61 reasoning)
16c4e58 Finalize app.js a6efdf1 WIP: more updates e71ec1a WIP: update app.js e3977ee Add app.js v1
JSON Schema Structured Output
(raw) { "commits": [ "16c4e58 Finalize app.js", "a6efdf1 WIP: more updates", "e71ec1a WIP: update app.js", "e3977ee Add app.js v1" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['Finalize app.js', 'WIP: more updates', 'WIP: update app.js', 'Add app.js v1']
minimax/minimax-m3:low__json_schema FAIL 0% 243 in → 63 out (62 reasoning)
0ea4a4c Finalize app.js dade4f9 WIP: more updates 06ec6f9 WIP: update app.js b848c2a Add app.js v1
JSON Schema Structured Output
(raw) { "commits": [ "0ea4a4c Finalize app.js", "dade4f9 WIP: more updates", "06ec6f9 WIP: update app.js", "b848c2a Add app.js v1" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['Finalize app.js', 'WIP: more updates', 'WIP: update app.js', 'Add app.js v1']
minimax/minimax-m3:medium__json_schema FAIL 0% 282 in → 7 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"commits": []}
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']
mistralai/devstral-2512 FAIL 0% 131 in → 27 out
b010676 WIP: more updates 39c8250 WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: update app.js WIP: more updates
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 144 in → 15 out (0 reasoning)
ab0277e 342e524
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 142 in → 51 out (0 reasoning)
fcc1e95 WIP: more updates 2673832 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "fcc1e95 WIP: more updates", "2673832 WIP: update app.js" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
nvidia/nemotron-3-super-120b-a12b:none FAIL 0% 143 in → 26 out (0 reasoning)
a85ae81 WIP: more updates 91767f3 WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
openai/gpt-5.4-nano:none FAIL 0% 124 in → 25 out (0 reasoning)
69db30e WIP: more updates b6497d4 WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
poolside/laguna-m.1:medium__json_schema FAIL 0% 132 in → 385 out (344 reasoning)
5a07dd8 WIP: more updates 2385450 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "5a07dd8 WIP: more updates", "2385450 WIP: update app.js" ]}
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
poolside/laguna-m.1:xhigh__json_schema FAIL 0% 132 in → 336 out (297 reasoning)
1f92b1e WIP: more updates 40c0fe1 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "1f92b1e WIP: more updates", "40c0fe1 WIP: update app.js" ]}
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
poolside/laguna-xs.2:low FAIL 0% 169 in → 258 out (231 reasoning)
7825114 WIP: more updates 5340982 WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 139 in → 35 out (0 reasoning)
7287ff8 7465919
JSON Schema Structured Output
(raw) { "commits": [ "7287ff8", "7465919" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-010bcd62-28f8-97e2-8341-63f222dc3dcd","request_id":"010bcd62-28f8-97e2-8341-63f222dc3dcd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6b538169-bc14-9e18-a5a7-f290d62c713a","request_id":"6b538169-bc14-9e18-a5a7-f290d62c713a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cbb6d5d1-a6fc-95e5-9ab4-942557acf67c","request_id":"cbb6d5d1-a6fc-95e5-9ab4-942557acf67c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8bb35eab-613d-9f55-ae65-3a1bdf91f9a7","request_id":"8bb35eab-613d-9f55-ae65-3a1bdf91f9a7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b1f07472-6fc6-9b32-a4c2-f62f83ee9f21","request_id":"b1f07472-6fc6-9b32-a4c2-f62f83ee9f21"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-41988ac9-552c-9218-ac92-ffe58d2fac63"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0f0a2581-cea7-9bf3-a46e-ee9d8e7db019"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8e06b0fd-d231-975e-b83b-21b158278bd9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-553880f2-0943-926d-9fd2-8a49e27f21db"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none FAIL 0% 137 in → 25 out (0 reasoning)
03b09b2 WIP: more updates fb1e789 WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a81d634a-3b41-9779-95a7-cb4dbf80e170"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f798698e-7f36-915b-b057-02b7f6207c62"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c4fb4c0b-3368-9c81-91c7-0a95864c5e9c","request_id":"c4fb4c0b-3368-9c81-91c7-0a95864c5e9c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-de9a86c4-5131-976d-8c94-03eee533b357","request_id":"de9a86c4-5131-976d-8c94-03eee533b357"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b764b00a-b790-91b3-9755-6890626c3a5f","request_id":"b764b00a-b790-91b3-9755-6890626c3a5f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1534eb6f-901b-9ccf-9675-869bf1663d4c","request_id":"1534eb6f-901b-9ccf-9675-869bf1663d4c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-17a84a5a-7fe5-945f-a9e5-1ae881269f9b","request_id":"17a84a5a-7fe5-945f-a9e5-1ae881269f9b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7d5a4523-b266-9d1b-9845-899acc06d8ec","request_id":"7d5a4523-b266-9d1b-9845-899acc06d8ec"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-df47663e-5f0a-9a3d-89d7-ad5d27d6eaa9","request_id":"df47663e-5f0a-9a3d-89d7-ad5d27d6eaa9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7e5c54d6-61de-9511-bae5-a69128a6f280","request_id":"7e5c54d6-61de-9511-bae5-a69128a6f280"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c81dd575-2944-9545-9a81-0136b694de62","request_id":"c81dd575-2944-9545-9a81-0136b694de62"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-23c53355-d0bb-9e39-972f-8a1d4dcce207","request_id":"23c53355-d0bb-9e39-972f-8a1d4dcce207"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: WIP: update app.js WIP: more updates{ "commits": [ "WIP: update app.js", "WIP: more updates" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: more updates WIP: update app.js
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none FAIL 0% 292 in → 17 out (0 reasoning)
16261ab WIP: more updates 8d41111 WIP: update app.js
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: more updates WIP: update app.js
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: more updates WIP: update app.js
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:xhigh__json_schema FAIL 0% 131 in → 558 out (483 reasoning)
87cae98 WIP: more updates 53e84a6 WIP: update app.js
JSON Schema Structured Output
(raw) { "commits": [ "87cae98 WIP: more updates", "53e84a6 WIP: update app.js" ] }
Failure: Missing expected commit messages: ['WIP: update app.js', 'WIP: more updates']; Extra selected commit messages: ['WIP: more updates', 'WIP: update app.js']