Debug commits that should be removed/squashed
Tests ability to identify debug commits that should be removed rather than squashed. Evaluates recognizing debugging artifacts in history.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'real code' > app.js
  5. 05 git add app.js
  6. 06 git commit -m 'Add real feature'
  7. 07 echo 'console.log(debug1)' >> app.js
  8. 08 git add app.js
  9. 09 git commit -m 'WIP: debug commit 1'
  10. 10 echo 'console.log(debug2)' >> app.js
  11. 11 git add app.js
  12. 12 git commit -m 'WIP: debug commit 2'
  13. 13 grep -v debug app.js > app.js.tmp && mv app.js.tmp app.js
  14. 14 git add app.js
  15. 15 git commit -m 'Remove debug statements'
Prompt
The history contains debug commits that should be marked as squash/fixup and removed as part of the final feature commit. Identify the debug-related WIP commits only. Do not include the cleanup commit that removes the debug statements. Output ONLY the selected commit subject lines, one per line, with no explanation.
Expected
WIP: debug commit 1 WIP: debug commit 2

Scoped model quality, cost, API time, and token usage for commit_squash/f010.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 167 in → 37 out (11 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-fable-5:high__json_schema PASS 100% 407 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
anthropic/claude-fable-5:low PASS 100% 165 in → 23 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-fable-5:low__json_schema PASS 100% 409 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
anthropic/claude-fable-5:max PASS 100% 173 in → 335 out (70 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-fable-5:max__json_schema PASS 100% 405 in → 285 out (54 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
anthropic/claude-fable-5:medium PASS 100% 171 in → 35 out (10 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-fable-5:medium__json_schema PASS 100% 410 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
anthropic/claude-fable-5:xhigh PASS 100% 170 in → 42 out (17 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 413 in → 80 out (31 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
anthropic/claude-haiku-4.5:high PASS 100% 154 in → 289 out (245 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 337 in → 1,655 out (708 reasoning)
WIP: debug commit 1 WIP: debug commit 2
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 1", "WIP: debug commit 2"]}
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 340 in → 777 out (630 reasoning)
WIP: debug commit 1 WIP: debug commit 2
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 1", "WIP: debug commit 2"]}
anthropic/claude-haiku-4.5:medium PASS 100% 150 in → 350 out (317 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 341 in → 940 out (567 reasoning)
WIP: debug commit 1 WIP: debug commit 2
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 1", "WIP: debug commit 2"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 154 in → 346 out (293 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 339 in → 735 out (493 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
anthropic/claude-opus-4.6:high PASS 100% 126 in → 121 out (88 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.6:high__json_schema PASS 100% 313 in → 156 out (107 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.6:low PASS 100% 121 in → 18 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.6:max PASS 100% 124 in → 175 out (140 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.6:max__json_schema PASS 100% 316 in → 199 out (141 reasoning)
WIP: debug commit 1 WIP: debug commit 2
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 1","WIP: debug commit 2"]}
anthropic/claude-opus-4.6:medium PASS 100% 123 in → 18 out (0 reasoning)
WIP: debug commit 1 WIP: debug commit 2
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 313 in → 24 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.6:none PASS 100% 124 in → 18 out (0 reasoning)
WIP: debug commit 1 WIP: debug commit 2
anthropic/claude-opus-4.6:none__json_schema PASS 100% 316 in → 24 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 127 in → 95 out (65 reasoning)
WIP: debug commit 1 WIP: debug commit 2
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 313 in → 130 out (87 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.7:high PASS 100% 171 in → 25 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.7:high__json_schema PASS 100% 415 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.7:low PASS 100% 177 in → 25 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.7:low__json_schema PASS 100% 416 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.7:max PASS 100% 174 in → 259 out (65 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.7:max__json_schema PASS 100% 417 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.7:medium PASS 100% 178 in → 25 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 414 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.7:none PASS 100% 174 in → 25 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.7:none__json_schema PASS 100% 412 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 172 in → 25 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 410 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.8:high PASS 100% 166 in → 23 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.8:high__json_schema PASS 100% 409 in → 32 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.8:low PASS 100% 169 in → 23 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.8:low__json_schema PASS 100% 410 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
anthropic/claude-opus-4.8:max PASS 100% 171 in → 348 out (90 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.8:max__json_schema PASS 100% 410 in → 257 out (50 reasoning)
WIP: debug commit 1 WIP: debug commit 2
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 1", "WIP: debug commit 2"]}
anthropic/claude-opus-4.8:medium PASS 100% 171 in → 49 out (32 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 411 in → 32 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.8:none PASS 100% 169 in → 23 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.8:none__json_schema PASS 100% 409 in → 31 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 167 in → 194 out (48 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 412 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 311 in → 24 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-sonnet-4.6:low PASS 100% 126 in → 18 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 309 in → 24 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-sonnet-4.6:max PASS 100% 126 in → 206 out (163 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 310 in → 196 out (146 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 124 in → 18 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 310 in → 24 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-sonnet-4.6:none PASS 100% 124 in → 18 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 313 in → 24 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 126 in → 18 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 310 in → 24 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-sonnet-5:high PASS 100% 166 in → 41 out (18 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-sonnet-5:high__json_schema PASS 100% 412 in → 32 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-sonnet-5:low FAIL 100% 166 in → 27 out (0 reasoning)
``` WIP: debug commit 2 WIP: debug commit 1 ```
Failure: Extra selected commit messages: ['```', '```']
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 407 in → 32 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-sonnet-5:none PASS 100% 169 in → 23 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
anthropic/claude-sonnet-5:none__json_schema PASS 100% 407 in → 31 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
anthropic/claude-sonnet-5:xhigh FAIL 100% 170 in → 27 out (0 reasoning)
``` WIP: debug commit 2 WIP: debug commit 1 ```
Failure: Extra selected commit messages: ['```', '```']
arcee-ai/trinity-large-thinking:high PASS 100% 113 in → 896 out (910 reasoning)
WIP: debug commit 2 WIP: debug commit 1
arcee-ai/trinity-large-thinking:low PASS 100% 118 in → 647 out (653 reasoning)
WIP: debug commit 2 WIP: debug commit 1
arcee-ai/trinity-large-thinking:medium PASS 100% 117 in → 487 out (492 reasoning)
WIP: debug commit 2 WIP: debug commit 1
arcee-ai/trinity-large-thinking:xhigh PASS 100% 116 in → 554 out (560 reasoning)
WIP: debug commit 2 WIP: debug commit 1
arcee-ai/trinity-mini:high PASS 100% 117 in → 286 out (296 reasoning)
WIP: debug commit 2 WIP: debug commit 1
arcee-ai/trinity-mini:low PASS 100% 118 in → 305 out (323 reasoning)
WIP: debug commit 2 WIP: debug commit 1
arcee-ai/trinity-mini:xhigh PASS 100% 115 in → 404 out (395 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-flash-0731:max PASS 100% 206 in → 245 out (229 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 192 in → 1,040 out (1,038 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 194 in → 238 out (222 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-flash:high PASS 100% 115 in → 188 out (171 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-flash:low PASS 100% 114 in → 248 out (231 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-flash:medium PASS 100% 116 in → 134 out (112 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-flash:none PASS 100% 112 in → 15 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-flash:xhigh PASS 100% 190 in → 333 out (325 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-pro:high PASS 100% 112 in → 180 out (163 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 317 in → 94 out (69 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
deepseek/deepseek-v4-pro:low PASS 100% 116 in → 424 out (408 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 117 in → 130 out (98 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 118 in → 30 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 196 in → 368 out (352 reasoning)
WIP: debug commit 2 WIP: debug commit 1
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 114 in → 31 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemini-3-flash-preview:high PASS 100% 115 in → 349 out (334 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3-flash-preview:high__json_schema PASS 100% 197 in → 731 out (713 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3-flash-preview:low PASS 100% 116 in → 295 out (280 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3-flash-preview:low__json_schema PASS 100% 196 in → 415 out (396 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3-flash-preview:medium PASS 100% 118 in → 342 out (327 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3-flash-preview:medium__json_schema PASS 100% 119 in → 388 out (369 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3-flash-preview:xhigh PASS 100% 116 in → 285 out (270 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 119 in → 373 out (355 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 120 in → 496 out (481 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 118 in → 997 out (976 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 116 in → 195 out (180 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.1-flash-lite-preview:medium PASS 100% 116 in → 279 out (264 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 115 in → 324 out (305 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 119 in → 513 out (498 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.1-pro-preview:high PASS 100% 115 in → 452 out (437 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 197 in → 314 out (295 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.1-pro-preview:low PASS 100% 118 in → 316 out (301 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 195 in → 397 out (378 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.1-pro-preview:medium PASS 100% 119 in → 340 out (325 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 194 in → 465 out (446 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 117 in → 492 out (477 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 198 in → 616 out (597 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.5-flash-lite:high PASS 100% 116 in → 814 out (799 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 119 in → 967 out (948 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.5-flash-lite:low PASS 100% 119 in → 15 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 117 in → 34 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemini-3.5-flash-lite:max PASS 100% 115 in → 896 out (881 reasoning)
WIP: debug commit 1 WIP: debug commit 2
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 194 in → 915 out (896 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.5-flash-lite:medium PASS 100% 116 in → 603 out (588 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 119 in → 582 out (548 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemini-3.5-flash-lite:xhigh PASS 100% 119 in → 745 out (730 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 115 in → 801 out (780 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
google/gemini-3.5-flash:high PASS 100% 116 in → 338 out (323 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.5-flash:high__json_schema PASS 100% 195 in → 600 out (579 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
google/gemini-3.5-flash:low PASS 100% 117 in → 264 out (249 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.5-flash:low__json_schema PASS 100% 198 in → 248 out (214 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemini-3.5-flash:medium PASS 100% 119 in → 387 out (372 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.5-flash:medium__json_schema PASS 100% 194 in → 444 out (423 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
google/gemini-3.5-flash:xhigh PASS 100% 117 in → 474 out (459 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 193 in → 379 out (360 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.6-flash:high PASS 100% 120 in → 480 out (465 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.6-flash:high__json_schema PASS 100% 198 in → 812 out (793 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.6-flash:low PASS 100% 120 in → 355 out (340 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.6-flash:low__json_schema PASS 100% 116 in → 10 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemini-3.6-flash:max PASS 100% 120 in → 2,771 out (2,756 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.6-flash:max__json_schema PASS 100% 119 in → 699 out (680 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.6-flash:medium PASS 100% 119 in → 518 out (503 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.6-flash:medium__json_schema PASS 100% 120 in → 608 out (589 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemini-3.6-flash:xhigh PASS 100% 117 in → 444 out (429 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 198 in → 1,073 out (1,054 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemma-4-26b-a4b-it:high PASS 100% 135 in → 376 out (360 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 133 in → 306 out (234 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 134 in → 335 out (267 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 132 in → 263 out (193 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 135 in → 371 out (320 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 132 in → 310 out (236 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 128 in → 16 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 131 in → 31 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 134 in → 555 out (539 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 134 in → 264 out (191 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemma-4-31b-it:high PASS 100% 133 in → 226 out (178 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemma-4-31b-it:high__json_schema PASS 100% 132 in → 259 out (1 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
google/gemma-4-31b-it:low PASS 100% 134 in → 281 out (225 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemma-4-31b-it:low__json_schema PASS 100% 134 in → 248 out (227 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
google/gemma-4-31b-it:medium PASS 100% 136 in → 219 out (203 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemma-4-31b-it:medium__json_schema PASS 100% 129 in → 274 out (221 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
google/gemma-4-31b-it:none PASS 100% 129 in → 16 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemma-4-31b-it:none__json_schema PASS 100% 130 in → 22 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
google/gemma-4-31b-it:xhigh PASS 100% 134 in → 235 out (193 reasoning)
WIP: debug commit 2 WIP: debug commit 1
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 133 in → 252 out (191 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
ibm-granite/granite-4.1-8b PASS 100% 122 in → 16 out
WIP: debug commit 2 WIP: debug commit 1
ibm-granite/granite-4.1-8b__json_schema PASS 100% 121 in → 32 out
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
inclusionai/ling-2.6-flash PASS 100% 138 in → 16 out
WIP: debug commit 2 WIP: debug commit 1
inclusionai/ling-2.6-flash__json_schema PASS 100% 136 in → 36 out
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
minimax/minimax-m2.5:high PASS 100% 149 in → 190 out (176 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m2.5:high__json_schema PASS 100% 149 in → 247 out (225 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
minimax/minimax-m2.5:low PASS 100% 149 in → 212 out (197 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m2.5:xhigh PASS 100% 148 in → 197 out (198 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m2.7:high PASS 100% 153 in → 237 out (221 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m2.7:low PASS 100% 148 in → 326 out (308 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m2.7:low__json_schema PASS 100% 307 in → 486 out (463 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
minimax/minimax-m2.7:medium PASS 100% 148 in → 191 out (182 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m2.7:medium__json_schema PASS 100% 299 in → 989 out (957 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
minimax/minimax-m2.7:xhigh PASS 100% 148 in → 485 out (497 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m3:high PASS 100% 288 in → 137 out (116 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m3:high__json_schema PASS 100% 274 in → 24 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
minimax/minimax-m3:low PASS 100% 287 in → 159 out (134 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m3:low__json_schema PASS 100% 274 in → 31 out (0 reasoning)
WIP: debug commit 1 WIP: debug commit 2
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 1", "WIP: debug commit 2" ] }
minimax/minimax-m3:medium PASS 100% 289 in → 131 out (108 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m3:xhigh PASS 100% 288 in → 163 out (149 reasoning)
WIP: debug commit 2 WIP: debug commit 1
minimax/minimax-m3:xhigh__json_schema PASS 100% 277 in → 31 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
mistralai/devstral-2512__json_schema PASS 100% 124 in → 24 out
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
mistralai/mistral-medium-3-5:high PASS 100% 137 in → 630 out (578 reasoning)
WIP: debug commit 2 WIP: debug commit 1
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 123 in → 300 out (258 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
mistralai/mistral-medium-3-5:low PASS 100% 135 in → 322 out (272 reasoning)
WIP: debug commit 2 WIP: debug commit 1
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 125 in → 357 out (298 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
mistralai/mistral-medium-3-5:medium PASS 100% 135 in → 515 out (487 reasoning)
WIP: debug commit 2 WIP: debug commit 1
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 124 in → 332 out (296 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 121 in → 24 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 135 in → 248 out (219 reasoning)
WIP: debug commit 2 WIP: debug commit 1
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 125 in → 881 out (906 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
moonshotai/kimi-k2.6:high PASS 100% 116 in → 186 out (169 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k2.6:high__json_schema PASS 100% 115 in → 469 out (405 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
moonshotai/kimi-k2.6:low PASS 100% 121 in → 306 out (289 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k2.6:low__json_schema PASS 100% 116 in → 1,158 out (947 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits" : [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
moonshotai/kimi-k2.6:medium PASS 100% 116 in → 405 out (388 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k2.6:xhigh PASS 100% 114 in → 261 out (272 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 308 in → 1,011 out (989 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
moonshotai/kimi-k2.7-code:high PASS 100% 115 in → 128 out (107 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k2.7-code:low PASS 100% 113 in → 154 out (137 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 115 in → 171 out (143 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
moonshotai/kimi-k2.7-code:medium PASS 100% 117 in → 136 out (119 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 213 in → 169 out (145 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 117 in → 160 out (143 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 117 in → 194 out (163 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
moonshotai/kimi-k3:high PASS 100% 193 in → 131 out (101 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k3:high__json_schema PASS 100% 196 in → 98 out (68 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k3:low PASS 100% 196 in → 104 out (74 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k3:low__json_schema PASS 100% 195 in → 186 out (156 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k3:max PASS 100% 195 in → 185 out (155 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k3:max__json_schema PASS 100% 195 in → 203 out (173 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k3:medium PASS 100% 194 in → 135 out (105 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k3:medium__json_schema PASS 100% 194 in → 164 out (134 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k3:xhigh PASS 100% 197 in → 155 out (125 reasoning)
WIP: debug commit 2 WIP: debug commit 1
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 198 in → 169 out (139 reasoning)
WIP: debug commit 2 WIP: debug commit 1
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 137 in → 344 out (316 reasoning)
WIP: debug commit 2 WIP: debug commit 1
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 139 in → 343 out (320 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 132 in → 434 out (430 reasoning)
WIP: debug commit 2 WIP: debug commit 1
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 135 in → 331 out (318 reasoning)
WIP: debug commit 2 WIP: debug commit 1
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 138 in → 266 out (233 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 137 in → 492 out (458 reasoning)
WIP: debug commit 2 WIP: debug commit 1
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 135 in → 421 out (406 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 134 in → 176 out (151 reasoning)
WIP: debug commit 2 WIP: debug commit 1
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 137 in → 89 out (57 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 136 in → 197 out (168 reasoning)
WIP: debug commit 2 WIP: debug commit 1
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 136 in → 226 out (182 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 136 in → 213 out (176 reasoning)
WIP: debug commit 2 WIP: debug commit 1
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 139 in → 247 out (194 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 138 in → 31 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 137 in → 286 out (252 reasoning)
WIP: debug commit 2 WIP: debug commit 1
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 137 in → 175 out (142 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
openai/gpt-5.4-mini:high PASS 100% 117 in → 83 out (62 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4-mini:high__json_schema PASS 100% 163 in → 173 out (142 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4-mini:low PASS 100% 117 in → 42 out (21 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4-mini:low__json_schema PASS 100% 163 in → 77 out (46 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4-mini:medium PASS 100% 117 in → 156 out (135 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4-mini:medium__json_schema PASS 100% 165 in → 273 out (242 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4-mini:none PASS 100% 115 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4-mini:none__json_schema PASS 100% 163 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4-mini:xhigh PASS 100% 119 in → 265 out (244 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 162 in → 547 out (516 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4-nano:high PASS 100% 116 in → 63 out (42 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4-nano:high__json_schema PASS 100% 164 in → 66 out (35 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4-nano:low PASS 100% 119 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4-nano:low__json_schema PASS 100% 166 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4-nano:medium__json_schema PASS 100% 162 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4-nano:none__json_schema PASS 100% 163 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4-nano:xhigh PASS 100% 116 in → 163 out (142 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 162 in → 527 out (496 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4:high PASS 100% 116 in → 109 out (88 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4:high__json_schema PASS 100% 163 in → 88 out (57 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4:low PASS 100% 116 in → 43 out (22 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4:low__json_schema PASS 100% 166 in → 51 out (20 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4:medium PASS 100% 117 in → 76 out (55 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4:medium__json_schema PASS 100% 164 in → 82 out (51 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4:none PASS 100% 115 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4:none__json_schema PASS 100% 166 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.4:xhigh PASS 100% 119 in → 148 out (127 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.4:xhigh__json_schema PASS 100% 165 in → 148 out (117 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.5:high PASS 100% 116 in → 70 out (49 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.5:high__json_schema PASS 100% 165 in → 135 out (104 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.5:low PASS 100% 113 in → 37 out (16 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.5:low__json_schema PASS 100% 167 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.5:medium PASS 100% 114 in → 59 out (38 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.5:medium__json_schema PASS 100% 164 in → 57 out (26 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.5:none PASS 100% 116 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.5:none__json_schema PASS 100% 168 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.5:xhigh PASS 100% 118 in → 99 out (78 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.5:xhigh__json_schema PASS 100% 163 in → 143 out (112 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-luna:high PASS 100% 118 in → 40 out (19 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-luna:high__json_schema PASS 100% 166 in → 51 out (20 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-luna:low PASS 100% 116 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-luna:low__json_schema PASS 100% 163 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-luna:max PASS 100% 116 in → 159 out (138 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-luna:max__json_schema PASS 100% 160 in → 81 out (50 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-luna:medium PASS 100% 117 in → 41 out (20 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-luna:medium__json_schema PASS 100% 166 in → 51 out (20 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-luna:none PASS 100% 116 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-luna:none__json_schema PASS 100% 161 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-luna:xhigh PASS 100% 115 in → 68 out (47 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 165 in → 351 out (320 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-sol:high PASS 100% 115 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-sol:high__json_schema PASS 100% 165 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-sol:low PASS 100% 118 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-sol:low__json_schema PASS 100% 162 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-sol:max PASS 100% 118 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-sol:max__json_schema PASS 100% 162 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-sol:medium PASS 100% 113 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-sol:medium__json_schema PASS 100% 164 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-sol:none PASS 100% 115 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-sol:none__json_schema PASS 100% 164 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-sol:xhigh PASS 100% 113 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 167 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-terra:high PASS 100% 115 in → 32 out (11 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-terra:high__json_schema PASS 100% 166 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-terra:low PASS 100% 115 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-terra:low__json_schema PASS 100% 167 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-terra:max PASS 100% 115 in → 47 out (26 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-terra:max__json_schema PASS 100% 163 in → 85 out (54 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-terra:medium PASS 100% 116 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-terra:medium__json_schema PASS 100% 168 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-terra:none PASS 100% 116 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-terra:none__json_schema PASS 100% 163 in → 29 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-5.6-terra:xhigh PASS 100% 116 in → 39 out (18 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 163 in → 48 out (17 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-oss-120b:high PASS 100% 176 in → 261 out (237 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-oss-120b:high__json_schema PASS 100% 165 in → 410 out (418 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-oss-120b:low PASS 100% 162 in → 62 out (34 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-oss-120b:low__json_schema PASS 100% 177 in → 174 out (131 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-oss-120b:medium PASS 100% 166 in → 175 out (156 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-oss-120b:medium__json_schema PASS 100% 173 in → 91 out (59 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
openai/gpt-oss-120b:xhigh PASS 100% 182 in → 84 out (59 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 173 in → 297 out (281 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-oss-20b:high PASS 100% 163 in → 235 out (228 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-oss-20b:high__json_schema PASS 100% 163 in → 557 out (557 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-oss-20b:low PASS 100% 166 in → 53 out (23 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-oss-20b:low__json_schema PASS 100% 176 in → 82 out (41 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
openai/gpt-oss-20b:medium PASS 100% 176 in → 112 out (97 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-oss-20b:medium__json_schema PASS 100% 241 in → 329 out (324 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
openai/gpt-oss-20b:xhigh PASS 100% 178 in → 146 out (131 reasoning)
WIP: debug commit 2 WIP: debug commit 1
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 179 in → 324 out (319 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
poolside/laguna-m.1:high PASS 100% 126 in → 278 out (262 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-m.1:high__json_schema PASS 100% 129 in → 335 out (313 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
poolside/laguna-m.1:low PASS 100% 132 in → 273 out (257 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-m.1:low__json_schema PASS 100% 128 in → 325 out (304 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
poolside/laguna-m.1:medium PASS 100% 130 in → 218 out (202 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-m.1:medium__json_schema PASS 100% 126 in → 317 out (282 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
poolside/laguna-m.1:none PASS 100% 132 in → 15 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-m.1:xhigh PASS 100% 128 in → 262 out (246 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-m.1:xhigh__json_schema PASS 100% 129 in → 328 out (295 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
poolside/laguna-xs-2.1:high PASS 100% 127 in → 280 out (266 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-xs-2.1:high__json_schema PASS 100% 131 in → 291 out (256 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
poolside/laguna-xs-2.1:low PASS 100% 130 in → 205 out (191 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-xs-2.1:low__json_schema PASS 100% 127 in → 225 out (194 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
poolside/laguna-xs-2.1:medium PASS 100% 127 in → 309 out (295 reasoning)
WIP: debug commit 1 WIP: debug commit 2
poolside/laguna-xs-2.1:none PASS 100% 129 in → 13 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-xs-2.1:xhigh PASS 100% 128 in → 255 out (241 reasoning)
WIP: debug commit 1 WIP: debug commit 2
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 132 in → 295 out (264 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
poolside/laguna-xs.2:high PASS 100% 169 in → 237 out (221 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-xs.2:high__json_schema PASS 100% 167 in → 160 out (125 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
poolside/laguna-xs.2:low PASS 100% 167 in → 212 out (196 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-xs.2:low__json_schema PASS 100% 167 in → 182 out (161 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
poolside/laguna-xs.2:medium PASS 100% 164 in → 194 out (178 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-xs.2:medium__json_schema PASS 100% 164 in → 491 out (469 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
poolside/laguna-xs.2:none PASS 100% 163 in → 15 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-xs.2:none__json_schema PASS 100% 167 in → 21 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
poolside/laguna-xs.2:xhigh PASS 100% 165 in → 175 out (159 reasoning)
WIP: debug commit 2 WIP: debug commit 1
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 168 in → 192 out (171 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
qwen/qwen3.6-27b:high PASS 100% 130 in → 830 out (765 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-27b:high__json_schema PASS 100% 125 in → 1,052 out (952 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
qwen/qwen3.6-27b:low PASS 100% 130 in → 1,047 out (967 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-27b:low__json_schema PASS 100% 125 in → 1,044 out (1 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
qwen/qwen3.6-27b:medium PASS 100% 123 in → 1,063 out (1,012 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-27b:medium__json_schema PASS 100% 127 in → 1,044 out (929 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
qwen/qwen3.6-27b:none PASS 100% 130 in → 16 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-27b:xhigh PASS 100% 128 in → 951 out (931 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 130 in → 837 out (755 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: debug commit 2", "WIP: debug commit 1"] }
qwen/qwen3.6-35b-a3b:high PASS 100% 131 in → 1,026 out (1,006 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 130 in → 1,002 out (897 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 128 in → 1,060 out (1,040 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 128 in → 1,070 out (986 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 127 in → 1,107 out (1,062 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 130 in → 1,485 out (1,355 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 133 in → 15 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 129 in → 23 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: debug commit 2", "WIP: debug commit 1"]}
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 130 in → 1,027 out (960 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 131 in → 1,330 out (1,197 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
qwen/qwen3.6-flash:high PASS 100% 128 in → 717 out (697 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-flash:low PASS 100% 128 in → 1,104 out (1,084 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-flash:medium PASS 100% 128 in → 1,365 out (1,345 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-flash:none PASS 100% 134 in → 15 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.6-flash:xhigh PASS 100% 128 in → 1,157 out (1,137 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-flash:high PASS 100% 130 in → 1,029 out (1,009 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-flash:low PASS 100% 129 in → 1,084 out (1,064 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-flash:max PASS 100% 130 in → 1,105 out (1,085 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-flash:medium PASS 100% 132 in → 1,062 out (1,042 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-flash:none PASS 100% 133 in → 15 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-flash:xhigh PASS 100% 130 in → 1,135 out (1,115 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-max:high PASS 100% 130 in → 164 out (144 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-max:low PASS 100% 129 in → 339 out (319 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-max:medium PASS 100% 128 in → 401 out (381 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-max:none PASS 100% 130 in → 15 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-max:xhigh PASS 100% 127 in → 430 out (410 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-plus:high PASS 100% 129 in → 749 out (729 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-plus:low PASS 100% 127 in → 840 out (820 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-plus:medium PASS 100% 128 in → 600 out (580 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-plus:none PASS 100% 130 in → 15 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
qwen/qwen3.7-plus:xhigh PASS 100% 127 in → 817 out (797 reasoning)
WIP: debug commit 2 WIP: debug commit 1
tencent/hy3:high PASS 100% 121 in → 429 out (412 reasoning)
WIP: debug commit 2 WIP: debug commit 1
tencent/hy3:high__json_schema PASS 100% 122 in → 415 out (367 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: debug commit 2", "WIP: debug commit 1"] }
tencent/hy3:low PASS 100% 121 in → 224 out (207 reasoning)
WIP: debug commit 2 WIP: debug commit 1
tencent/hy3:low__json_schema PASS 100% 123 in → 210 out (164 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
tencent/hy3:medium PASS 100% 120 in → 254 out (237 reasoning)
WIP: debug commit 2 WIP: debug commit 1
tencent/hy3:medium__json_schema PASS 100% 123 in → 289 out (231 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: debug commit 2", "WIP: debug commit 1"] }
tencent/hy3:none PASS 100% 125 in → 16 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
tencent/hy3:none__json_schema PASS 100% 126 in → 30 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
tencent/hy3:xhigh PASS 100% 124 in → 521 out (504 reasoning)
WIP: debug commit 2 WIP: debug commit 1
tencent/hy3:xhigh__json_schema PASS 100% 122 in → 246 out (216 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: debug commit 2", "WIP: debug commit 1"] }
thinkingmachines/inkling-small:high PASS 100% 125 in → 202 out (181 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling-small:high__json_schema PASS 100% 125 in → 315 out (280 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
thinkingmachines/inkling-small:low PASS 100% 122 in → 50 out (30 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling-small:low__json_schema PASS 100% 129 in → 66 out (37 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: debug commit 2", "WIP: debug commit 1"] }
thinkingmachines/inkling-small:max PASS 100% 124 in → 279 out (258 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling-small:max__json_schema PASS 100% 123 in → 208 out (177 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
thinkingmachines/inkling-small:medium PASS 100% 126 in → 203 out (182 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling-small:medium__json_schema PASS 100% 128 in → 250 out (215 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 129 in → 254 out (233 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 124 in → 380 out (351 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: debug commit 2", "WIP: debug commit 1"] }
thinkingmachines/inkling:high PASS 100% 124 in → 233 out (211 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling:high__json_schema PASS 100% 125 in → 286 out (251 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
thinkingmachines/inkling:low PASS 100% 127 in → 107 out (86 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling:max PASS 100% 128 in → 179 out (157 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling:max__json_schema PASS 100% 125 in → 253 out (227 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
thinkingmachines/inkling:medium PASS 100% 127 in → 107 out (85 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling:medium__json_schema PASS 100% 125 in → 214 out (179 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
thinkingmachines/inkling:none PASS 100% 122 in → 18 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling:xhigh PASS 100% 126 in → 177 out (156 reasoning)
WIP: debug commit 2 WIP: debug commit 1
thinkingmachines/inkling:xhigh__json_schema PASS 100% 125 in → 250 out (214 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
x-ai/grok-4.3:high PASS 100% 294 in → 472 out (457 reasoning)
WIP: debug commit 2 WIP: debug commit 1
x-ai/grok-4.3:high__json_schema PASS 100% 358 in → 723 out (704 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
x-ai/grok-4.3:low PASS 100% 295 in → 308 out (293 reasoning)
WIP: debug commit 2 WIP: debug commit 1
x-ai/grok-4.3:low__json_schema PASS 100% 364 in → 424 out (395 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
x-ai/grok-4.3:max PASS 100% 293 in → 391 out (376 reasoning)
WIP: debug commit 2 WIP: debug commit 1
x-ai/grok-4.3:max__json_schema PASS 100% 357 in → 598 out (579 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
x-ai/grok-4.3:medium PASS 100% 297 in → 372 out (357 reasoning)
WIP: debug commit 2 WIP: debug commit 1
x-ai/grok-4.3:medium__json_schema PASS 100% 364 in → 468 out (439 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
x-ai/grok-4.3:none__json_schema PASS 100% 356 in → 19 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
x-ai/grok-4.3:xhigh PASS 100% 292 in → 335 out (320 reasoning)
WIP: debug commit 2 WIP: debug commit 1
x-ai/grok-4.3:xhigh__json_schema PASS 100% 361 in → 660 out (631 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
x-ai/grok-4.5:high PASS 100% 315 in → 201 out (188 reasoning)
WIP: debug commit 2 WIP: debug commit 1
x-ai/grok-4.5:high__json_schema PASS 100% 392 in → 582 out (564 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
x-ai/grok-4.5:low PASS 100% 316 in → 155 out (142 reasoning)
WIP: debug commit 2 WIP: debug commit 1
x-ai/grok-4.5:low__json_schema PASS 100% 394 in → 349 out (321 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
x-ai/grok-4.5:max PASS 100% 313 in → 157 out (144 reasoning)
WIP: debug commit 2 WIP: debug commit 1
x-ai/grok-4.5:max__json_schema PASS 100% 395 in → 452 out (434 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
x-ai/grok-4.5:medium PASS 100% 312 in → 247 out (234 reasoning)
WIP: debug commit 2 WIP: debug commit 1
x-ai/grok-4.5:medium__json_schema PASS 100% 395 in → 570 out (552 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
x-ai/grok-4.5:xhigh PASS 100% 315 in → 192 out (179 reasoning)
WIP: debug commit 2 WIP: debug commit 1
x-ai/grok-4.5:xhigh__json_schema PASS 100% 393 in → 485 out (467 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
z-ai/glm-4.7-flash:high PASS 100% 119 in → 465 out (382 reasoning)
WIP: debug commit 2 WIP: debug commit 1
z-ai/glm-4.7-flash:high__json_schema PASS 100% 117 in → 1,196 out (1,112 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
z-ai/glm-4.7-flash:low PASS 100% 119 in → 451 out (378 reasoning)
WIP: debug commit 2 WIP: debug commit 1
z-ai/glm-4.7-flash:medium PASS 100% 118 in → 742 out (708 reasoning)
WIP: debug commit 2 WIP: debug commit 1
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 120 in → 511 out (405 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
z-ai/glm-4.7-flash:none PASS 100% 118 in → 15 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
z-ai/glm-4.7-flash:none__json_schema PASS 100% 120 in → 31 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
z-ai/glm-4.7-flash:xhigh PASS 100% 118 in → 381 out (323 reasoning)
WIP: debug commit 2 WIP: debug commit 1
z-ai/glm-5.2:high PASS 100% 126 in → 437 out (384 reasoning)
WIP: debug commit 2 WIP: debug commit 1
z-ai/glm-5.2:low PASS 100% 124 in → 332 out (316 reasoning)
WIP: debug commit 2 WIP: debug commit 1
z-ai/glm-5.2:medium PASS 100% 124 in → 270 out (253 reasoning)
WIP: debug commit 2 WIP: debug commit 1
z-ai/glm-5.2:none PASS 100% 120 in → 16 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
z-ai/glm-5.2:none__json_schema PASS 100% 119 in → 21 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["WIP: debug commit 2","WIP: debug commit 1"]}
z-ai/glm-5.2:xhigh PASS 100% 124 in → 466 out (441 reasoning)
WIP: debug commit 2 WIP: debug commit 1
z-ai/glm-5.2:xhigh__json_schema PASS 100% 125 in → 394 out (345 reasoning)
WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
arcee-ai/trinity-mini:medium FAIL 50% 118 in → 333 out (345 reasoning)
WIP:debug commit 2 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 2']; Extra selected commit messages: ['WIP:debug commit 2']
liquid/lfm-2-24b-a2b FAIL 50% 128 in → 22 out
WIP: debug commit 2 d69a1a5 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1']; Extra selected commit messages: ['WIP: debug commit 1']
thinkingmachines/inkling-small:none FAIL 50% 127 in → 18 out (0 reasoning)
WIP: debug commit 2 WIP: debug commit 1<|end_message|>
Failure: Missing expected commit messages: ['WIP: debug commit 1']; Extra selected commit messages: ['WIP: debug commit 1']
anthropic/claude-haiku-4.5:low FAIL 0% 154 in → 272 out (217 reasoning)
3f143cb WIP: debug commit 2 b99e6d3 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
anthropic/claude-haiku-4.5:none FAIL 0% 124 in → 30 out (0 reasoning)
3802bd3 WIP: debug commit 2 6c3e790 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 314 in → 21 out (0 reasoning)
a99daa6 b58ed5d
JSON Schema Structured Output
(raw) {"commits": ["a99daa6", "b58ed5d"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
anthropic/claude-opus-4.6:low__json_schema FAIL 0% 312 in → 37 out (0 reasoning)
fa3a9a5 WIP: debug commit 2 dba8ed1 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["fa3a9a5 WIP: debug commit 2","dba8ed1 WIP: debug commit 1"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
anthropic/claude-sonnet-4.6:high FAIL 0% 124 in → 27 out (0 reasoning)
dd7d273 WIP: debug commit 2 e42fdae WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 412 in → 45 out (0 reasoning)
987ebf8 WIP: debug commit 2 b5c2af3 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["987ebf8 WIP: debug commit 2","b5c2af3 WIP: debug commit 1"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
anthropic/claude-sonnet-5:medium FAIL 0% 167 in → 56 out (21 reasoning)
859854 WIP: debug commit 2 7506c8f WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 408 in → 42 out (0 reasoning)
b65b740 WIP: debug commit 2 88a6005 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["b65b740 WIP: debug commit 2","88a6005 WIP: debug commit 1"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high__json_schema FAIL 0% 116 in → 369 out (365 reasoning)
04efc09 WIP: debug commit 2 04d84bb WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "04efc09 WIP: debug commit 2", "04d84bb WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 118 in → 288 out (240 reasoning)
a6931ed Remove debug statements 6e8bf76 WIP: debug commit 2 48d69b2 WIP: debug commit 1 7a45cfc Add real feature
JSON Schema Structured Output
(raw) { "commits": [ "a6931ed Remove debug statements", "6e8bf76 WIP: debug commit 2", "48d69b2 WIP: debug commit 1", "7a45cfc Add real feature" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['Remove debug statements', 'WIP: debug commit 2', 'WIP: debug commit 1', 'Add real feature']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 117 in → 414 out (417 reasoning)
deef170 WIP: debug commit 2 193fca8 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "deef170 WIP: debug commit 2", "193fca8 WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 117 in → 327 out (325 reasoning)
a18c8cc WIP: debug commit 2 0730cca WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "a18c8cc WIP: debug commit 2", "0730cca WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash-0731:high FAIL 0% 114 in → 72 out (55 reasoning)
524f15e WIP: debug commit 2 d93182d WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash-0731:high__json_schema FAIL 0% 112 in → 93 out (63 reasoning)
d78f240 WIP: debug commit 2 d71d124 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "d78f240 WIP: debug commit 2", "d71d124 WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash-0731:low FAIL 0% 113 in → 52 out (27 reasoning)
ca16775 WIP: debug commit 2 02a9b4e WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 0% 115 in → 67 out (35 reasoning)
dc6e3a8 WIP: debug commit 2 feaa939 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["dc6e3a8 WIP: debug commit 2", "feaa939 WIP: debug commit 1"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash-0731:medium FAIL 0% 113 in → 54 out (29 reasoning)
0bc4b34 WIP: debug commit 2 c882fe3 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash-0731:medium__json_schema FAIL 0% 112 in → 63 out (40 reasoning)
150e343 WIP: debug commit 2 3232b36 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": ["150e343 WIP: debug commit 2", "3232b36 WIP: debug commit 1"] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 112 in → 24 out (0 reasoning)
e64de64 WIP: debug commit 2 6c40c81 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 114 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "commits": [] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 111 in → 59 out (31 reasoning)
1906887 WIP: debug commit 2 feb4926 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": ["1906887 WIP: debug commit 2", "feb4926 WIP: debug commit 1"] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash:high__json_schema FAIL 0% 111 in → 128 out (87 reasoning)
ea9264c WIP: debug commit 2 62cb2f8 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "ea9264c WIP: debug commit 2", "62cb2f8 WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash:low__json_schema FAIL 0% 117 in → 187 out (145 reasoning)
fffcfe4 WIP: debug commit 2 f9efc5f WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "fffcfe4 WIP: debug commit 2", "f9efc5f WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash:medium__json_schema FAIL 0% 121 in → 184 out (142 reasoning)
16f2e1c WIP: debug commit 2 c83c529 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "16f2e1c WIP: debug commit 2", "c83c529 WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 219 in → 31 out (0 reasoning)
cedcb49 WIP: debug commit 2 f571c32 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["cedcb49 WIP: debug commit 2", "f571c32 WIP: debug commit 1"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
Invalid JSON. Output: WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:medium FAIL 0% 115 in → 26 out (0 reasoning)
3b29e25 WIP: debug commit 2 6a71bfd WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 115 in → 244 out (205 reasoning)
7bfcabc WIP: debug commit 2 645b326 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "7bfcabc WIP: debug commit 2", "645b326 WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
deepseek/deepseek-v4-pro:none FAIL 0% 114 in → 23 out (0 reasoning)
4c81983 WIP: debug commit 2 9db1493 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 0% 195 in → 141 out (121 reasoning)
1808bf6 5712e43
JSON Schema Structured Output
(raw) {"commits": ["1808bf6", "5712e43"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
google/gemini-3.1-flash-lite-preview:xhigh__json_schema FAIL 0% 120 in → 706 out (676 reasoning)
045d8f2 WIP: debug commit 2 335cd51 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["045d8f2 WIP: debug commit 2","335cd51 WIP: debug commit 1"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
Invalid JSON. Output: 040fbc3 WIP: debug commit 2 4476e3d WIP: debug commit 1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
minimax/minimax-m2.5:low__json_schema FAIL 0% 158 in → 218 out (176 reasoning)
d03bf7a WIP: debug commit 2 f4f52f0 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "d03bf7a WIP: debug commit 2", "f4f52f0 WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
minimax/minimax-m2.5:medium FAIL 0% 147 in → 212 out (189 reasoning)
1304cec WIP: debug commit 2 fb76261 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
minimax/minimax-m2.5:medium__json_schema FAIL 0% 154 in → 276 out (220 reasoning)
b683c70 WIP: debug commit 2 bd4ed7b WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "b683c70 WIP: debug commit 2", "bd4ed7b WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
minimax/minimax-m2.5:xhigh__json_schema FAIL 0% 148 in → 174 out (140 reasoning)
f15b01d WIP: debug commit 2 67d211d WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": ["f15b01d WIP: debug commit 2", "67d211d WIP: debug commit 1"] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:medium__json_schema FAIL 0% 274 in → 40 out (0 reasoning)
7b33f60 WIP: debug commit 2 0035d37 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "7b33f60 WIP: debug commit 2", "0035d37 WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
mistralai/devstral-2512 FAIL 0% 126 in → 30 out
b8d805f WIP: debug commit 2 1b562e8 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
mistralai/mistral-medium-3-5:none FAIL 0% 136 in → 30 out (0 reasoning)
b7a374a WIP: debug commit 2 8615419 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
moonshotai/kimi-k2.6:medium__json_schema FAIL 0% 116 in → 540 out (436 reasoning)
3bb0214 WIP: debug commit 2 731679c WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "3bb0214 WIP: debug commit 2", "731679c WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
moonshotai/kimi-k2.7-code:high__json_schema FAIL 0% 115 in → 179 out (151 reasoning)
485c068 WIP: debug commit 2 eb463ae WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "485c068 WIP: debug commit 2", "eb463ae WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 135 in → 195 out (136 reasoning)
e1cfad8 WIP: debug commit 2 97490c4 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "e1cfad8 WIP: debug commit 2", "97490c4 WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 138 in → 40 out (0 reasoning)
762336a Remove debug statements c3d9681 WIP: debug commit 2 185a0fd WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['Remove debug statements', 'WIP: debug commit 2', 'WIP: debug commit 1']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 136 in → 22 out (0 reasoning)
a54e2e1 30dd943
JSON Schema Structured Output
(raw) {"commits":["a54e2e1","30dd943"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']
nvidia/nemotron-3-super-120b-a12b:none FAIL 0% 137 in → 28 out (0 reasoning)
c235d16 WIP: debug commit 2 dd75afb WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
openai/gpt-5.4-nano:medium FAIL 0% 115 in → 25 out (0 reasoning)
705d413 WIP: debug commit 2 8122739 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
openai/gpt-5.4-nano:none FAIL 0% 119 in → 29 out (0 reasoning)
f6e92b4 WIP: debug commit 2 3831f99 WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
poolside/laguna-m.1:none__json_schema FAIL 0% 130 in → 35 out (0 reasoning)
e0c389d WIP: debug commit 2 15d2942 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["e0c389d WIP: debug commit 2", "15d2942 WIP: debug commit 1"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
poolside/laguna-xs-2.1:medium__json_schema FAIL 0% 129 in → 308 out (275 reasoning)
4017381 WIP: debug commit 2 5fdf669 WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits": ["4017381 WIP: debug commit 2", "5fdf669 WIP: debug commit 1"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 128 in → 46 out (0 reasoning)
f9a6ded WIP: debug commit 2 84f6d88 WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "f9a6ded WIP: debug commit 2", "84f6d88 WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 129 in → 39 out (0 reasoning)
cf620c7 WIP: debug commit 2 a8350bb WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": ["cf620c7 WIP: debug commit 2", "a8350bb WIP: debug commit 1"] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a274511f-e788-93a7-b329-0de8d358a96a","request_id":"a274511f-e788-93a7-b329-0de8d358a96a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fee395bc-5791-97a0-9325-acf3ee4dbc65","request_id":"fee395bc-5791-97a0-9325-acf3ee4dbc65"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2c68729b-b787-9263-95ba-b8dd1048b56e","request_id":"2c68729b-b787-9263-95ba-b8dd1048b56e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6a19894a-4720-9f54-b451-87d4c2afd458","request_id":"6a19894a-4720-9f54-b451-87d4c2afd458"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-16da71a7-f55a-9c4f-b5fd-9e003e4d0bab","request_id":"16da71a7-f55a-9c4f-b5fd-9e003e4d0bab"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-eb04f465-0aef-9e9d-9d99-9346bb5ff4d2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b1d0a85f-bc6e-99b8-aa7b-fe359011d28f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-207bc391-6b5b-9b70-bc23-0bb8c43bbfa9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7e49d918-cd58-97ed-b38a-0a1a28899c04"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8a09c389-d7c7-9443-9407-73c6b52e91f2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e95e39eb-5b9b-9844-99df-6bb5d31a7094"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b0c9cfbe-bd05-9028-960c-cb67b448b0e0","request_id":"b0c9cfbe-bd05-9028-960c-cb67b448b0e0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-503c93ce-fa06-9ca6-a677-a7f74d6bffdd","request_id":"503c93ce-fa06-9ca6-a677-a7f74d6bffdd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-99931de9-f571-9a30-b2c5-ea9820e6ef39","request_id":"99931de9-f571-9a30-b2c5-ea9820e6ef39"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d7307ef9-430b-9928-82fe-e652fa538161","request_id":"d7307ef9-430b-9928-82fe-e652fa538161"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c6cb586a-ab5f-96c3-96f6-f9b96d81381b","request_id":"c6cb586a-ab5f-96c3-96f6-f9b96d81381b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d621201f-de40-90cd-a00e-ceb1d3be677a","request_id":"d621201f-de40-90cd-a00e-ceb1d3be677a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-aed0b2c8-336b-9ff1-9185-dfcbaf5e810c","request_id":"aed0b2c8-336b-9ff1-9185-dfcbaf5e810c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-388b1727-c87e-9d39-a4d9-33c0c5cd23d9","request_id":"388b1727-c87e-9d39-a4d9-33c0c5cd23d9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-14950952-572b-9322-9a93-62fde738514c","request_id":"14950952-572b-9322-9a93-62fde738514c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a1278c1f-4a0b-941e-94be-fbedaff99e85","request_id":"a1278c1f-4a0b-941e-94be-fbedaff99e85"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: WIP: debug commit 2 WIP: debug commit 1{ "commits": [ "3765a32", "317aae9" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: debug commit 2 WIP: debug commit 1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: debug commit 2 WIP: debug commit 1{ "commits": [ "WIP: debug commit 2", "WIP: debug commit 1" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none FAIL 0% 289 in → 21 out (0 reasoning)
7ffcef4 WIP: debug commit 2 c06981c WIP: debug commit 1
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:high__json_schema FAIL 0% 310 in → 321 out (290 reasoning)
bc33cc9 WIP: debug commit 2 24dab1d WIP: debug commit 1
JSON Schema Structured Output
(raw) {"commits":["bc33cc9 WIP: debug commit 2","24dab1d WIP: debug commit 1"]}
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
z-ai/glm-5.2:low__json_schema FAIL 0% 127 in → 899 out (858 reasoning)
f7848e0 WIP: debug commit 2 7b640cc WIP: debug commit 1
JSON Schema Structured Output
(raw) { "commits": [ "f7848e0 WIP: debug commit 2", "7b640cc WIP: debug commit 1" ] }
Failure: Missing expected commit messages: ['WIP: debug commit 1', 'WIP: debug commit 2']; Extra selected commit messages: ['WIP: debug commit 2', 'WIP: debug commit 1']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)