Documentation then fixup pattern
Tests ability to squash documentation-only fixup commits into the implementation commit. Evaluates recognizing doc-then-fixup patterns.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'content' > main.md
  5. 05 git add main.md
  6. 06 git commit -m 'Add documentation'
  7. 07 echo 'content more' > main.md
  8. 08 git add main.md
  9. 09 git commit -m 'WIP: update docs'
  10. 10 echo 'content more done' > main.md
  11. 11 git add main.md
  12. 12 git commit -m 'WIP: finalize docs'
Prompt
Identify the commits that represent incremental work-in-progress updates to documentation and should be marked as squash/fixup. Do not include the documentation anchor commit they would be squashed into. Output ONLY the selected commit subject lines, one per line, with no explanation.
Expected
WIP: update docs WIP: finalize docs

Scoped model quality, cost, API time, and token usage for commit_squash/f006.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 141 in → 60 out (39 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-fable-5:high__json_schema PASS 100% 382 in → 56 out (22 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
anthropic/claude-fable-5:low PASS 100% 141 in → 43 out (23 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-fable-5:low__json_schema PASS 100% 385 in → 30 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
anthropic/claude-fable-5:max PASS 100% 141 in → 274 out (84 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-fable-5:max__json_schema PASS 100% 377 in → 241 out (75 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
anthropic/claude-fable-5:medium PASS 100% 143 in → 45 out (22 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-fable-5:medium__json_schema PASS 100% 380 in → 40 out (13 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-fable-5:xhigh PASS 100% 141 in → 67 out (31 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 387 in → 79 out (44 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
anthropic/claude-haiku-4.5:high PASS 100% 136 in → 377 out (353 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-haiku-4.5:low PASS 100% 132 in → 235 out (223 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 321 in → 866 out (569 reasoning)
WIP: update docs WIP: finalize docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: update docs", "WIP: finalize docs"]}
anthropic/claude-haiku-4.5:medium PASS 100% 131 in → 256 out (223 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 322 in → 2,254 out (835 reasoning)
WIP: update docs WIP: finalize docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: update docs", "WIP: finalize docs"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 132 in → 281 out (240 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 319 in → 1,395 out (697 reasoning)
WIP: update docs WIP: finalize docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: update docs", "WIP: finalize docs"]}
anthropic/claude-opus-4.6:high PASS 100% 102 in → 179 out (167 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.6:high__json_schema PASS 100% 292 in → 134 out (96 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.6:low PASS 100% 106 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.6:low__json_schema PASS 100% 293 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.6:max PASS 100% 104 in → 211 out (192 reasoning)
WIP: update docs WIP: finalize docs
anthropic/claude-opus-4.6:max__json_schema PASS 100% 288 in → 194 out (151 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.6:medium PASS 100% 101 in → 16 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 291 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.6:none PASS 100% 105 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.6:none__json_schema PASS 100% 294 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 104 in → 175 out (141 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 291 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.7:high PASS 100% 147 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.7:high__json_schema PASS 100% 387 in → 30 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.7:low PASS 100% 143 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.7:low__json_schema PASS 100% 387 in → 30 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.7:max PASS 100% 143 in → 181 out (85 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.7:max__json_schema PASS 100% 386 in → 30 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.7:medium PASS 100% 149 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 388 in → 30 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.7:none PASS 100% 149 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.7:none__json_schema PASS 100% 387 in → 30 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 152 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 384 in → 30 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.8:high PASS 100% 139 in → 102 out (48 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.8:high__json_schema PASS 100% 381 in → 28 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.8:low PASS 100% 145 in → 45 out (25 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.8:low__json_schema PASS 100% 380 in → 28 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.8:max PASS 100% 139 in → 322 out (77 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.8:max__json_schema PASS 100% 381 in → 28 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.8:medium PASS 100% 142 in → 57 out (25 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 383 in → 28 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.8:none PASS 100% 141 in → 19 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.8:none__json_schema PASS 100% 383 in → 27 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 139 in → 193 out (65 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 378 in → 30 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
anthropic/claude-sonnet-4.6:high PASS 100% 105 in → 140 out (110 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 291 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-sonnet-4.6:low PASS 100% 106 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 289 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-sonnet-4.6:max PASS 100% 104 in → 188 out (164 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 292 in → 177 out (136 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 103 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 291 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-sonnet-4.6:none PASS 100% 107 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 291 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 104 in → 142 out (107 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 290 in → 100 out (74 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-sonnet-5:high PASS 100% 141 in → 56 out (43 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-sonnet-5:high__json_schema PASS 100% 383 in → 28 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
anthropic/claude-sonnet-5:low PASS 100% 144 in → 19 out (0 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-sonnet-5:medium PASS 100% 139 in → 50 out (39 reasoning)
WIP: finalize docs WIP: update docs
anthropic/claude-sonnet-5:xhigh PASS 100% 141 in → 47 out (29 reasoning)
WIP: finalize docs WIP: update docs
arcee-ai/trinity-large-thinking:low PASS 100% 96 in → 1,319 out (1,371 reasoning)
WIP: finalize docs WIP: update docs
arcee-ai/trinity-large-thinking:medium PASS 100% 95 in → 619 out (639 reasoning)
WIP: finalize docs WIP: update docs
arcee-ai/trinity-large-thinking:xhigh PASS 100% 95 in → 1,793 out (1,876 reasoning)
WIP: finalize docs WIP: update docs
arcee-ai/trinity-mini:low PASS 100% 92 in → 420 out (468 reasoning)
WIP: finalize docs WIP: update docs
arcee-ai/trinity-mini:xhigh PASS 100% 96 in → 316 out (345 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash-0731:high PASS 100% 174 in → 152 out (139 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash-0731:low PASS 100% 95 in → 181 out (171 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 98 in → 104 out (76 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ]}
deepseek/deepseek-v4-flash-0731:max PASS 100% 185 in → 161 out (154 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 173 in → 171 out (149 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 93 in → 286 out (295 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 93 in → 136 out (110 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 93 in → 13 out (0 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 93 in → 28 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
deepseek/deepseek-v4-flash:high PASS 100% 95 in → 79 out (71 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 93 in → 121 out (99 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
deepseek/deepseek-v4-flash:low PASS 100% 93 in → 109 out (95 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash:medium PASS 100% 93 in → 70 out (61 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 204 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
deepseek/deepseek-v4-flash:none PASS 100% 94 in → 13 out (0 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash:xhigh PASS 100% 172 in → 420 out (471 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 172 in → 335 out (310 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
deepseek/deepseek-v4-pro:high PASS 100% 94 in → 215 out (201 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 296 in → 231 out (207 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["WIP: finalize docs", "WIP: update docs"] }
deepseek/deepseek-v4-pro:low PASS 100% 95 in → 202 out (189 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 97 in → 168 out (159 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": [ "WIP: finalize docs", "WIP: update docs" ]}
deepseek/deepseek-v4-pro:medium PASS 100% 93 in → 13 out (0 reasoning)
WIP: finalize docs WIP: update docs
deepseek/deepseek-v4-pro:xhigh PASS 100% 173 in → 854 out (840 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3-flash-preview:high PASS 100% 96 in → 339 out (328 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3-flash-preview:low PASS 100% 95 in → 326 out (315 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3-flash-preview:low__json_schema PASS 100% 176 in → 350 out (333 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
google/gemini-3-flash-preview:medium PASS 100% 97 in → 243 out (232 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3-flash-preview:medium__json_schema PASS 100% 175 in → 425 out (410 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3-flash-preview:xhigh PASS 100% 95 in → 286 out (275 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 174 in → 449 out (434 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 97 in → 468 out (457 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 175 in → 462 out (447 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 96 in → 134 out (123 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.1-flash-lite-preview:medium PASS 100% 98 in → 297 out (286 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 95 in → 625 out (614 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 94 in → 940 out (923 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
google/gemini-3.1-pro-preview:high PASS 100% 94 in → 274 out (263 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 175 in → 463 out (448 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.1-pro-preview:low PASS 100% 97 in → 123 out (112 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 176 in → 271 out (241 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
google/gemini-3.1-pro-preview:medium PASS 100% 93 in → 314 out (303 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 175 in → 393 out (378 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 97 in → 418 out (407 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.5-flash-lite:high PASS 100% 97 in → 1,428 out (1,417 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 95 in → 523 out (508 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.5-flash-lite:low PASS 100% 96 in → 11 out (0 reasoning)
WIP: update docs WIP: finalize docs
google/gemini-3.5-flash-lite:max PASS 100% 98 in → 671 out (660 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 174 in → 818 out (803 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.5-flash-lite:medium PASS 100% 95 in → 724 out (713 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 174 in → 688 out (671 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 95 in → 690 out (679 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 176 in → 532 out (515 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
google/gemini-3.5-flash:high PASS 100% 95 in → 407 out (396 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.5-flash:high__json_schema PASS 100% 174 in → 542 out (525 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
google/gemini-3.5-flash:low PASS 100% 97 in → 143 out (132 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.5-flash:low__json_schema PASS 100% 173 in → 165 out (150 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.5-flash:medium PASS 100% 97 in → 428 out (417 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.5-flash:medium__json_schema PASS 100% 173 in → 386 out (356 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
google/gemini-3.5-flash:xhigh PASS 100% 96 in → 410 out (399 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 176 in → 455 out (440 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.6-flash:high PASS 100% 96 in → 476 out (465 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.6-flash:high__json_schema PASS 100% 96 in → 905 out (890 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.6-flash:low PASS 100% 96 in → 141 out (130 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.6-flash:max PASS 100% 96 in → 382 out (371 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.6-flash:max__json_schema PASS 100% 174 in → 1,387 out (1,372 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.6-flash:medium PASS 100% 95 in → 517 out (506 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.6-flash:medium__json_schema PASS 100% 95 in → 1,054 out (1,039 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemini-3.6-flash:xhigh PASS 100% 93 in → 536 out (525 reasoning)
WIP: finalize docs WIP: update docs
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 95 in → 1,392 out (1,377 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
google/gemma-4-26b-a4b-it:high PASS 100% 109 in → 305 out (250 reasoning)
WIP: finalize docs WIP: update docs
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 110 in → 379 out (309 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 113 in → 785 out (691 reasoning)
WIP: finalize docs WIP: update docs
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 111 in → 664 out (579 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 113 in → 319 out (275 reasoning)
WIP: finalize docs WIP: update docs
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 110 in → 395 out (312 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 108 in → 12 out (0 reasoning)
WIP: finalize docs WIP: update docs
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 106 in → 27 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 111 in → 490 out (412 reasoning)
WIP: finalize docs WIP: update docs
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 110 in → 511 out (415 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
google/gemma-4-31b-it:high PASS 100% 111 in → 251 out (1 reasoning)
WIP: update docs WIP: finalize docs
google/gemma-4-31b-it:high__json_schema PASS 100% 110 in → 223 out (171 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
google/gemma-4-31b-it:low PASS 100% 110 in → 250 out (189 reasoning)
WIP: finalize docs WIP: update docs
google/gemma-4-31b-it:low__json_schema PASS 100% 110 in → 347 out (261 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
google/gemma-4-31b-it:medium PASS 100% 111 in → 209 out (177 reasoning)
WIP: finalize docs WIP: update docs
google/gemma-4-31b-it:medium__json_schema PASS 100% 113 in → 265 out (247 reasoning)
WIP: update docs WIP: finalize docs
JSON Schema Structured Output
(raw) {"commits":["WIP: update docs","WIP: finalize docs"]}
google/gemma-4-31b-it:none PASS 100% 105 in → 12 out (0 reasoning)
WIP: finalize docs WIP: update docs
google/gemma-4-31b-it:none__json_schema PASS 100% 109 in → 27 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
google/gemma-4-31b-it:xhigh PASS 100% 115 in → 424 out (412 reasoning)
WIP: finalize docs WIP: update docs
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 112 in → 246 out (193 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
ibm-granite/granite-4.1-8b__json_schema PASS 100% 101 in → 27 out
WIP: update docs WIP: finalize docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update docs", "WIP: finalize docs" ] }
inclusionai/ling-2.6-flash__json_schema PASS 100% 116 in → 36 out
WIP: update docs WIP: finalize docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update docs", "WIP: finalize docs" ] }
minimax/minimax-m2.5:high PASS 100% 125 in → 182 out (183 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m2.5:low PASS 100% 125 in → 178 out (184 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m2.5:low__json_schema PASS 100% 126 in → 194 out (168 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
minimax/minimax-m2.5:medium PASS 100% 126 in → 249 out (269 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m2.5:medium__json_schema PASS 100% 131 in → 250 out (225 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
minimax/minimax-m2.5:xhigh PASS 100% 123 in → 220 out (228 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 126 in → 351 out (325 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
minimax/minimax-m2.7:high PASS 100% 125 in → 316 out (303 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m2.7:high__json_schema PASS 100% 126 in → 20 out (20 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
minimax/minimax-m2.7:low PASS 100% 126 in → 351 out (380 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m2.7:low__json_schema PASS 100% 281 in → 515 out (496 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
minimax/minimax-m2.7:medium PASS 100% 126 in → 372 out (398 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m2.7:medium__json_schema PASS 100% 276 in → 645 out (617 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
minimax/minimax-m2.7:xhigh PASS 100% 125 in → 418 out (473 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m3:high PASS 100% 265 in → 154 out (153 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m3:low PASS 100% 266 in → 146 out (145 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m3:medium PASS 100% 262 in → 168 out (173 reasoning)
WIP: finalize docs WIP: update docs
minimax/minimax-m3:xhigh PASS 100% 265 in → 172 out (168 reasoning)
WIP: finalize docs WIP: update docs
mistralai/devstral-2512 PASS 100% 101 in → 13 out
WIP: finalize docs WIP: update docs
mistralai/devstral-2512__json_schema PASS 100% 101 in → 28 out
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
mistralai/mistral-medium-3-5:high PASS 100% 111 in → 713 out (711 reasoning)
WIP: finalize docs WIP: update docs
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 103 in → 270 out (220 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
mistralai/mistral-medium-3-5:low PASS 100% 112 in → 267 out (239 reasoning)
WIP: finalize docs WIP: update docs
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 100 in → 484 out (462 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
mistralai/mistral-medium-3-5:medium PASS 100% 114 in → 343 out (316 reasoning)
WIP: finalize docs WIP: update docs
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 101 in → 488 out (486 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
mistralai/mistral-medium-3-5:none PASS 100% 109 in → 13 out (0 reasoning)
WIP: finalize docs WIP: update docs
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 100 in → 21 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 113 in → 344 out (315 reasoning)
WIP: finalize docs WIP: update docs
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 100 in → 336 out (302 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
moonshotai/kimi-k2.6:high PASS 100% 94 in → 364 out (411 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k2.6:high__json_schema PASS 100% 286 in → 926 out (908 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
moonshotai/kimi-k2.6:low PASS 100% 96 in → 215 out (223 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k2.6:low__json_schema PASS 100% 96 in → 493 out (467 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
moonshotai/kimi-k2.6:medium PASS 100% 94 in → 615 out (602 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k2.6:xhigh PASS 100% 96 in → 466 out (453 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 93 in → 568 out (541 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
moonshotai/kimi-k2.7-code:high PASS 100% 95 in → 246 out (264 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 190 in → 156 out (136 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
moonshotai/kimi-k2.7-code:low PASS 100% 98 in → 156 out (156 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 92 in → 275 out (255 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
moonshotai/kimi-k2.7-code:medium PASS 100% 97 in → 130 out (140 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 94 in → 190 out (184 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["WIP: finalize docs", "WIP: update docs"] }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 93 in → 149 out (136 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k3:high PASS 100% 170 in → 207 out (181 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k3:high__json_schema PASS 100% 173 in → 160 out (134 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k3:low PASS 100% 171 in → 360 out (334 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k3:low__json_schema PASS 100% 172 in → 216 out (190 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k3:max PASS 100% 173 in → 293 out (267 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k3:max__json_schema PASS 100% 172 in → 192 out (166 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k3:medium PASS 100% 171 in → 190 out (164 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k3:medium__json_schema PASS 100% 171 in → 186 out (160 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k3:xhigh PASS 100% 169 in → 290 out (264 reasoning)
WIP: finalize docs WIP: update docs
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 172 in → 186 out (160 reasoning)
WIP: finalize docs WIP: update docs
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 113 in → 205 out (215 reasoning)
WIP: finalize docs WIP: update docs
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 112 in → 5,389 out (5,895 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 114 in → 267 out (283 reasoning)
WIP: finalize docs WIP: update docs
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 115 in → 596 out (609 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 114 in → 265 out (275 reasoning)
WIP: finalize docs WIP: update docs
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 114 in → 278 out (267 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 113 in → 28 out (0 reasoning)
WIP: update docs WIP: finalize docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update docs", "WIP: finalize docs" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 113 in → 227 out (221 reasoning)
WIP: finalize docs WIP: update docs
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 113 in → 210 out (194 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 110 in → 190 out (166 reasoning)
WIP: finalize docs WIP: update docs
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 114 in → 233 out (186 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 114 in → 256 out (266 reasoning)
WIP: finalize docs WIP: update docs
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 113 in → 247 out (204 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 115 in → 301 out (267 reasoning)
WIP: finalize docs WIP: update docs
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 114 in → 263 out (224 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 113 in → 227 out (203 reasoning)
WIP: finalize docs WIP: update docs
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 115 in → 248 out (207 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
openai/gpt-5.4-mini:high PASS 100% 98 in → 168 out (151 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4-mini:high__json_schema PASS 100% 139 in → 150 out (123 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4-mini:low PASS 100% 96 in → 82 out (65 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4-mini:low__json_schema PASS 100% 145 in → 71 out (44 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4-mini:medium PASS 100% 93 in → 102 out (85 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4-mini:medium__json_schema PASS 100% 140 in → 128 out (101 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4-mini:none PASS 100% 97 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4-mini:xhigh PASS 100% 93 in → 239 out (222 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 144 in → 388 out (361 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4-nano:high PASS 100% 95 in → 88 out (71 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4-nano:high__json_schema PASS 100% 142 in → 94 out (67 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4-nano:low PASS 100% 92 in → 50 out (33 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4-nano:low__json_schema PASS 100% 143 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4-nano:medium PASS 100% 94 in → 62 out (45 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4-nano:medium__json_schema PASS 100% 142 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4-nano:none__json_schema PASS 100% 143 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4-nano:xhigh PASS 100% 93 in → 160 out (143 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 144 in → 558 out (531 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4:high PASS 100% 94 in → 121 out (104 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4:high__json_schema PASS 100% 141 in → 82 out (55 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4:low PASS 100% 95 in → 35 out (18 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4:low__json_schema PASS 100% 141 in → 42 out (15 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4:medium PASS 100% 96 in → 109 out (92 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4:medium__json_schema PASS 100% 144 in → 83 out (56 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4:none PASS 100% 95 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4:none__json_schema PASS 100% 141 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.4:xhigh PASS 100% 96 in → 161 out (144 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.4:xhigh__json_schema PASS 100% 139 in → 161 out (134 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.5:high PASS 100% 93 in → 75 out (58 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.5:high__json_schema PASS 100% 141 in → 113 out (86 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.5:low PASS 100% 92 in → 43 out (26 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.5:low__json_schema PASS 100% 140 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.5:medium PASS 100% 96 in → 65 out (48 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.5:medium__json_schema PASS 100% 144 in → 66 out (39 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.5:none PASS 100% 96 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.5:none__json_schema PASS 100% 142 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.5:xhigh PASS 100% 97 in → 115 out (98 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.5:xhigh__json_schema PASS 100% 144 in → 133 out (106 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-luna:high PASS 100% 94 in → 58 out (41 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-luna:high__json_schema PASS 100% 144 in → 53 out (26 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-luna:low PASS 100% 93 in → 38 out (21 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-luna:low__json_schema PASS 100% 141 in → 42 out (15 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-luna:max PASS 100% 92 in → 110 out (93 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-luna:max__json_schema PASS 100% 143 in → 124 out (97 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-luna:medium PASS 100% 91 in → 47 out (30 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-luna:medium__json_schema PASS 100% 140 in → 46 out (19 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-luna:none PASS 100% 93 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-luna:none__json_schema PASS 100% 141 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-luna:xhigh PASS 100% 93 in → 87 out (70 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 142 in → 504 out (477 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-sol:high PASS 100% 94 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-sol:high__json_schema PASS 100% 146 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-sol:low PASS 100% 98 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-sol:low__json_schema PASS 100% 142 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-sol:max PASS 100% 93 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-sol:max__json_schema PASS 100% 139 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-sol:medium PASS 100% 93 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-sol:medium__json_schema PASS 100% 144 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-sol:none PASS 100% 92 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-sol:none__json_schema PASS 100% 144 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-sol:xhigh PASS 100% 95 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 142 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-terra:high PASS 100% 92 in → 39 out (22 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-terra:high__json_schema PASS 100% 141 in → 47 out (20 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-terra:low PASS 100% 92 in → 37 out (20 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-terra:low__json_schema PASS 100% 143 in → 47 out (20 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-terra:max PASS 100% 91 in → 47 out (30 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-terra:max__json_schema PASS 100% 143 in → 69 out (42 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-terra:medium PASS 100% 93 in → 40 out (23 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-terra:medium__json_schema PASS 100% 142 in → 47 out (20 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-terra:none PASS 100% 97 in → 15 out (0 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-terra:none__json_schema PASS 100% 144 in → 25 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-5.6-terra:xhigh PASS 100% 98 in → 52 out (35 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 142 in → 52 out (25 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-oss-120b:high PASS 100% 154 in → 310 out (335 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-oss-120b:high__json_schema PASS 100% 156 in → 246 out (251 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["WIP: finalize docs","WIP: update docs"] }
openai/gpt-oss-120b:low PASS 100% 143 in → 86 out (68 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-oss-120b:low__json_schema PASS 100% 144 in → 92 out (68 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["WIP: finalize docs", "WIP: update docs"] }
openai/gpt-oss-120b:medium PASS 100% 155 in → 192 out (196 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-oss-120b:medium__json_schema PASS 100% 153 in → 26 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
openai/gpt-oss-120b:xhigh PASS 100% 141 in → 213 out (230 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 153 in → 313 out (343 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-oss-20b:high PASS 100% 156 in → 270 out (298 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-oss-20b:low PASS 100% 156 in → 110 out (89 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-oss-20b:low__json_schema PASS 100% 151 in → 97 out (77 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
openai/gpt-oss-20b:medium PASS 100% 154 in → 177 out (166 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-oss-20b:xhigh PASS 100% 152 in → 151 out (140 reasoning)
WIP: finalize docs WIP: update docs
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 158 in → 326 out (358 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
poolside/laguna-m.1:high PASS 100% 105 in → 407 out (395 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-m.1:high__json_schema PASS 100% 107 in → 688 out (660 reasoning)
WIP: update docs WIP: finalize docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update docs", "WIP: finalize docs" ] }
poolside/laguna-m.1:low PASS 100% 106 in → 210 out (198 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-m.1:low__json_schema PASS 100% 105 in → 509 out (492 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
poolside/laguna-m.1:medium PASS 100% 105 in → 239 out (227 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-m.1:medium__json_schema PASS 100% 102 in → 260 out (242 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
poolside/laguna-m.1:xhigh PASS 100% 108 in → 374 out (362 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-m.1:xhigh__json_schema PASS 100% 105 in → 315 out (298 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
poolside/laguna-xs-2.1:high PASS 100% 104 in → 252 out (242 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-xs-2.1:high__json_schema PASS 100% 108 in → 236 out (206 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
poolside/laguna-xs-2.1:low PASS 100% 103 in → 207 out (197 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-xs-2.1:low__json_schema PASS 100% 106 in → 257 out (226 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
poolside/laguna-xs-2.1:medium PASS 100% 106 in → 238 out (228 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 107 in → 349 out (318 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
poolside/laguna-xs-2.1:none PASS 100% 107 in → 9 out (0 reasoning)
WIP: update docs WIP: finalize docs
poolside/laguna-xs-2.1:xhigh PASS 100% 104 in → 181 out (171 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-xs.2:high PASS 100% 145 in → 198 out (186 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-xs.2:high__json_schema PASS 100% 143 in → 224 out (194 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
poolside/laguna-xs.2:low PASS 100% 144 in → 179 out (167 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-xs.2:low__json_schema PASS 100% 139 in → 165 out (147 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
poolside/laguna-xs.2:medium PASS 100% 143 in → 210 out (198 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-xs.2:medium__json_schema PASS 100% 142 in → 153 out (136 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
poolside/laguna-xs.2:none PASS 100% 143 in → 11 out (0 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-xs.2:none__json_schema PASS 100% 142 in → 16 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
poolside/laguna-xs.2:xhigh PASS 100% 143 in → 194 out (182 reasoning)
WIP: finalize docs WIP: update docs
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 141 in → 181 out (151 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
qwen/qwen3.6-27b:high PASS 100% 106 in → 971 out (941 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-27b:high__json_schema PASS 100% 105 in → 1,356 out (1,352 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
qwen/qwen3.6-27b:low PASS 100% 104 in → 1,017 out (989 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-27b:low__json_schema PASS 100% 108 in → 26 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
qwen/qwen3.6-27b:medium PASS 100% 105 in → 318 out (326 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-27b:medium__json_schema PASS 100% 106 in → 1,142 out (1,070 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 101 in → 920 out (904 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 107 in → 939 out (1 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 104 in → 936 out (897 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 106 in → 942 out (913 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 107 in → 1,048 out (983 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 108 in → 359 out (297 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 107 in → 1,933 out (1,828 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 107 in → 1,139 out (1,125 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 107 in → 12 out (0 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 109 in → 32 out (0 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 108 in → 683 out (645 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 105 in → 1,300 out (1,203 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["WIP: finalize docs", "WIP: update docs"] }
qwen/qwen3.6-flash:high PASS 100% 106 in → 412 out (396 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-flash:low PASS 100% 107 in → 1,246 out (1,230 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-flash:medium PASS 100% 108 in → 1,024 out (1,008 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-flash:none PASS 100% 107 in → 11 out (0 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.6-flash:xhigh PASS 100% 107 in → 1,482 out (1,466 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-flash:high PASS 100% 106 in → 1,217 out (1,201 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-flash:low PASS 100% 107 in → 1,110 out (1,094 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-flash:max PASS 100% 104 in → 1,165 out (1,149 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-flash:medium PASS 100% 108 in → 777 out (761 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-flash:none PASS 100% 108 in → 11 out (0 reasoning)
WIP: update docs WIP: finalize docs
qwen/qwen3.7-flash:xhigh PASS 100% 103 in → 1,151 out (1,135 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-max:high PASS 100% 106 in → 277 out (261 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-max:low PASS 100% 106 in → 384 out (368 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-max:medium PASS 100% 106 in → 914 out (898 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-max:none PASS 100% 110 in → 11 out (0 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-max:xhigh PASS 100% 106 in → 207 out (191 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-plus:high PASS 100% 108 in → 651 out (635 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-plus:low PASS 100% 107 in → 782 out (766 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-plus:medium PASS 100% 104 in → 592 out (576 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-plus:none PASS 100% 107 in → 11 out (0 reasoning)
WIP: finalize docs WIP: update docs
qwen/qwen3.7-plus:xhigh PASS 100% 108 in → 577 out (561 reasoning)
WIP: finalize docs WIP: update docs
tencent/hy3:high PASS 100% 101 in → 422 out (441 reasoning)
WIP: finalize docs WIP: update docs
tencent/hy3:high__json_schema PASS 100% 104 in → 344 out (326 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["WIP: finalize docs", "WIP: update docs"] }
tencent/hy3:low PASS 100% 101 in → 284 out (270 reasoning)
WIP: finalize docs WIP: update docs
tencent/hy3:low__json_schema PASS 100% 102 in → 239 out (211 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
tencent/hy3:medium PASS 100% 104 in → 262 out (251 reasoning)
WIP: finalize docs WIP: update docs
tencent/hy3:medium__json_schema PASS 100% 102 in → 296 out (272 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["WIP: finalize docs", "WIP: update docs"] }
tencent/hy3:none PASS 100% 103 in → 13 out (0 reasoning)
WIP: finalize docs WIP: update docs
tencent/hy3:none__json_schema PASS 100% 102 in → 24 out (0 reasoning)
WIP: update docs WIP: finalize docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: update docs", "WIP: finalize docs" ] }
tencent/hy3:xhigh PASS 100% 101 in → 369 out (392 reasoning)
WIP: finalize docs WIP: update docs
tencent/hy3:xhigh__json_schema PASS 100% 100 in → 511 out (501 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
thinkingmachines/inkling-small:high PASS 100% 106 in → 173 out (156 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling-small:high__json_schema PASS 100% 103 in → 162 out (135 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
thinkingmachines/inkling-small:low PASS 100% 102 in → 51 out (37 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling-small:low__json_schema PASS 100% 102 in → 90 out (66 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
thinkingmachines/inkling-small:max PASS 100% 103 in → 153 out (136 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling-small:max__json_schema PASS 100% 101 in → 205 out (180 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["WIP: finalize docs", "WIP: update docs"] }
thinkingmachines/inkling-small:medium PASS 100% 108 in → 191 out (174 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling-small:medium__json_schema PASS 100% 104 in → 171 out (144 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
thinkingmachines/inkling-small:none PASS 100% 100 in → 14 out (0 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling-small:xhigh PASS 100% 104 in → 182 out (181 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 106 in → 213 out (188 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["WIP: finalize docs", "WIP: update docs"] }
thinkingmachines/inkling:high PASS 100% 102 in → 226 out (232 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling:high__json_schema PASS 100% 101 in → 144 out (119 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["WIP: finalize docs", "WIP: update docs"] }
thinkingmachines/inkling:low PASS 100% 99 in → 45 out (28 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling:low__json_schema PASS 100% 102 in → 63 out (32 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
thinkingmachines/inkling:max PASS 100% 99 in → 190 out (172 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling:medium PASS 100% 101 in → 118 out (109 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling:none PASS 100% 102 in → 14 out (0 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling:xhigh PASS 100% 106 in → 196 out (190 reasoning)
WIP: finalize docs WIP: update docs
thinkingmachines/inkling:xhigh__json_schema PASS 100% 101 in → 204 out (180 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
x-ai/grok-4.3:high PASS 100% 270 in → 386 out (375 reasoning)
WIP: finalize docs WIP: update docs
x-ai/grok-4.3:high__json_schema PASS 100% 336 in → 620 out (605 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
x-ai/grok-4.3:low PASS 100% 277 in → 238 out (227 reasoning)
WIP: finalize docs WIP: update docs
x-ai/grok-4.3:low__json_schema PASS 100% 345 in → 375 out (360 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
x-ai/grok-4.3:max PASS 100% 269 in → 478 out (467 reasoning)
WIP: finalize docs WIP: update docs
x-ai/grok-4.3:max__json_schema PASS 100% 336 in → 481 out (456 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
x-ai/grok-4.3:medium PASS 100% 276 in → 322 out (311 reasoning)
WIP: finalize docs WIP: update docs
x-ai/grok-4.3:medium__json_schema PASS 100% 343 in → 418 out (393 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
x-ai/grok-4.3:xhigh PASS 100% 272 in → 346 out (335 reasoning)
WIP: finalize docs WIP: update docs
x-ai/grok-4.3:xhigh__json_schema PASS 100% 336 in → 586 out (561 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
x-ai/grok-4.5:high PASS 100% 295 in → 223 out (214 reasoning)
WIP: finalize docs WIP: update docs
x-ai/grok-4.5:high__json_schema PASS 100% 374 in → 454 out (430 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
x-ai/grok-4.5:low PASS 100% 291 in → 141 out (132 reasoning)
WIP: finalize docs WIP: update docs
x-ai/grok-4.5:low__json_schema PASS 100% 373 in → 379 out (365 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
x-ai/grok-4.5:max PASS 100% 296 in → 156 out (147 reasoning)
WIP: finalize docs WIP: update docs
x-ai/grok-4.5:max__json_schema PASS 100% 374 in → 426 out (412 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
x-ai/grok-4.5:medium PASS 100% 292 in → 265 out (256 reasoning)
WIP: finalize docs WIP: update docs
x-ai/grok-4.5:medium__json_schema PASS 100% 374 in → 474 out (460 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
x-ai/grok-4.5:xhigh PASS 100% 292 in → 184 out (175 reasoning)
WIP: finalize docs WIP: update docs
x-ai/grok-4.5:xhigh__json_schema PASS 100% 374 in → 393 out (379 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs"]}
z-ai/glm-4.7-flash:high PASS 100% 97 in → 738 out (701 reasoning)
WIP: update docs WIP: finalize docs
z-ai/glm-4.7-flash:low PASS 100% 96 in → 2,330 out (2,317 reasoning)
WIP: update docs WIP: finalize docs
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 96 in → 389 out (366 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
z-ai/glm-5.2:high PASS 100% 96 in → 338 out (279 reasoning)
WIP: finalize docs WIP: update docs
z-ai/glm-5.2:high__json_schema PASS 100% 103 in → 166 out (138 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
z-ai/glm-5.2:low PASS 100% 100 in → 139 out (152 reasoning)
WIP: finalize docs WIP: update docs
z-ai/glm-5.2:low__json_schema PASS 100% 103 in → 223 out (203 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: finalize docs", "WIP: update docs"]}
z-ai/glm-5.2:medium PASS 100% 103 in → 158 out (152 reasoning)
WIP: finalize docs WIP: update docs
z-ai/glm-5.2:medium__json_schema PASS 100% 105 in → 191 out (175 reasoning)
WIP: finalize docs WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "WIP: finalize docs", "WIP: update docs" ] }
arcee-ai/trinity-mini:high FAIL 50% 95 in → 383 out (421 reasoning)
WIP: finalize docs WIP: updatedocs
Failure: Missing expected commit messages: ['WIP: update docs']; Extra selected commit messages: ['WIP: updatedocs']
arcee-ai/trinity-mini:medium FAIL 50% 97 in → 409 out (416 reasoning)
WIP:finalize docs WIP: update docs
Failure: Missing expected commit messages: ['WIP: finalize docs']; Extra selected commit messages: ['WIP:finalize docs']
google/gemini-3.1-pro-preview:xhigh__json_schema FAIL 50% 176 in → 419 out (402 reasoning)
WIP: finalize docs WIP: update docs update docs
JSON Schema Structured Output
(raw) {"commits":["WIP: finalize docs","WIP: update docs update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs']; Extra selected commit messages: ['WIP: update docs']
z-ai/glm-4.7-flash:high__json_schema FAIL 50% 97 in → 738 out (680 reasoning)
WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: finalize docs']
z-ai/glm-4.7-flash:medium FAIL 50% 94 in → 617 out (564 reasoning)
WIP: update docs
Failure: Missing expected commit messages: ['WIP: finalize docs']
anthropic/claude-haiku-4.5:high__json_schema FAIL 0% 317 in → 750 out (538 reasoning)
a236eee WIP: update docs ca83325 WIP: finalize docs
JSON Schema Structured Output
(raw) {"commits": ["a236eee WIP: update docs", "ca83325 WIP: finalize docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: update docs', 'WIP: finalize docs']
anthropic/claude-haiku-4.5:none FAIL 0% 104 in → 26 out (0 reasoning)
2ba0800 WIP: finalize docs ce82d52 WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 289 in → 19 out (0 reasoning)
c14a235 405bc7b
JSON Schema Structured Output
(raw) {"commits": ["c14a235", "405bc7b"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 382 in → 41 out (0 reasoning)
9cdc010 WIP: finalize docs 4d56cc6 WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["9cdc010 WIP: finalize docs","4d56cc6 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 380 in → 37 out (0 reasoning)
6253a41 WIP: finalize docs 9388763 WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["6253a41 WIP: finalize docs","9388763 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
anthropic/claude-sonnet-5:none FAIL 0% 142 in → 36 out (0 reasoning)
``` 6e97ba0 WIP: finalize docs 6dd7495 WIP: update docs ```
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['```', 'WIP: finalize docs', 'WIP: update docs', '```']
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 382 in → 38 out (0 reasoning)
54a1ba1 WIP: finalize docs 540c759 WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["54a1ba1 WIP: finalize docs","540c759 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 382 in → 90 out (40 reasoning)
2b4c95f WIP: finalize docs ad7f594 WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["2b4c95f WIP: finalize docs","ad7f594 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
arcee-ai/trinity-large-thinking:high FAIL 0% 96 in → 627 out (628 reasoning)
b4e4d2a WIP: finalize docs 977151e WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high__json_schema FAIL 0% 96 in → 351 out (354 reasoning)
ed34d22 WIP: finalize docs 7064b92 WIP: update docs e9a5ec1 Add documentation
JSON Schema Structured Output
(raw) { "commits": [ "ed34d22 WIP: finalize docs", "7064b92 WIP: update docs", "e9a5ec1 Add documentation" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs', 'Add documentation']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 92 in → 348 out (365 reasoning)
517d4cb WIP: finalize docs 688e899 WIP: update docs 238090c Add documentation
JSON Schema Structured Output
(raw) { "commits": [ "517d4cb WIP: finalize docs", "688e899 WIP: update docs", "238090c Add documentation" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs', 'Add documentation']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 96 in → 357 out (366 reasoning)
4a3c17b WIP: finalize docs c413c01 WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "4a3c17b WIP: finalize docs", "c413c01 WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 93 in → 452 out (499 reasoning)
c874202 WIP: finalize docs 5e81428 WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "c874202 WIP: finalize docs", "5e81428 WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
deepseek/deepseek-v4-flash-0731:high__json_schema FAIL 0% 94 in → 602 out (569 reasoning)
43d0762 WIP: finalize docs 5c5f593 WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["43d0762 WIP: finalize docs", "5c5f593 WIP: update docs"] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
deepseek/deepseek-v4-flash-0731:xhigh FAIL 0% 93 in → 303 out (283 reasoning)
2e85798 WIP: finalize docs da8c53d WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 179 in → 147 out (146 reasoning)
2601a95 57d6193
JSON Schema Structured Output
(raw) { "commits": ["2601a95", "57d6193"] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
deepseek/deepseek-v4-flash:low__json_schema FAIL 0% 97 in → 145 out (113 reasoning)
f80db03 WIP: finalize docs a489a9d WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["f80db03 WIP: finalize docs", "a489a9d WIP: update docs"] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 95 in → 31 out (0 reasoning)
8cee7a5 WIP: finalize docs 0586c30 WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["8cee7a5 WIP: finalize docs", "0586c30 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
Invalid JSON. Output: WIP: update docs WIP: finalize docs
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:none FAIL 0% 91 in → 20 out (0 reasoning)
a62e061 WIP: finalize docs a942667 WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 95 in → 37 out (0 reasoning)
6fd5e1c WIP: finalize docs 397aeff WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "6fd5e1c WIP: finalize docs", "397aeff WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 175 in → 686 out (662 reasoning)
7176fec WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "7176fec WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: update docs']
google/gemini-3-flash-preview:high__json_schema FAIL 0% 172 in → 387 out (361 reasoning)
8881fc7 WIP: finalize docs fff86d3 WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["8881fc7 WIP: finalize docs", "fff86d3 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 0% 94 in → 122 out (105 reasoning)
fdd306b 9b174dd
JSON Schema Structured Output
(raw) {"commits":["fdd306b","9b174dd"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 95 in → 375 out (334 reasoning)
37c10c8 WIP: finalize docs f228fe8 WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "37c10c8 WIP: finalize docs", "f228fe8 WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 174 in → 40 out (0 reasoning)
ae9a0b4 WIP: update docs 0e67dc2 WIP: finalize docs
JSON Schema Structured Output
(raw) { "commits": [ "ae9a0b4 WIP: update docs", "0e67dc2 WIP: finalize docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: update docs', 'WIP: finalize docs']
google/gemini-3.6-flash:low__json_schema FAIL 0% 94 in → 12 out (0 reasoning)
3077a76 WIP: finalize docs efef644 WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "3077a76 WIP: finalize docs", "efef644 WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
ibm-granite/granite-4.1-8b FAIL 0% 101 in → 20 out
19e4334 WIP: update docs b8ea776 WIP: finalize docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: update docs', 'WIP: finalize docs']
inclusionai/ling-2.6-flash FAIL 0% 115 in → 26 out
c95e97b WIP: finalize docs c2d9492 WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
liquid/lfm-2-24b-a2b FAIL 0% 107 in → 23 out
72aca18 WIP: update docs 1b36fb1 WIP: finalize docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: update docs', 'WIP: finalize docs']
Invalid JSON. Output: f9bea77
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:high__json_schema FAIL 0% 123 in → 218 out (208 reasoning)
976adfa WIP: finalize docs abdf887 WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "976adfa WIP: finalize docs", "abdf887 WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high__json_schema FAIL 0% 252 in → 20 out (0 reasoning)
db4d24e 75f3310
JSON Schema Structured Output
(raw) { "commits": ["db4d24e", "75f3310"] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
minimax/minimax-m3:low__json_schema FAIL 0% 250 in → 34 out (0 reasoning)
b351110 WIP: finalize docs 8308ea5 WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "b351110 WIP: finalize docs", "8308ea5 WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
minimax/minimax-m3:medium__json_schema FAIL 0% 250 in → 28 out (0 reasoning)
27feded WIP: finalize docs ea13001 WIP: update docs
JSON Schema Structured Output
(raw) { "commits": ["27feded WIP: finalize docs", "ea13001 WIP: update docs"] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
minimax/minimax-m3:xhigh__json_schema FAIL 0% 253 in → 30 out (0 reasoning)
3c3a2ee WIP: finalize docs 0dff364 WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["3c3a2ee WIP: finalize docs", "0dff364 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
moonshotai/kimi-k2.6:medium__json_schema FAIL 0% 93 in → 43 out (0 reasoning)
8bd7614 WIP: finalize docs 436d6fb WIP: update docs 515d487 Add documentation
JSON Schema Structured Output
(raw) { "commits": [ "8bd7614 WIP: finalize docs", "436d6fb WIP: update docs", "515d487 Add documentation" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs', 'Add documentation']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 114 in → 16 out (0 reasoning)
2a488e3 0414b5c
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: update docs', 'Add documentation']
nvidia/nemotron-3-super-120b-a12b:none FAIL 0% 116 in → 27 out (0 reasoning)
9986c6f WIP: finalize docs 46f129c WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
nvidia/nemotron-3-super-120b-a12b:none__json_schema FAIL 0% 110 in → 39 out (0 reasoning)
f6cc2fc WIP: finalize docs beb7d4b WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "f6cc2fc WIP: finalize docs", "beb7d4b WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
openai/gpt-5.4-mini:none__json_schema FAIL 0% 141 in → 34 out (0 reasoning)
f200b9b WIP: finalize docs 2379b40 WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["f200b9b WIP: finalize docs","2379b40 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
openai/gpt-5.4-nano:none FAIL 0% 93 in → 23 out (0 reasoning)
28fa397 WIP: finalize docs 5d500a2 WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
openai/gpt-oss-20b:high__json_schema FAIL 0% 152 in → 348 out (386 reasoning)
792f971 f79be7f
JSON Schema Structured Output
(raw) {"commits": ["792f971", "f79be7f"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
openai/gpt-oss-20b:medium__json_schema FAIL 0% 142 in → 229 out (219 reasoning)
b14ca41: WIP: finalize docs b7a58f0: WIP: update docs
JSON Schema Structured Output
(raw) {"commits":[ "b14ca41: WIP: finalize docs", "b7a58f0: WIP: update docs" ]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
poolside/laguna-m.1:none FAIL 0% 104 in → 21 out (0 reasoning)
cd907ab WIP: finalize docs aaec444 WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
poolside/laguna-m.1:none__json_schema FAIL 0% 106 in → 30 out (0 reasoning)
e962719 WIP: finalize docs f17e636 WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["e962719 WIP: finalize docs", "f17e636 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 106 in → 44 out (0 reasoning)
a130b88 WIP: finalize docs 4f0700d WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "a130b88 WIP: finalize docs", "4f0700d WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
poolside/laguna-xs-2.1:xhigh__json_schema FAIL 0% 105 in → 247 out (218 reasoning)
b3f425b WIP: finalize docs fa8aac0 WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["b3f425b WIP: finalize docs", "fa8aac0 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
qwen/qwen3.6-27b:none FAIL 0% 109 in → 25 out (0 reasoning)
9b42f83 WIP: finalize docs e6d0556 WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 108 in → 34 out (0 reasoning)
76ba3c0 da550d4
JSON Schema Structured Output
(raw) { "commits": [ "76ba3c0", "da550d4" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b79b873a-e786-984b-8356-caaa05243c6e","request_id":"b79b873a-e786-984b-8356-caaa05243c6e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6a284d03-bbb7-9ba9-ac0b-72a9c3efbfc4","request_id":"6a284d03-bbb7-9ba9-ac0b-72a9c3efbfc4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-23303818-b563-956c-ab97-c3163b2680fb","request_id":"23303818-b563-956c-ab97-c3163b2680fb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1dc842c7-76da-94b9-a7fb-9b3596328938","request_id":"1dc842c7-76da-94b9-a7fb-9b3596328938"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-56c1f0dd-20b9-9d2f-b7e3-ee09c3adbe86","request_id":"56c1f0dd-20b9-9d2f-b7e3-ee09c3adbe86"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0d3acedc-9b88-92d0-a3dd-21b601cd0894"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b565df22-7de2-9832-a5d7-a3fed73273dc"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f5835b00-5fe9-9ea7-b88a-1fc3177e4f34"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1b361dfa-c3fc-98a1-b73e-c4aaf1a8450c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-50d2ee1c-30b1-90e4-adea-465337a087da"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-22d8c755-a7bb-9759-8ad0-0f6cefec7c6e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-43a13238-486b-9b1d-a4a6-03d077deb8bb","request_id":"43a13238-486b-9b1d-a4a6-03d077deb8bb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-941d3e39-88f1-9515-8d94-9b88f90148bc","request_id":"941d3e39-88f1-9515-8d94-9b88f90148bc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4959b423-abf3-9800-878b-c6880129f637","request_id":"4959b423-abf3-9800-878b-c6880129f637"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e687364d-efe8-917f-b4f0-2ea732b40096","request_id":"e687364d-efe8-917f-b4f0-2ea732b40096"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-58825d8c-1740-96d1-8bda-6f763d288f5a","request_id":"58825d8c-1740-96d1-8bda-6f763d288f5a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2646ffaf-ebde-9825-b2a1-3acef9535273","request_id":"2646ffaf-ebde-9825-b2a1-3acef9535273"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d588e114-1272-9a63-a6f7-fddf9d78851a","request_id":"d588e114-1272-9a63-a6f7-fddf9d78851a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-812d0738-e27e-94af-8f29-28b55d74c4c1","request_id":"812d0738-e27e-94af-8f29-28b55d74c4c1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f5bba8b4-7b3a-9894-8723-144cbd9aee34","request_id":"f5bba8b4-7b3a-9894-8723-144cbd9aee34"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-276ddb0d-4675-97e4-a40e-1552c54674b9","request_id":"276ddb0d-4675-97e4-a40e-1552c54674b9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: WIP: finalize docs WIP: update docs{"commits": ["22665af", "444c14f"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: finalize docs WIP: update docs
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: finalize docs WIP: update docs
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0% 102 in → 33 out (0 reasoning)
2e3b646 WIP: finalize docs 99798d9 WIP: update docs
JSON Schema Structured Output
(raw) {"commits": [ "2e3b646 WIP: finalize docs", "99798d9 WIP: update docs" ]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
x-ai/grok-4.3:none FAIL 0% 268 in → 17 out (0 reasoning)
9616fa2 WIP: finalize docs 4050e27 WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
x-ai/grok-4.3:none__json_schema FAIL 0% 335 in → 21 out (0 reasoning)
2b40492 WIP: finalize docs 61b847b WIP: update docs
JSON Schema Structured Output
(raw) {"commits":["2b40492 WIP: finalize docs","61b847b WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 97 in → 21 out (0 reasoning)
eaf94c5 WIP: update docs 1bf4f82 WIP: finalize docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: update docs', 'WIP: finalize docs']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 96 in → 35 out (0 reasoning)
4c42caa WIP: update docs 6b55d49 WIP: finalize docs
JSON Schema Structured Output
(raw) { "commits": [ "4c42caa WIP: update docs", "6b55d49 WIP: finalize docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: update docs', 'WIP: finalize docs']
z-ai/glm-4.7-flash:xhigh FAIL 0% 95 in → 495 out (446 reasoning)
3ac9225 WIP: update docs cf32d18 WIP: finalize docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: update docs', 'WIP: finalize docs']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
Failure: Model 'z-ai/glm-5.2:none' violated the reasoning_level=none invariant: provider reported 762 reasoning token(s)
z-ai/glm-5.2:none__json_schema FAIL 0% 96 in → 23 out (0 reasoning)
24ff358 WIP: update docs
JSON Schema Structured Output
(raw) { "commits": [ "24ff358 WIP: update docs" ] }
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: update docs']
z-ai/glm-5.2:xhigh FAIL 0% 103 in → 913 out (827 reasoning)
8d9f8df WIP: finalize docs db00854 WIP: update docs
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']
z-ai/glm-5.2:xhigh__json_schema FAIL 0% 102 in → 436 out (406 reasoning)
0f1de56 WIP: finalize docs de802a7 WIP: update docs
JSON Schema Structured Output
(raw) {"commits": ["0f1de56 WIP: finalize docs", "de802a7 WIP: update docs"]}
Failure: Missing expected commit messages: ['WIP: update docs', 'WIP: finalize docs']; Extra selected commit messages: ['WIP: finalize docs', 'WIP: update docs']