Multiple separate feature development commits
Tests ability to squash multiple separate feature-development commits into logical groups. Evaluates feature-level commit organization.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 touch base.txt
  5. 05 git add base.txt
  6. 06 git commit -m 'Base commit'
  7. 07 echo 'feat1a' > feature1.txt
  8. 08 git add feature1.txt
  9. 09 git commit -m 'WIP: start feature 1'
  10. 10 echo 'feat1b' >> feature1.txt
  11. 11 git add feature1.txt
  12. 12 git commit -m 'WIP: continue feature 1'
  13. 13 echo 'feat2a' > feature2.txt
  14. 14 git add feature2.txt
  15. 15 git commit -m 'Start feature 2'
Prompt
The second feature was done cleanly in one commit, but feature 1 has WIP commits that should be marked as squash/fixup. Identify only the WIP commits for feature 1 that should be combined. Do not include the clean feature 2 commit. Output ONLY the selected commit subject lines, one per line, with no explanation.
Expected
WIP: start feature 1 WIP: continue feature 1

Scoped model quality, cost, API time, and token usage for commit_squash/f011.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 173 in → 60 out (36 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-fable-5:high__json_schema PASS 100% 414 in → 70 out (29 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
anthropic/claude-fable-5:low PASS 100% 176 in → 37 out (18 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-fable-5:low__json_schema PASS 100% 413 in → 31 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-fable-5:max PASS 100% 179 in → 275 out (65 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-fable-5:max__json_schema PASS 100% 412 in → 274 out (79 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-fable-5:medium PASS 100% 173 in → 47 out (24 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-fable-5:medium__json_schema PASS 100% 414 in → 33 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
anthropic/claude-fable-5:xhigh PASS 100% 177 in → 68 out (41 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 414 in → 71 out (40 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
anthropic/claude-haiku-4.5:high PASS 100% 166 in → 236 out (178 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 350 in → 1,052 out (495 reasoning)
WIP: start feature 1 WIP: continue feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: start feature 1", "WIP: continue feature 1"]}
anthropic/claude-haiku-4.5:low PASS 100% 164 in → 539 out (471 reasoning)
WIP: start feature 1 WIP: continue feature 1
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 351 in → 1,572 out (776 reasoning)
WIP: start feature 1 WIP: continue feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: start feature 1", "WIP: continue feature 1"]}
anthropic/claude-haiku-4.5:medium PASS 100% 162 in → 872 out (486 reasoning)
WIP: start feature 1 WIP: continue feature 1
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 355 in → 1,062 out (578 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 164 in → 7,527 out (2,392 reasoning)
WIP: start feature 1 WIP: continue feature 1
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 350 in → 1,484 out (663 reasoning)
WIP: start feature 1 WIP: continue feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: start feature 1", "WIP: continue feature 1"]}
anthropic/claude-opus-4.6:high PASS 100% 139 in → 177 out (116 reasoning)
WIP: start feature 1 WIP: continue feature 1
anthropic/claude-opus-4.6:high__json_schema PASS 100% 323 in → 116 out (64 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.6:low PASS 100% 134 in → 18 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.6:low__json_schema PASS 100% 323 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.6:max PASS 100% 133 in → 189 out (139 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.6:max__json_schema PASS 100% 327 in → 203 out (140 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.6:medium PASS 100% 134 in → 18 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 321 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.6:none PASS 100% 132 in → 18 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.6:none__json_schema PASS 100% 323 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 134 in → 175 out (114 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 322 in → 118 out (64 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.7:high PASS 100% 178 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.7:high__json_schema PASS 100% 422 in → 33 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.7:low PASS 100% 177 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.7:low__json_schema PASS 100% 422 in → 33 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.7:max PASS 100% 181 in → 231 out (68 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.7:max__json_schema PASS 100% 420 in → 33 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.7:medium PASS 100% 181 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 422 in → 33 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.7:none PASS 100% 180 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.7:none__json_schema PASS 100% 420 in → 33 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 180 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 421 in → 33 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.8:high PASS 100% 178 in → 87 out (55 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.8:high__json_schema PASS 100% 415 in → 31 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.8:low PASS 100% 177 in → 44 out (18 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.8:low__json_schema PASS 100% 414 in → 33 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
anthropic/claude-opus-4.8:max PASS 100% 171 in → 565 out (78 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.8:max__json_schema PASS 100% 418 in → 265 out (72 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
anthropic/claude-opus-4.8:medium PASS 100% 175 in → 136 out (66 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 419 in → 31 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.8:none PASS 100% 176 in → 22 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.8:none__json_schema PASS 100% 415 in → 30 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 179 in → 217 out (58 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 417 in → 33 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
anthropic/claude-sonnet-4.6:high PASS 100% 135 in → 118 out (70 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 320 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-sonnet-4.6:low PASS 100% 136 in → 18 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 323 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-sonnet-4.6:max PASS 100% 137 in → 196 out (142 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 320 in → 195 out (119 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 134 in → 18 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 321 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-sonnet-4.6:none PASS 100% 135 in → 18 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 325 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 134 in → 159 out (98 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 322 in → 182 out (96 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-sonnet-5:high__json_schema PASS 100% 414 in → 31 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-sonnet-5:medium PASS 100% 173 in → 103 out (36 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 414 in → 31 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-sonnet-5:none PASS 100% 179 in → 22 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
anthropic/claude-sonnet-5:none__json_schema PASS 100% 416 in → 30 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 170 in → 44 out (7 reasoning)
WIP: continue feature 1 WIP: start feature 1
arcee-ai/trinity-large-thinking:high PASS 100% 124 in → 1,208 out (1,229 reasoning)
WIP: continue feature 1 WIP: start feature 1
arcee-ai/trinity-large-thinking:low PASS 100% 123 in → 542 out (555 reasoning)
WIP: continue feature 1 WIP: start feature 1
arcee-ai/trinity-large-thinking:medium PASS 100% 122 in → 633 out (645 reasoning)
WIP: continue feature 1 WIP: start feature 1
arcee-ai/trinity-large-thinking:xhigh PASS 100% 127 in → 451 out (452 reasoning)
WIP: continue feature 1 WIP: start feature 1
arcee-ai/trinity-mini:medium PASS 100% 122 in → 436 out (430 reasoning)
WIP: start feature 1 WIP: continue feature 1
arcee-ai/trinity-mini:xhigh PASS 100% 127 in → 350 out (315 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-flash-0731:high PASS 100% 205 in → 223 out (207 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-flash-0731:low PASS 100% 123 in → 47 out (33 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 132 in → 453 out (395 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
deepseek/deepseek-v4-flash-0731:max PASS 100% 216 in → 6,250 out (6,234 reasoning)
WIP: start feature 1 WIP: continue feature 1
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 203 in → 3,184 out (3,076 reasoning)
WIP: start feature 1 WIP: continue feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: start feature 1", "WIP: continue feature 1" ] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 122 in → 243 out (283 reasoning)
WIP: start feature 1 WIP: continue feature 1
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 123 in → 46 out (29 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-flash:high PASS 100% 126 in → 226 out (209 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 125 in → 365 out (319 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue feature 1", "WIP: start feature 1"] }
deepseek/deepseek-v4-flash:low PASS 100% 127 in → 326 out (309 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-flash:medium PASS 100% 123 in → 267 out (251 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 123 in → 369 out (341 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
deepseek/deepseek-v4-flash:none PASS 100% 128 in → 15 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-flash:xhigh PASS 100% 122 in → 215 out (198 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 324 in → 612 out (587 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
deepseek/deepseek-v4-pro:low PASS 100% 123 in → 313 out (297 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 327 in → 151 out (126 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
deepseek/deepseek-v4-pro:medium PASS 100% 124 in → 223 out (184 reasoning)
WIP: continue feature 1 WIP: start feature 1
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 324 in → 31 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 200 in → 493 out (476 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3-flash-preview:high PASS 100% 125 in → 321 out (306 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3-flash-preview:high__json_schema PASS 100% 121 in → 480 out (461 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3-flash-preview:low PASS 100% 125 in → 274 out (259 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3-flash-preview:low__json_schema PASS 100% 203 in → 379 out (360 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3-flash-preview:medium PASS 100% 121 in → 300 out (285 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3-flash-preview:medium__json_schema PASS 100% 204 in → 477 out (458 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3-flash-preview:xhigh PASS 100% 124 in → 272 out (257 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 199 in → 260 out (241 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 122 in → 502 out (487 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 204 in → 333 out (312 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 122 in → 195 out (180 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.1-flash-lite-preview:medium PASS 100% 124 in → 305 out (290 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 123 in → 543 out (528 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 197 in → 1,114 out (1,095 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.1-pro-preview:high PASS 100% 122 in → 526 out (511 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 125 in → 389 out (355 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
google/gemini-3.1-pro-preview:low PASS 100% 122 in → 336 out (321 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 203 in → 393 out (374 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.1-pro-preview:medium PASS 100% 124 in → 380 out (365 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 204 in → 486 out (467 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 123 in → 459 out (444 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 205 in → 578 out (559 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.5-flash-lite:high PASS 100% 123 in → 1,356 out (1,341 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 125 in → 1,769 out (1,750 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.5-flash-lite:low PASS 100% 126 in → 15 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.5-flash-lite:max PASS 100% 124 in → 1,602 out (1,587 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 205 in → 1,136 out (1,102 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
google/gemini-3.5-flash-lite:medium PASS 100% 123 in → 1,163 out (1,148 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 126 in → 582 out (563 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 125 in → 1,369 out (1,354 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 121 in → 651 out (632 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.5-flash:high PASS 100% 124 in → 464 out (449 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.5-flash:high__json_schema PASS 100% 204 in → 704 out (685 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.5-flash:low PASS 100% 126 in → 325 out (310 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.5-flash:low__json_schema PASS 100% 204 in → 239 out (220 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.5-flash:medium PASS 100% 123 in → 378 out (363 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.5-flash:medium__json_schema PASS 100% 203 in → 826 out (807 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.5-flash:xhigh PASS 100% 123 in → 546 out (531 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 198 in → 854 out (835 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.6-flash:high PASS 100% 123 in → 707 out (692 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.6-flash:high__json_schema PASS 100% 205 in → 852 out (833 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.6-flash:low PASS 100% 125 in → 346 out (331 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.6-flash:low__json_schema PASS 100% 204 in → 34 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
google/gemini-3.6-flash:max PASS 100% 124 in → 4,423 out (4,408 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.6-flash:max__json_schema PASS 100% 126 in → 1,067 out (1,048 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.6-flash:medium PASS 100% 123 in → 2,382 out (2,367 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.6-flash:medium__json_schema PASS 100% 200 in → 542 out (523 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemini-3.6-flash:xhigh PASS 100% 125 in → 530 out (515 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 121 in → 2,198 out (2,179 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
google/gemma-4-26b-a4b-it:high PASS 100% 141 in → 287 out (271 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 136 in → 291 out (203 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 139 in → 333 out (256 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 136 in → 334 out (230 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 138 in → 321 out (251 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 139 in → 346 out (230 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 133 in → 16 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 136 in → 31 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 141 in → 291 out (220 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 140 in → 331 out (227 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
google/gemma-4-31b-it:high PASS 100% 140 in → 242 out (1 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemma-4-31b-it:high__json_schema PASS 100% 138 in → 228 out (145 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
google/gemma-4-31b-it:low PASS 100% 138 in → 230 out (162 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemma-4-31b-it:medium PASS 100% 141 in → 243 out (227 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemma-4-31b-it:medium__json_schema PASS 100% 140 in → 283 out (191 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
google/gemma-4-31b-it:none PASS 100% 135 in → 16 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemma-4-31b-it:xhigh PASS 100% 138 in → 295 out (231 reasoning)
WIP: continue feature 1 WIP: start feature 1
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 141 in → 327 out (244 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
ibm-granite/granite-4.1-8b PASS 100% 132 in → 16 out
WIP: continue feature 1 WIP: start feature 1
ibm-granite/granite-4.1-8b__json_schema PASS 100% 132 in → 25 out
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue feature 1", "WIP: start feature 1"] }
inclusionai/ling-2.6-flash PASS 100% 149 in → 16 out
WIP: start feature 1 WIP: continue feature 1
inclusionai/ling-2.6-flash__json_schema PASS 100% 149 in → 36 out
WIP: start feature 1 WIP: continue feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: start feature 1", "WIP: continue feature 1" ] }
minimax/minimax-m2.5:high PASS 100% 156 in → 664 out (651 reasoning)
WIP: continue feature 1 WIP: start feature 1
minimax/minimax-m2.5:high__json_schema PASS 100% 156 in → 355 out (324 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
minimax/minimax-m2.5:low PASS 100% 157 in → 665 out (642 reasoning)
WIP: continue feature 1 WIP: start feature 1
minimax/minimax-m2.5:low__json_schema PASS 100% 164 in → 305 out (276 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
minimax/minimax-m2.5:medium__json_schema PASS 100% 157 in → 328 out (306 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
minimax/minimax-m2.5:xhigh PASS 100% 157 in → 419 out (389 reasoning)
WIP: continue feature 1 WIP: start feature 1
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 157 in → 491 out (476 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
minimax/minimax-m2.7:high PASS 100% 159 in → 354 out (336 reasoning)
WIP: continue feature 1 WIP: start feature 1
minimax/minimax-m2.7:high__json_schema PASS 100% 307 in → 560 out (537 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
minimax/minimax-m2.7:low PASS 100% 160 in → 781 out (765 reasoning)
WIP: continue feature 1 WIP: start feature 1
minimax/minimax-m2.7:low__json_schema PASS 100% 309 in → 773 out (741 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
minimax/minimax-m2.7:medium PASS 100% 155 in → 527 out (552 reasoning)
WIP: continue feature 1 WIP: start feature 1
minimax/minimax-m2.7:xhigh PASS 100% 157 in → 308 out (297 reasoning)
WIP: continue feature 1 WIP: start feature 1
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 155 in → 427 out (422 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
minimax/minimax-m3:high PASS 100% 289 in → 165 out (133 reasoning)
WIP: continue feature 1 WIP: start feature 1
minimax/minimax-m3:low PASS 100% 295 in → 174 out (135 reasoning)
WIP: continue feature 1 WIP: start feature 1
minimax/minimax-m3:low__json_schema PASS 100% 283 in → 31 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
minimax/minimax-m3:medium PASS 100% 295 in → 175 out (134 reasoning)
WIP: continue feature 1 WIP: start feature 1
minimax/minimax-m3:medium__json_schema PASS 100% 280 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
minimax/minimax-m3:xhigh PASS 100% 296 in → 156 out (126 reasoning)
WIP: continue feature 1 WIP: start feature 1
mistralai/devstral-2512__json_schema PASS 100% 133 in → 24 out
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
mistralai/mistral-medium-3-5:high PASS 100% 144 in → 420 out (347 reasoning)
WIP: continue feature 1 WIP: start feature 1
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 134 in → 400 out (302 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
mistralai/mistral-medium-3-5:low PASS 100% 144 in → 438 out (363 reasoning)
WIP: continue feature 1 WIP: start feature 1
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 135 in → 1,179 out (1,129 reasoning)
WIP: start feature 1 WIP: continue feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: start feature 1", "WIP: continue feature 1"]}
mistralai/mistral-medium-3-5:medium PASS 100% 143 in → 529 out (480 reasoning)
WIP: continue feature 1 WIP: start feature 1
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 132 in → 486 out (433 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 130 in → 24 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 146 in → 3,443 out (3,131 reasoning)
WIP: continue feature 1 WIP: start feature 1
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 132 in → 289 out (235 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
moonshotai/kimi-k2.6:high PASS 100% 123 in → 2,287 out (2,271 reasoning)
WIP: start feature 1 WIP: continue feature 1
moonshotai/kimi-k2.6:low PASS 100% 126 in → 665 out (669 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k2.6:low__json_schema PASS 100% 122 in → 1,536 out (985 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
moonshotai/kimi-k2.6:medium PASS 100% 126 in → 457 out (442 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 122 in → 989 out (957 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
moonshotai/kimi-k2.6:xhigh PASS 100% 125 in → 646 out (655 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 126 in → 214 out (192 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
moonshotai/kimi-k2.7-code:high PASS 100% 122 in → 298 out (281 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 128 in → 209 out (167 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
moonshotai/kimi-k2.7-code:low PASS 100% 126 in → 246 out (229 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 128 in → 254 out (224 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
moonshotai/kimi-k2.7-code:medium PASS 100% 123 in → 180 out (157 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 219 in → 287 out (263 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 123 in → 251 out (234 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 125 in → 156 out (124 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
moonshotai/kimi-k3:high PASS 100% 200 in → 126 out (96 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k3:high__json_schema PASS 100% 201 in → 182 out (152 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k3:low PASS 100% 201 in → 836 out (806 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k3:low__json_schema PASS 100% 201 in → 169 out (139 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k3:max PASS 100% 202 in → 315 out (285 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k3:max__json_schema PASS 100% 201 in → 230 out (200 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k3:medium PASS 100% 201 in → 130 out (100 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k3:medium__json_schema PASS 100% 200 in → 145 out (115 reasoning)
WIP: continue feature 1 WIP: start feature 1
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 201 in → 259 out (229 reasoning)
WIP: continue feature 1 WIP: start feature 1
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 143 in → 417 out (388 reasoning)
WIP: continue feature 1 WIP: start feature 1
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 145 in → 302 out (273 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 145 in → 444 out (398 reasoning)
WIP: continue feature 1 WIP: start feature 1
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 141 in → 812 out (804 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 143 in → 446 out (414 reasoning)
WIP: continue feature 1 WIP: start feature 1
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 146 in → 374 out (341 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 145 in → 16 out (0 reasoning)
WIP: start feature 1 WIP: continue feature 1
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 143 in → 41 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 145 in → 414 out (417 reasoning)
WIP: continue feature 1 WIP: start feature 1
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 146 in → 364 out (335 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 143 in → 164 out (141 reasoning)
WIP: continue feature 1 WIP: start feature 1
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 145 in → 198 out (159 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 146 in → 200 out (152 reasoning)
WIP: continue feature 1 WIP: start feature 1
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 145 in → 245 out (207 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 146 in → 274 out (257 reasoning)
WIP: continue feature 1 WIP: start feature 1
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 146 in → 164 out (124 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 145 in → 16 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 144 in → 31 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 143 in → 253 out (208 reasoning)
WIP: continue feature 1 WIP: start feature 1
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 144 in → 281 out (240 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
openai/gpt-5.4-mini:high PASS 100% 128 in → 117 out (96 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4-mini:high__json_schema PASS 100% 174 in → 151 out (120 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4-mini:low PASS 100% 125 in → 51 out (30 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4-mini:low__json_schema PASS 100% 173 in → 52 out (21 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4-mini:medium PASS 100% 124 in → 179 out (158 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4-mini:medium__json_schema PASS 100% 173 in → 121 out (90 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4-mini:none PASS 100% 123 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4-mini:none__json_schema PASS 100% 174 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4-mini:xhigh PASS 100% 123 in → 179 out (158 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 174 in → 348 out (317 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4-nano:high PASS 100% 125 in → 77 out (56 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4-nano:high__json_schema PASS 100% 170 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4-nano:low PASS 100% 122 in → 49 out (28 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4-nano:low__json_schema PASS 100% 174 in → 59 out (28 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4-nano:medium PASS 100% 127 in → 49 out (28 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4-nano:medium__json_schema PASS 100% 175 in → 64 out (33 reasoning)
WIP: start feature 1 WIP: continue feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: start feature 1","WIP: continue feature 1"]}
openai/gpt-5.4-nano:none__json_schema PASS 100% 171 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4-nano:xhigh PASS 100% 124 in → 265 out (244 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 169 in → 1,394 out (1,363 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4:high PASS 100% 126 in → 328 out (307 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4:high__json_schema PASS 100% 173 in → 303 out (272 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4:low PASS 100% 124 in → 40 out (19 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4:low__json_schema PASS 100% 171 in → 49 out (18 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4:medium PASS 100% 130 in → 74 out (53 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4:medium__json_schema PASS 100% 169 in → 81 out (50 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4:none PASS 100% 126 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4:none__json_schema PASS 100% 171 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.4:xhigh PASS 100% 125 in → 147 out (126 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.4:xhigh__json_schema PASS 100% 176 in → 161 out (130 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.5:high PASS 100% 128 in → 102 out (81 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.5:high__json_schema PASS 100% 174 in → 84 out (53 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.5:low PASS 100% 126 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.5:low__json_schema PASS 100% 172 in → 61 out (30 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.5:medium PASS 100% 128 in → 68 out (47 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.5:medium__json_schema PASS 100% 174 in → 77 out (46 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.5:none PASS 100% 129 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.5:none__json_schema PASS 100% 176 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.5:xhigh PASS 100% 124 in → 189 out (168 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.5:xhigh__json_schema PASS 100% 171 in → 185 out (154 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-luna:high PASS 100% 125 in → 38 out (17 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-luna:high__json_schema PASS 100% 173 in → 69 out (38 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-luna:low PASS 100% 123 in → 39 out (18 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-luna:low__json_schema PASS 100% 172 in → 44 out (13 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-luna:max PASS 100% 126 in → 98 out (77 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-luna:max__json_schema PASS 100% 170 in → 547 out (516 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-luna:medium PASS 100% 125 in → 39 out (18 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-luna:medium__json_schema PASS 100% 172 in → 56 out (25 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-luna:none PASS 100% 126 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-luna:none__json_schema PASS 100% 173 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-luna:xhigh PASS 100% 123 in → 98 out (77 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 173 in → 93 out (62 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-sol:high PASS 100% 128 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-sol:high__json_schema PASS 100% 174 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-sol:low PASS 100% 127 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-sol:low__json_schema PASS 100% 173 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-sol:max PASS 100% 127 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-sol:max__json_schema PASS 100% 173 in → 238 out (207 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-sol:medium PASS 100% 124 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-sol:medium__json_schema PASS 100% 169 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-sol:none PASS 100% 125 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-sol:none__json_schema PASS 100% 171 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-sol:xhigh PASS 100% 127 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 172 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-terra:high PASS 100% 125 in → 35 out (14 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-terra:high__json_schema PASS 100% 172 in → 46 out (15 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-terra:low PASS 100% 125 in → 36 out (15 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-terra:low__json_schema PASS 100% 171 in → 46 out (15 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-terra:max PASS 100% 126 in → 59 out (38 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-terra:max__json_schema PASS 100% 173 in → 104 out (73 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-terra:medium PASS 100% 128 in → 36 out (15 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-terra:medium__json_schema PASS 100% 173 in → 46 out (15 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-terra:none PASS 100% 126 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-terra:none__json_schema PASS 100% 171 in → 29 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-5.6-terra:xhigh PASS 100% 129 in → 35 out (14 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 176 in → 48 out (17 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-oss-120b:high PASS 100% 187 in → 246 out (225 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-oss-120b:high__json_schema PASS 100% 172 in → 299 out (279 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
openai/gpt-oss-120b:low PASS 100% 183 in → 53 out (24 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-oss-120b:low__json_schema PASS 100% 188 in → 157 out (116 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-oss-120b:medium PASS 100% 193 in → 234 out (210 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-oss-120b:medium__json_schema PASS 100% 184 in → 178 out (138 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
openai/gpt-oss-120b:xhigh PASS 100% 185 in → 354 out (343 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-oss-20b:high PASS 100% 190 in → 392 out (377 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-oss-20b:high__json_schema PASS 100% 182 in → 463 out (451 reasoning)
WIP: start feature 1 WIP: continue feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: start feature 1","WIP: continue feature 1"]}
openai/gpt-oss-20b:low PASS 100% 184 in → 52 out (22 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-oss-20b:low__json_schema PASS 100% 172 in → 54 out (18 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
openai/gpt-oss-20b:medium PASS 100% 190 in → 261 out (234 reasoning)
WIP: continue feature 1 WIP: start feature 1
openai/gpt-oss-20b:medium__json_schema PASS 100% 184 in → 152 out (99 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
openai/gpt-oss-20b:xhigh PASS 100% 184 in → 711 out (686 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-m.1:high PASS 100% 138 in → 298 out (282 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-m.1:high__json_schema PASS 100% 133 in → 395 out (374 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
poolside/laguna-m.1:low PASS 100% 136 in → 278 out (262 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-m.1:low__json_schema PASS 100% 137 in → 312 out (238 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
poolside/laguna-m.1:medium PASS 100% 139 in → 290 out (274 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-m.1:medium__json_schema PASS 100% 136 in → 214 out (192 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
poolside/laguna-m.1:none PASS 100% 134 in → 15 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-m.1:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
poolside/laguna-m.1:xhigh__json_schema PASS 100% 130 in → 311 out (289 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
poolside/laguna-xs-2.1:high PASS 100% 137 in → 231 out (217 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-xs-2.1:low PASS 100% 139 in → 237 out (223 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-xs-2.1:low__json_schema PASS 100% 137 in → 535 out (504 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
poolside/laguna-xs-2.1:medium PASS 100% 137 in → 390 out (376 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-xs-2.1:none PASS 100% 134 in → 13 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-xs-2.1:xhigh PASS 100% 133 in → 254 out (240 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 133 in → 249 out (214 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
poolside/laguna-xs.2:high PASS 100% 172 in → 163 out (147 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-xs.2:high__json_schema PASS 100% 176 in → 258 out (236 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
poolside/laguna-xs.2:low PASS 100% 173 in → 177 out (161 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-xs.2:low__json_schema PASS 100% 171 in → 175 out (154 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
poolside/laguna-xs.2:medium PASS 100% 174 in → 183 out (167 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-xs.2:medium__json_schema PASS 100% 170 in → 185 out (164 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
poolside/laguna-xs.2:none__json_schema PASS 100% 175 in → 20 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
poolside/laguna-xs.2:xhigh PASS 100% 172 in → 259 out (242 reasoning)
WIP: continue feature 1 WIP: start feature 1
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 174 in → 242 out (208 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
qwen/qwen3.6-27b:high PASS 100% 136 in → 696 out (616 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-27b:high__json_schema PASS 100% 139 in → 1,125 out (990 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
qwen/qwen3.6-27b:low PASS 100% 139 in → 827 out (728 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-27b:low__json_schema PASS 100% 138 in → 929 out (1 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
qwen/qwen3.6-27b:medium PASS 100% 134 in → 1,066 out (973 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-27b:none PASS 100% 137 in → 15 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-27b:none__json_schema PASS 100% 139 in → 36 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 136 in → 976 out (849 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-35b-a3b:high PASS 100% 136 in → 1,078 out (1,058 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 138 in → 549 out (441 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 137 in → 546 out (486 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 137 in → 889 out (785 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 137 in → 932 out (816 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 135 in → 760 out (670 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue feature 1", "WIP: start feature 1"] }
qwen/qwen3.6-35b-a3b:none PASS 100% 140 in → 16 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 135 in → 467 out (374 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 137 in → 1,396 out (1,255 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
qwen/qwen3.6-flash:high PASS 100% 138 in → 779 out (759 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-flash:low PASS 100% 138 in → 1,190 out (1,170 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-flash:medium PASS 100% 137 in → 1,469 out (1,449 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-flash:none PASS 100% 139 in → 15 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.6-flash:xhigh PASS 100% 138 in → 1,009 out (989 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-flash:high PASS 100% 137 in → 830 out (810 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-flash:low PASS 100% 138 in → 843 out (823 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-flash:max PASS 100% 139 in → 995 out (975 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-flash:medium PASS 100% 139 in → 1,118 out (1,098 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-flash:none PASS 100% 137 in → 15 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-flash:xhigh PASS 100% 139 in → 1,019 out (999 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-max:high PASS 100% 137 in → 507 out (487 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-max:low PASS 100% 140 in → 390 out (370 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-max:medium PASS 100% 140 in → 248 out (228 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-max:none PASS 100% 138 in → 15 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-max:xhigh PASS 100% 136 in → 490 out (470 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-plus:high PASS 100% 138 in → 207 out (187 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-plus:low PASS 100% 140 in → 816 out (796 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-plus:medium PASS 100% 139 in → 253 out (233 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-plus:none PASS 100% 138 in → 15 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
qwen/qwen3.7-plus:xhigh PASS 100% 132 in → 803 out (783 reasoning)
WIP: continue feature 1 WIP: start feature 1
tencent/hy3:high PASS 100% 130 in → 801 out (784 reasoning)
WIP: continue feature 1 WIP: start feature 1
tencent/hy3:high__json_schema PASS 100% 134 in → 591 out (524 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
tencent/hy3:low PASS 100% 133 in → 322 out (305 reasoning)
WIP: continue feature 1 WIP: start feature 1
tencent/hy3:low__json_schema PASS 100% 134 in → 379 out (338 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue feature 1", "WIP: start feature 1"] }
tencent/hy3:medium PASS 100% 133 in → 367 out (319 reasoning)
WIP: continue feature 1 WIP: start feature 1
tencent/hy3:medium__json_schema PASS 100% 130 in → 511 out (484 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue feature 1", "WIP: start feature 1"] }
tencent/hy3:none PASS 100% 135 in → 16 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
tencent/hy3:none__json_schema PASS 100% 131 in → 27 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
tencent/hy3:xhigh PASS 100% 132 in → 452 out (435 reasoning)
WIP: continue feature 1 WIP: start feature 1
tencent/hy3:xhigh__json_schema PASS 100% 132 in → 438 out (390 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
thinkingmachines/inkling-small:high PASS 100% 135 in → 426 out (405 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling-small:high__json_schema PASS 100% 135 in → 494 out (465 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
thinkingmachines/inkling-small:low PASS 100% 130 in → 59 out (31 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling-small:low__json_schema PASS 100% 137 in → 116 out (81 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
thinkingmachines/inkling-small:max PASS 100% 136 in → 364 out (323 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling-small:max__json_schema PASS 100% 135 in → 561 out (526 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
thinkingmachines/inkling-small:medium PASS 100% 134 in → 408 out (367 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling-small:medium__json_schema PASS 100% 135 in → 312 out (281 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 135 in → 233 out (212 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 135 in → 250 out (222 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
thinkingmachines/inkling:high PASS 100% 134 in → 247 out (220 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling:low PASS 100% 135 in → 51 out (30 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling:max PASS 100% 136 in → 298 out (267 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling:max__json_schema PASS 100% 134 in → 281 out (255 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
thinkingmachines/inkling:medium PASS 100% 133 in → 219 out (197 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling:medium__json_schema PASS 100% 137 in → 110 out (81 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": ["WIP: continue feature 1", "WIP: start feature 1"] }
thinkingmachines/inkling:none PASS 100% 130 in → 19 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling:xhigh PASS 100% 139 in → 192 out (170 reasoning)
WIP: continue feature 1 WIP: start feature 1
thinkingmachines/inkling:xhigh__json_schema PASS 100% 136 in → 293 out (258 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
x-ai/grok-4.3:high PASS 100% 296 in → 451 out (436 reasoning)
WIP: continue feature 1 WIP: start feature 1
x-ai/grok-4.3:high__json_schema PASS 100% 364 in → 970 out (951 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
x-ai/grok-4.3:low PASS 100% 307 in → 379 out (364 reasoning)
WIP: continue feature 1 WIP: start feature 1
x-ai/grok-4.3:low__json_schema PASS 100% 373 in → 412 out (383 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
x-ai/grok-4.3:max PASS 100% 297 in → 247 out (232 reasoning)
WIP: continue feature 1 WIP: start feature 1
x-ai/grok-4.3:max__json_schema PASS 100% 365 in → 676 out (647 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
x-ai/grok-4.3:medium PASS 100% 303 in → 857 out (842 reasoning)
WIP: continue feature 1 WIP: start feature 1
x-ai/grok-4.3:medium__json_schema PASS 100% 371 in → 726 out (697 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
x-ai/grok-4.3:xhigh PASS 100% 295 in → 356 out (341 reasoning)
WIP: continue feature 1 WIP: start feature 1
x-ai/grok-4.3:xhigh__json_schema PASS 100% 367 in → 631 out (612 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
x-ai/grok-4.5:high PASS 100% 322 in → 251 out (238 reasoning)
WIP: continue feature 1 WIP: start feature 1
x-ai/grok-4.5:high__json_schema PASS 100% 403 in → 580 out (552 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
x-ai/grok-4.5:low PASS 100% 324 in → 218 out (205 reasoning)
WIP: continue feature 1 WIP: start feature 1
x-ai/grok-4.5:low__json_schema PASS 100% 401 in → 387 out (359 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
x-ai/grok-4.5:max PASS 100% 321 in → 234 out (221 reasoning)
WIP: continue feature 1 WIP: start feature 1
x-ai/grok-4.5:max__json_schema PASS 100% 402 in → 631 out (603 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
x-ai/grok-4.5:medium PASS 100% 319 in → 292 out (279 reasoning)
WIP: continue feature 1 WIP: start feature 1
x-ai/grok-4.5:medium__json_schema PASS 100% 404 in → 455 out (437 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
x-ai/grok-4.5:xhigh PASS 100% 323 in → 259 out (246 reasoning)
WIP: continue feature 1 WIP: start feature 1
x-ai/grok-4.5:xhigh__json_schema PASS 100% 400 in → 647 out (629 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["WIP: continue feature 1","WIP: start feature 1"]}
z-ai/glm-4.7-flash:high PASS 100% 123 in → 453 out (372 reasoning)
WIP: continue feature 1 WIP: start feature 1
z-ai/glm-4.7-flash:low PASS 100% 123 in → 1,998 out (1,902 reasoning)
WIP: start feature 1 WIP: continue feature 1
z-ai/glm-4.7-flash:medium PASS 100% 127 in → 497 out (415 reasoning)
WIP: start feature 1 WIP: continue feature 1
z-ai/glm-4.7-flash:none PASS 100% 125 in → 16 out (0 reasoning)
WIP: start feature 1 WIP: continue feature 1
z-ai/glm-4.7-flash:none__json_schema PASS 100% 127 in → 27 out (0 reasoning)
WIP: start feature 1 WIP: continue feature 1
JSON Schema Structured Output
(raw) {"commits": [ "WIP: start feature 1", "WIP: continue feature 1" ]}
z-ai/glm-4.7-flash:xhigh PASS 100% 122 in → 551 out (471 reasoning)
WIP: start feature 1 WIP: continue feature 1
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 125 in → 522 out (428 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["WIP: continue feature 1", "WIP: start feature 1"]}
z-ai/glm-5.2:high PASS 100% 133 in → 492 out (455 reasoning)
WIP: continue feature 1 WIP: start feature 1
z-ai/glm-5.2:high__json_schema PASS 100% 131 in → 459 out (387 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
z-ai/glm-5.2:low PASS 100% 131 in → 522 out (505 reasoning)
WIP: continue feature 1 WIP: start feature 1
z-ai/glm-5.2:low__json_schema PASS 100% 137 in → 343 out (311 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ]}
z-ai/glm-5.2:none PASS 100% 127 in → 16 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
z-ai/glm-5.2:none__json_schema PASS 100% 126 in → 27 out (0 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
z-ai/glm-5.2:xhigh PASS 100% 132 in → 722 out (681 reasoning)
WIP: continue feature 1 WIP: start feature 1
z-ai/glm-5.2:xhigh__json_schema PASS 100% 319 in → 524 out (492 reasoning)
WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
arcee-ai/trinity-mini:low FAIL 50% 123 in → 501 out (487 reasoning)
WIP: continue feature1 WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature1']
minimax/minimax-m2.5:medium FAIL 50% 161 in → 261 out (245 reasoning)
: continue feature 1 WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: continue feature 1']; Extra selected commit messages: [': continue feature 1']
thinkingmachines/inkling-small:none FAIL 50% 132 in → 18 out (0 reasoning)
WIP: start feature 1 WIP: continue feature 1<|end_message|>
Failure: Missing expected commit messages: ['WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1']
anthropic/claude-haiku-4.5:none FAIL 0% 137 in → 30 out (0 reasoning)
79fb365 WIP: continue feature 1 0cdf4a0 WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 321 in → 19 out (0 reasoning)
ec501e8 cffb367
JSON Schema Structured Output
(raw) {"commits": ["ec501e8", "cffb367"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
anthropic/claude-sonnet-5:high FAIL 0% 174 in → 38 out (0 reasoning)
``` d414ed3 WIP: continue feature 1 19e69e3 WIP: start feature 1 ```
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['```', 'WIP: continue feature 1', 'WIP: start feature 1', '```']
anthropic/claude-sonnet-5:low FAIL 0% 176 in → 37 out (0 reasoning)
``` e7cc848 WIP: continue feature 1 7f34168 WIP: start feature 1 ```
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['```', 'WIP: continue feature 1', 'WIP: start feature 1', '```']
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 415 in → 43 out (0 reasoning)
8a2f2fc WIP: continue feature 1 8b32111 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["8a2f2fc WIP: continue feature 1","8b32111 WIP: start feature 1"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 415 in → 139 out (32 reasoning)
3deaa7f WIP: continue feature 1 dba6cb3 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["3deaa7f WIP: continue feature 1","dba6cb3 WIP: start feature 1"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 124 in → 309 out (313 reasoning)
WIP: continue feature 1WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1']
arcee-ai/trinity-mini:high__json_schema FAIL 0% 121 in → 467 out (426 reasoning)
975d960 Start feature 2 30cc3ba WIP: continue feature 1 d149422 WIP: start feature 1 b35025e Base commit
JSON Schema Structured Output
(raw) { "commits": [ "975d960 Start feature 2", "30cc3ba WIP: continue feature 1", "d149422 WIP: start feature 1", "b35025e Base commit" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['Start feature 2', 'WIP: continue feature 1', 'WIP: start feature 1', 'Base commit']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 123 in → 349 out (296 reasoning)
632386b Start feature 2 8506646 WIP: continue feature 1 95f0eb8 WIP: start feature 1 d51df5a Base commit
JSON Schema Structured Output
(raw) { "commits": [ "632386b Start feature 2", "8506646 WIP: continue feature 1", "95f0eb8 WIP: start feature 1", "d51df5a Base commit" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['Start feature 2', 'WIP: continue feature 1', 'WIP: start feature 1', 'Base commit']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 126 in → 392 out (345 reasoning)
4f39d93 WIP: continue feature 1 54eac7c WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits":[ "4f39d93 WIP: continue feature 1", "54eac7c WIP: start feature 1" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 126 in → 437 out (405 reasoning)
45177d4 WIP: continue feature 1 fd3a0a3 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "45177d4 WIP: continue feature 1", "fd3a0a3 WIP: start feature 1" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
deepseek/deepseek-v4-flash-0731:high__json_schema FAIL 0% 121 in → 80 out (48 reasoning)
4539eaf WIP: continue feature 1 5c84883 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": ["4539eaf WIP: continue feature 1", "5c84883 WIP: start feature 1"] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
deepseek/deepseek-v4-flash-0731:medium__json_schema FAIL 0% 124 in → 64 out (37 reasoning)
86d982b WIP: continue feature 1 d20f118 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": ["86d982b WIP: continue feature 1", "d20f118 WIP: start feature 1"] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 122 in → 22 out (0 reasoning)
462775f WIP: continue feature 1 50e83ce WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 123 in → 35 out (0 reasoning)
db5be09 WIP: continue feature 1 c4a3508 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["db5be09 WIP: continue feature 1", "c4a3508 WIP: start feature 1"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 124 in → 56 out (38 reasoning)
a544b21 524f4ef
JSON Schema Structured Output
(raw) { "commits": ["a544b21", "524f4ef"] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
Invalid JSON. Output: WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 126 in → 42 out (0 reasoning)
0b6b0f6 WIP: continue feature 1 aaf3819 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "0b6b0f6 WIP: continue feature 1", "aaf3819 WIP: start feature 1" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0% 203 in → 741 out (715 reasoning)
ed1b07e 5904508
JSON Schema Structured Output
(raw) { "commits": [ "ed1b07e", "5904508" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
deepseek/deepseek-v4-pro:high FAIL 0% 125 in → 149 out (123 reasoning)
7f89112 WIP: continue feature 1 75c8c1e WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 120 in → 38 out (0 reasoning)
b668df8 WIP: continue feature 1 220b077 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "b668df8 WIP: continue feature 1", "220b077 WIP: start feature 1" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
deepseek/deepseek-v4-pro:none FAIL 0% 122 in → 24 out (0 reasoning)
7c7f030 WIP: continue feature 1 6122412 WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 121 in → 33 out (0 reasoning)
6390986 WIP: continue feature 1 c687344 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "6390986 WIP: continue feature 1", "c687344 WIP: start feature 1" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 0% 203 in → 154 out (108 reasoning)
2f3795e WIP: continue feature 1 f8a81a8 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "2f3795e WIP: continue feature 1", "f8a81a8 WIP: start feature 1" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 123 in → 350 out (320 reasoning)
6cf4be3 WIP: continue feature 1 ad892e6 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["6cf4be3 WIP: continue feature 1", "ad892e6 WIP: start feature 1"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 203 in → 45 out (0 reasoning)
bd80b76 WIP: continue feature 1 67f63a4 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "bd80b76 WIP: continue feature 1", "67f63a4 WIP: start feature 1" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
google/gemma-4-31b-it:low__json_schema FAIL 0% 141 in → 327 out (305 reasoning)
> WIP: continue feature 1 > WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["> WIP: continue feature 1\n> WIP: start feature 1"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
google/gemma-4-31b-it:none__json_schema FAIL 0% 136 in → 33 out (0 reasoning)
ab4a623 WIP: continue feature 1 d5b6c24 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["ab4a623 WIP: continue feature 1","d5b6c24 WIP: start feature 1"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
liquid/lfm-2-24b-a2b FAIL 0% 133 in → 12 out
1f4bbd6 58b81cd
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
Invalid JSON. Output: 81b4210 WIP: continue feature 1 f5d8043 WIP: start feature 1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: OpenAI-compatible response contained provider error: Upstream error from Morph: undefined
minimax/minimax-m3:xhigh__json_schema FAIL 0% 243 in → 61 out (60 reasoning)
fb5e228 Start feature 2 6e8f193 WIP: continue feature 1 8b7ad9e WIP: start feature 1 2146811 Base commit
JSON Schema Structured Output
(raw) { "commits": [ "fb5e228 Start feature 2", "6e8f193 WIP: continue feature 1", "8b7ad9e WIP: start feature 1", "2146811 Base commit" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['Start feature 2', 'WIP: continue feature 1', 'WIP: start feature 1', 'Base commit']
mistralai/devstral-2512 FAIL 0% 134 in → 29 out
d348323 WIP: continue feature 1 38cd970 WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
mistralai/mistral-medium-3-5:none FAIL 0% 144 in → 28 out (0 reasoning)
e15bd38 WIP: continue feature 1 6b971fd WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
moonshotai/kimi-k2.6:high__json_schema FAIL 0% 126 in → 2,598 out (2,593 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"commits":[]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']
(empty output)
Failure: OpenAI-compatible response contained provider error: Provider returned error
openai/gpt-5.4-nano:none FAIL 0% 127 in → 31 out (0 reasoning)
3dcb7d4 WIP: continue feature 1 ec5c0a1 WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
openai/gpt-oss-120b:xhigh__json_schema FAIL 0% 185 in → 264 out (227 reasoning)
641cf31 WIP: continue feature 1 7b15b7a WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":[ "641cf31 WIP: continue feature 1", "7b15b7a WIP: start feature 1" ]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
openai/gpt-oss-20b:xhigh__json_schema FAIL 0% 190 in → 11 out (0 reasoning)
a3c132f
JSON Schema Structured Output
(raw) {"commits":["a3c132f"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['a3c132f']
poolside/laguna-m.1:xhigh FAIL 0% 133 in → 269 out (240 reasoning)
b3bb98e WIP: continue feature 1 7e6f183 WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
poolside/laguna-xs-2.1:high__json_schema FAIL 0% 137 in → 279 out (232 reasoning)
460f76e WIP: continue feature 1 57a80ab WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "460f76e WIP: continue feature 1", "57a80ab WIP: start feature 1" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
poolside/laguna-xs-2.1:medium__json_schema FAIL 0% 139 in → 321 out (285 reasoning)
e64d742 WIP: continue feature 1 3e4485a WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["e64d742 WIP: continue feature 1", "3e4485a WIP: start feature 1"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 134 in → 71 out (0 reasoning)
bea0668 Start feature 2 bbf1a9f WIP: continue feature 1 112352e WIP: start feature 1 ab2d4d8 Base commit
JSON Schema Structured Output
(raw) { "commits": [ "bea0668 Start feature 2", "bbf1a9f WIP: continue feature 1", "112352e WIP: start feature 1", "ab2d4d8 Base commit" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['Start feature 2', 'WIP: continue feature 1', 'WIP: start feature 1', 'Base commit']
poolside/laguna-xs.2:none FAIL 0% 170 in → 26 out (0 reasoning)
effb8d5 WIP: start feature 1 a842bc5 WIP: continue feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']
Invalid JSON. Output: WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:xhigh__json_schema FAIL 0% 139 in → 48 out (0 reasoning)
1dcb3cc WIP: continue feature 1 e046444 WIP: start feature 1
JSON Schema Structured Output
(raw) { "commits": [ "1dcb3cc WIP: continue feature 1", "e046444 WIP: start feature 1" ] }
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 139 in → 34 out (0 reasoning)
0dea569 WIP: continue feature 1 e4b4cd0 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits": ["0dea569 WIP: continue feature 1", "e4b4cd0 WIP: start feature 1"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1533f373-bf43-9452-8613-93adb3341630","request_id":"1533f373-bf43-9452-8613-93adb3341630"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-76b131e3-0bbe-9504-89b0-a15dafe311e9","request_id":"76b131e3-0bbe-9504-89b0-a15dafe311e9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1d6e603b-91ef-9295-b6cb-553ae1a1149d","request_id":"1d6e603b-91ef-9295-b6cb-553ae1a1149d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9d239d57-822d-943a-b393-71a400e840d6","request_id":"9d239d57-822d-943a-b393-71a400e840d6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a49f5765-332b-9bbf-8607-7bbcc7d02b03","request_id":"a49f5765-332b-9bbf-8607-7bbcc7d02b03"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-45cbc81d-1eb6-99cf-9477-e106300093f4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d6bf25ea-9282-926e-829e-05a425cd144d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a168bca0-1897-9604-a046-53b9c8734847"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c93014d9-cf0a-91dd-bf33-7014e9d1f6e1"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4ab58219-a234-9ef0-a362-03bb094c3aaa"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4212ea32-6a13-9417-bcdb-a5cc17d4b89e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ddb2c628-9cb1-9657-a056-1a0435e1c8ec","request_id":"ddb2c628-9cb1-9657-a056-1a0435e1c8ec"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1bd772c4-ce21-9599-a50d-3076c8cdeacd","request_id":"1bd772c4-ce21-9599-a50d-3076c8cdeacd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-77b1b29c-8d19-9292-8084-7a0899025d25","request_id":"77b1b29c-8d19-9292-8084-7a0899025d25"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-613ebba9-e826-9124-ae82-3a132ef52e28","request_id":"613ebba9-e826-9124-ae82-3a132ef52e28"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6deb7ddd-f55a-9b8a-b4f9-708266e5e8d4","request_id":"6deb7ddd-f55a-9b8a-b4f9-708266e5e8d4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3de9d6eb-8d4d-985d-816f-fa87bb1a391e","request_id":"3de9d6eb-8d4d-985d-816f-fa87bb1a391e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5b8c6dee-4268-95e6-8c49-8ca5705dda34","request_id":"5b8c6dee-4268-95e6-8c49-8ca5705dda34"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6bd04488-16f7-9a50-89e2-27fee264484d","request_id":"6bd04488-16f7-9a50-89e2-27fee264484d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-220e5cb3-6768-9a94-87f5-e1592c3e4e23","request_id":"220e5cb3-6768-9a94-87f5-e1592c3e4e23"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-226fc3ab-8447-91f9-991d-c723d8ada1b3","request_id":"226fc3ab-8447-91f9-991d-c723d8ada1b3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: WIP: continue feature 1 WIP: start feature 1{ "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue feature 1 WIP: start feature 1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: WIP: continue feature 1 WIP: start feature 1{ "commits": [ "WIP: continue feature 1", "WIP: start feature 1" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none FAIL 0% 298 in → 22 out (0 reasoning)
85e2d09 WIP: continue feature 1 c722f81 WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
x-ai/grok-4.3:none__json_schema FAIL 0% 365 in → 26 out (0 reasoning)
350380a WIP: continue feature 1 e2c6f61 WIP: start feature 1
JSON Schema Structured Output
(raw) {"commits":["350380a WIP: continue feature 1","e2c6f61 WIP: start feature 1"]}
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:medium FAIL 0% 131 in → 461 out (434 reasoning)
e344617 WIP: continue feature 1 32844e5 WIP: start feature 1
Failure: Missing expected commit messages: ['WIP: start feature 1', 'WIP: continue feature 1']; Extra selected commit messages: ['WIP: continue feature 1', 'WIP: start feature 1']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)