f012 — commit_squash

← commit_squash
Long-running feature with many small commits
Tests ability to handle a long-running feature with many small commits requiring reorganization. Evaluates scaling of squash analysis.
hard commit-squash long-running many-commits scalability squash
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'start' > feature.py
  5. 05 git add feature.py
  6. 06 git commit -m 'Initial feature'
  7. 07 echo 'start line1' > feature.py
  8. 08 git add feature.py
  9. 09 git commit -m 'WIP: add line1'
  10. 10 echo 'start line1 line2' > feature.py
  11. 11 git add feature.py
  12. 12 git commit -m 'WIP: add line2'
  13. 13 echo 'start line1 line2 line3' > feature.py
  14. 14 git add feature.py
  15. 15 git commit -m 'WIP: add line3'
  16. 16 echo 'start line1 line2 line3 line4' > feature.py
  17. 17 git add feature.py
  18. 18 git commit -m 'Complete feature'
Prompt
This repository has a feature built incrementally with many small commits. Identify all the WIP commits that should be marked as squash/fixup for a cleaner history. Do not include the initial or final feature anchor commit. Output ONLY the selected commit subject lines, one per line, with no explanation.
Expected
WIP: add line1 WIP: add line2 WIP: add line3
Fixture Metrics

Scoped model quality, cost, API time, and token usage for commit_squash/f012.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 173 in → 37 out (10 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-fable-5:high__json_schema PASS 100% 414 in → 49 out (10 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
anthropic/claude-fable-5:low PASS 100% 170 in → 25 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-fable-5:low__json_schema PASS 100% 413 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-fable-5:max PASS 100% 171 in → 303 out (129 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-fable-5:max__json_schema PASS 100% 407 in → 244 out (62 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
anthropic/claude-fable-5:medium PASS 100% 173 in → 37 out (10 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-fable-5:medium__json_schema PASS 100% 410 in → 37 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
anthropic/claude-fable-5:xhigh PASS 100% 169 in → 41 out (15 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 412 in → 56 out (26 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
anthropic/claude-haiku-4.5:high PASS 100% 162 in → 360 out (270 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 348 in → 645 out (512 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line1", "WIP: add line2", "WIP: add line3"]}
anthropic/claude-haiku-4.5:low PASS 100% 159 in → 275 out (234 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 346 in → 837 out (530 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line1", "WIP: add line2", "WIP: add line3"]}
anthropic/claude-haiku-4.5:medium PASS 100% 158 in → 290 out (222 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 348 in → 962 out (511 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line1", "WIP: add line2", "WIP: add line3"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 159 in → 425 out (338 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 350 in → 2,320 out (876 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line1", "WIP: add line2", "WIP: add line3"]}
anthropic/claude-opus-4.6:high PASS 100% 132 in → 23 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
anthropic/claude-opus-4.6:high__json_schema PASS 100% 321 in → 127 out (68 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.6:low PASS 100% 132 in → 23 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
anthropic/claude-opus-4.6:low__json_schema PASS 100% 316 in → 29 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.6:max PASS 100% 132 in → 217 out (152 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.6:max__json_schema PASS 100% 319 in → 155 out (93 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits":["WIP: add line1","WIP: add line2","WIP: add line3"]}
anthropic/claude-opus-4.6:medium PASS 100% 131 in → 23 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 316 in → 29 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.6:none PASS 100% 136 in → 23 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.6:none__json_schema PASS 100% 316 in → 29 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 130 in → 23 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 319 in → 124 out (68 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits":["WIP: add line1","WIP: add line2","WIP: add line3"]}
anthropic/claude-opus-4.7:high PASS 100% 174 in → 27 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.7:high__json_schema PASS 100% 416 in → 36 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.7:low PASS 100% 174 in → 27 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.7:low__json_schema PASS 100% 411 in → 37 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.7:max PASS 100% 178 in → 165 out (49 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.7:max__json_schema PASS 100% 414 in → 36 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.7:medium PASS 100% 179 in → 27 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 411 in → 36 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.7:none PASS 100% 175 in → 27 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.7:none__json_schema PASS 100% 418 in → 36 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 180 in → 27 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 414 in → 36 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 408 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.8:low PASS 100% 172 in → 25 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.8:low__json_schema PASS 100% 409 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.8:max PASS 100% 173 in → 290 out (67 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.8:max__json_schema PASS 100% 408 in → 244 out (57 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
anthropic/claude-opus-4.8:medium PASS 100% 169 in → 25 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 411 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-opus-4.8:none PASS 100% 174 in → 25 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.8:none__json_schema PASS 100% 408 in → 36 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 174 in → 91 out (31 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 408 in → 37 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
anthropic/claude-sonnet-4.6:high PASS 100% 127 in → 160 out (106 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 320 in → 29 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-sonnet-4.6:low PASS 100% 128 in → 23 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 320 in → 29 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-sonnet-4.6:max PASS 100% 129 in → 184 out (126 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 319 in → 129 out (87 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 133 in → 23 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 315 in → 29 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-sonnet-4.6:none PASS 100% 127 in → 23 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 315 in → 29 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 132 in → 166 out (99 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 316 in → 29 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-sonnet-5:high FAIL 100% 172 in → 29 out (0 reasoning)
``` WIP: add line1 WIP: add line2 WIP: add line3 ```
Failure: Extra selected commit messages: ['```', '```']
anthropic/claude-sonnet-5:high__json_schema PASS 100% 408 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-sonnet-5:low PASS 100% 168 in → 25 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
anthropic/claude-sonnet-5:low__json_schema PASS 100% 412 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-sonnet-5:medium PASS 100% 167 in → 40 out (25 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 414 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-sonnet-5:none__json_schema PASS 100% 411 in → 33 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 414 in → 53 out (18 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
arcee-ai/trinity-large-thinking:high PASS 100% 124 in → 884 out (900 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
arcee-ai/trinity-large-thinking:low PASS 100% 120 in → 1,489 out (1,542 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
arcee-ai/trinity-large-thinking:xhigh PASS 100% 121 in → 541 out (545 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
arcee-ai/trinity-mini:xhigh PASS 100% 121 in → 343 out (335 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-flash-0731:high PASS 100% 119 in → 471 out (395 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 125 in → 523 out (487 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
deepseek/deepseek-v4-flash-0731:max PASS 100% 210 in → 10,350 out (10,139 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 211 in → 472 out (445 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 121 in → 78 out (38 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 202 in → 190 out (169 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-flash:high PASS 100% 120 in → 263 out (241 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 122 in → 117 out (85 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"] }
deepseek/deepseek-v4-flash:low PASS 100% 119 in → 248 out (198 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 127 in → 144 out (114 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
deepseek/deepseek-v4-flash:medium PASS 100% 120 in → 147 out (125 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 121 in → 205 out (169 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
deepseek/deepseek-v4-flash:none PASS 100% 122 in → 20 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-flash:xhigh PASS 100% 200 in → 1,943 out (1,921 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 228 in → 30 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line1", "WIP: add line2", "WIP: add line3"]}
deepseek/deepseek-v4-pro:high PASS 100% 120 in → 273 out (251 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 326 in → 488 out (459 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits":["WIP: add line1","WIP: add line2","WIP: add line3"]}
deepseek/deepseek-v4-pro:low PASS 100% 119 in → 386 out (364 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 322 in → 310 out (281 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
deepseek/deepseek-v4-pro:medium PASS 100% 118 in → 212 out (190 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
deepseek/deepseek-v4-pro:none PASS 100% 122 in → 21 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 120 in → 38 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line1", "WIP: add line2", "WIP: add line3" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 200 in → 768 out (746 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 403 in → 1,433 out (1,402 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line1", "WIP: add line2", "WIP: add line3"]}
google/gemini-3-flash-preview:high PASS 100% 119 in → 321 out (301 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3-flash-preview:high__json_schema PASS 100% 203 in → 512 out (488 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3-flash-preview:low PASS 100% 121 in → 265 out (245 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3-flash-preview:low__json_schema PASS 100% 202 in → 27 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
google/gemini-3-flash-preview:medium PASS 100% 122 in → 346 out (326 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3-flash-preview:medium__json_schema PASS 100% 124 in → 476 out (452 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3-flash-preview:xhigh PASS 100% 120 in → 305 out (285 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 122 in → 834 out (810 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 124 in → 443 out (423 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 202 in → 837 out (795 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
google/gemini-3.1-flash-lite-preview:low PASS 100% 124 in → 145 out (125 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 202 in → 650 out (608 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 122 in → 299 out (279 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 126 in → 297 out (270 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 125 in → 646 out (626 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.1-pro-preview:high PASS 100% 123 in → 439 out (419 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 203 in → 558 out (534 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3.1-pro-preview:low PASS 100% 123 in → 306 out (286 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 203 in → 438 out (414 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3.1-pro-preview:medium PASS 100% 125 in → 325 out (305 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 201 in → 558 out (534 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 120 in → 449 out (429 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 203 in → 699 out (676 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3.5-flash-lite:high PASS 100% 122 in → 1,094 out (1,074 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 202 in → 808 out (784 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3.5-flash-lite:max PASS 100% 121 in → 1,352 out (1,332 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 122 in → 1,246 out (1,219 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line1", "WIP: add line2", "WIP: add line3"]}
google/gemini-3.5-flash-lite:medium PASS 100% 126 in → 536 out (516 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 201 in → 612 out (585 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 125 in → 1,305 out (1,285 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 203 in → 646 out (622 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3.5-flash:high PASS 100% 125 in → 512 out (492 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.5-flash:high__json_schema PASS 100% 200 in → 564 out (540 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3.5-flash:low PASS 100% 121 in → 290 out (270 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.5-flash:low__json_schema PASS 100% 202 in → 338 out (296 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
google/gemini-3.5-flash:medium PASS 100% 122 in → 507 out (487 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.5-flash:medium__json_schema PASS 100% 201 in → 453 out (429 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3.5-flash:xhigh PASS 100% 124 in → 543 out (523 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 198 in → 812 out (785 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
google/gemini-3.6-flash:high PASS 100% 123 in → 378 out (358 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.6-flash:high__json_schema PASS 100% 203 in → 1,394 out (1,370 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemini-3.6-flash:low PASS 100% 122 in → 180 out (160 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.6-flash:max PASS 100% 124 in → 507 out (487 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.6-flash:max__json_schema PASS 100% 124 in → 1,091 out (1,049 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
google/gemini-3.6-flash:medium PASS 100% 125 in → 572 out (552 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.6-flash:medium__json_schema PASS 100% 203 in → 627 out (585 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
google/gemini-3.6-flash:xhigh PASS 100% 124 in → 835 out (815 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 202 in → 1,041 out (1,017 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemma-4-26b-a4b-it:high PASS 100% 140 in → 467 out (391 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 140 in → 652 out (472 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 138 in → 331 out (242 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 141 in → 337 out (237 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 137 in → 724 out (703 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 139 in → 520 out (396 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
google/gemma-4-26b-a4b-it:none PASS 100% 134 in → 21 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 137 in → 38 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 136 in → 642 out (532 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 135 in → 353 out (232 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
google/gemma-4-31b-it:high PASS 100% 138 in → 293 out (222 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemma-4-31b-it:high__json_schema PASS 100% 137 in → 455 out (1 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
google/gemma-4-31b-it:low PASS 100% 137 in → 333 out (262 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemma-4-31b-it:low__json_schema PASS 100% 143 in → 287 out (261 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemma-4-31b-it:medium PASS 100% 140 in → 301 out (280 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemma-4-31b-it:medium__json_schema PASS 100% 140 in → 273 out (247 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
google/gemma-4-31b-it:none PASS 100% 135 in → 21 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemma-4-31b-it:none__json_schema PASS 100% 137 in → 28 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
google/gemma-4-31b-it:xhigh PASS 100% 137 in → 415 out (311 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 139 in → 306 out (221 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
ibm-granite/granite-4.1-8b PASS 100% 127 in → 21 out
WIP: add line1 WIP: add line2 WIP: add line3
ibm-granite/granite-4.1-8b__json_schema PASS 100% 130 in → 31 out
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) { "commits": ["WIP: add line1", "WIP: add line2", "WIP: add line3"] }
inclusionai/ling-2.6-flash__json_schema PASS 100% 148 in → 45 out
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line1", "WIP: add line2", "WIP: add line3" ] }
minimax/minimax-m2.5:high PASS 100% 153 in → 619 out (597 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m2.5:high__json_schema PASS 100% 151 in → 404 out (375 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
minimax/minimax-m2.5:low PASS 100% 153 in → 634 out (610 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m2.5:low__json_schema PASS 100% 152 in → 212 out (155 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
minimax/minimax-m2.5:medium PASS 100% 151 in → 388 out (374 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m2.5:medium__json_schema PASS 100% 154 in → 417 out (380 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
minimax/minimax-m2.5:xhigh PASS 100% 149 in → 200 out (178 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 155 in → 452 out (428 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
minimax/minimax-m2.7:high PASS 100% 151 in → 318 out (296 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m2.7:high__json_schema PASS 100% 304 in → 715 out (676 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
minimax/minimax-m2.7:low PASS 100% 155 in → 275 out (254 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m2.7:medium PASS 100% 153 in → 560 out (538 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m2.7:medium__json_schema PASS 100% 306 in → 802 out (774 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
minimax/minimax-m2.7:xhigh PASS 100% 149 in → 411 out (389 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m3:high PASS 100% 294 in → 137 out (106 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m3:low PASS 100% 292 in → 153 out (114 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m3:medium PASS 100% 294 in → 160 out (123 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m3:medium__json_schema PASS 100% 276 in → 38 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line1", "WIP: add line2", "WIP: add line3" ] }
minimax/minimax-m3:xhigh PASS 100% 288 in → 157 out (122 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
minimax/minimax-m3:xhigh__json_schema PASS 100% 278 in → 38 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
mistralai/devstral-2512__json_schema PASS 100% 132 in → 30 out
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
mistralai/mistral-medium-3-5:high PASS 100% 143 in → 756 out (639 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 135 in → 363 out (266 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
mistralai/mistral-medium-3-5:low PASS 100% 144 in → 1,065 out (963 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 128 in → 320 out (246 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
mistralai/mistral-medium-3-5:medium PASS 100% 143 in → 4,888 out (4,879 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 133 in → 790 out (705 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
mistralai/mistral-medium-3-5:none PASS 100% 143 in → 21 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 133 in → 30 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 142 in → 631 out (546 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 129 in → 416 out (332 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
moonshotai/kimi-k2.6:high PASS 100% 122 in → 1,089 out (1,067 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k2.6:high__json_schema PASS 100% 124 in → 714 out (685 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
moonshotai/kimi-k2.6:low PASS 100% 120 in → 899 out (877 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k2.6:low__json_schema PASS 100% 121 in → 667 out (520 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
moonshotai/kimi-k2.6:medium PASS 100% 126 in → 914 out (924 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k2.6:xhigh PASS 100% 123 in → 675 out (655 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 123 in → 219 out (192 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
moonshotai/kimi-k2.7-code:high PASS 100% 121 in → 119 out (97 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 125 in → 1,269 out (1,303 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) { "commits": ["WIP: add line1", "WIP: add line2", "WIP: add line3"] }
moonshotai/kimi-k2.7-code:low PASS 100% 122 in → 149 out (127 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 125 in → 158 out (131 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) { "commits": ["WIP: add line1", "WIP: add line2", "WIP: add line3"] }
moonshotai/kimi-k2.7-code:medium PASS 100% 125 in → 227 out (202 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k2.7-code:xhigh PASS 100% 128 in → 85 out (63 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 121 in → 213 out (175 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
moonshotai/kimi-k3:high PASS 100% 200 in → 248 out (213 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k3:high__json_schema PASS 100% 201 in → 123 out (88 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k3:low PASS 100% 200 in → 175 out (140 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k3:low__json_schema PASS 100% 197 in → 190 out (155 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k3:max PASS 100% 198 in → 1,132 out (1,097 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k3:max__json_schema PASS 100% 201 in → 230 out (195 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k3:medium PASS 100% 202 in → 144 out (109 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k3:medium__json_schema PASS 100% 202 in → 208 out (173 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k3:xhigh PASS 100% 200 in → 207 out (172 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 203 in → 191 out (156 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 143 in → 407 out (367 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 148 in → 523 out (475 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 143 in → 467 out (384 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 143 in → 329 out (269 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
nvidia/nemotron-3-nano-30b-a3b:none FAIL 100% 145 in → 24 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3 Complete feature
Failure: Extra selected commit messages: ['Complete feature']
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 144 in → 494 out (402 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 146 in → 227 out (161 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 145 in → 217 out (180 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 144 in → 257 out (207 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 147 in → 225 out (175 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 145 in → 290 out (231 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 146 in → 208 out (167 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 144 in → 185 out (136 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 145 in → 21 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 147 in → 38 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line1", "WIP: add line2", "WIP: add line3" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 146 in → 109 out (70 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 145 in → 228 out (179 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
openai/gpt-5.4-mini:high PASS 100% 126 in → 209 out (183 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4-mini:high__json_schema PASS 100% 167 in → 144 out (108 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4-mini:low PASS 100% 120 in → 76 out (50 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4-mini:low__json_schema PASS 100% 174 in → 58 out (22 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4-mini:medium PASS 100% 121 in → 172 out (146 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4-mini:medium__json_schema PASS 100% 170 in → 94 out (58 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4-mini:none PASS 100% 124 in → 30 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4-mini:none__json_schema PASS 100% 173 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4-mini:xhigh PASS 100% 119 in → 15,877 out (15,851 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 173 in → 284 out (248 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4-nano:high PASS 100% 123 in → 134 out (108 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4-nano:high__json_schema PASS 100% 170 in → 81 out (45 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4-nano:low PASS 100% 124 in → 84 out (58 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4-nano:low__json_schema PASS 100% 168 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4-nano:medium PASS 100% 121 in → 118 out (92 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4-nano:medium__json_schema PASS 100% 172 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4-nano:none__json_schema PASS 100% 169 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4-nano:xhigh PASS 100% 125 in → 269 out (243 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 170 in → 272 out (236 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4:high PASS 100% 123 in → 159 out (133 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4:high__json_schema PASS 100% 171 in → 89 out (53 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4:low PASS 100% 124 in → 44 out (18 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4:low__json_schema PASS 100% 167 in → 53 out (17 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4:medium PASS 100% 121 in → 70 out (44 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4:medium__json_schema PASS 100% 172 in → 123 out (87 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4:none PASS 100% 119 in → 24 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4:none__json_schema PASS 100% 170 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.4:xhigh PASS 100% 125 in → 173 out (147 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.4:xhigh__json_schema PASS 100% 171 in → 552 out (516 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.5:high PASS 100% 124 in → 121 out (95 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.5:high__json_schema PASS 100% 172 in → 129 out (93 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.5:low PASS 100% 125 in → 56 out (30 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.5:low__json_schema PASS 100% 171 in → 59 out (23 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.5:medium PASS 100% 128 in → 50 out (24 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.5:medium__json_schema PASS 100% 170 in → 74 out (38 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.5:none PASS 100% 121 in → 24 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.5:none__json_schema PASS 100% 170 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.5:xhigh PASS 100% 122 in → 210 out (184 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.5:xhigh__json_schema PASS 100% 170 in → 203 out (167 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-luna:high PASS 100% 124 in → 50 out (24 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-luna:high__json_schema PASS 100% 166 in → 314 out (278 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-luna:low PASS 100% 126 in → 24 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-luna:low__json_schema PASS 100% 170 in → 48 out (12 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-luna:max PASS 100% 123 in → 156 out (130 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-luna:max__json_schema PASS 100% 172 in → 106 out (70 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-luna:medium PASS 100% 120 in → 61 out (35 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-luna:medium__json_schema PASS 100% 173 in → 56 out (20 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-luna:none PASS 100% 120 in → 24 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-luna:none__json_schema PASS 100% 179 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-luna:xhigh PASS 100% 123 in → 85 out (59 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 171 in → 66 out (30 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-sol:high PASS 100% 120 in → 24 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-sol:high__json_schema PASS 100% 168 in → 34 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits":["WIP: add line1","WIP: add line2","WIP: add line3"]}
openai/gpt-5.6-sol:low PASS 100% 124 in → 24 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-sol:low__json_schema PASS 100% 171 in → 34 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits":["WIP: add line1","WIP: add line2","WIP: add line3"]}
openai/gpt-5.6-sol:max PASS 100% 119 in → 24 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-sol:max__json_schema PASS 100% 171 in → 186 out (150 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-sol:medium PASS 100% 121 in → 24 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-sol:medium__json_schema PASS 100% 170 in → 34 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits":["WIP: add line1","WIP: add line2","WIP: add line3"]}
openai/gpt-5.6-sol:none PASS 100% 122 in → 24 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-sol:none__json_schema PASS 100% 171 in → 34 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits":["WIP: add line1","WIP: add line2","WIP: add line3"]}
openai/gpt-5.6-sol:xhigh PASS 100% 127 in → 24 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 174 in → 105 out (69 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
JSON Schema Structured Output
(raw) {"commits":["WIP: add line1","WIP: add line2","WIP: add line3"]}
openai/gpt-5.6-terra:high PASS 100% 123 in → 40 out (14 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-terra:high__json_schema PASS 100% 169 in → 53 out (17 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-terra:low PASS 100% 121 in → 61 out (35 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-terra:low__json_schema PASS 100% 173 in → 53 out (17 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-terra:max PASS 100% 124 in → 106 out (80 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-terra:max__json_schema PASS 100% 170 in → 127 out (91 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-terra:medium PASS 100% 120 in → 46 out (20 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-terra:medium__json_schema PASS 100% 173 in → 52 out (16 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-terra:none PASS 100% 124 in → 24 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-terra:none__json_schema PASS 100% 171 in → 34 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-5.6-terra:xhigh PASS 100% 121 in → 43 out (17 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 168 in → 67 out (31 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-oss-120b:high PASS 100% 185 in → 525 out (506 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-oss-120b:low PASS 100% 187 in → 99 out (69 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-oss-120b:low__json_schema PASS 100% 171 in → 107 out (61 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
openai/gpt-oss-120b:medium PASS 100% 188 in → 207 out (187 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-oss-120b:medium__json_schema PASS 100% 179 in → 340 out (304 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-oss-120b:xhigh PASS 100% 180 in → 557 out (536 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
openai/gpt-oss-20b:high PASS 100% 188 in → 510 out (490 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
openai/gpt-oss-20b:high__json_schema PASS 100% 184 in → 300 out (247 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
openai/gpt-oss-20b:low PASS 100% 173 in → 80 out (42 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-oss-20b:low__json_schema PASS 100% 248 in → 88 out (49 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
openai/gpt-oss-20b:medium PASS 100% 181 in → 391 out (371 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
openai/gpt-oss-20b:medium__json_schema PASS 100% 184 in → 182 out (144 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
openai/gpt-oss-20b:xhigh PASS 100% 179 in → 508 out (484 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 188 in → 499 out (461 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":[ "WIP: add line3", "WIP: add line2", "WIP: add line1" ]}
poolside/laguna-m.1:high PASS 100% 137 in → 385 out (365 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-m.1:high__json_schema PASS 100% 132 in → 539 out (513 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
poolside/laguna-m.1:low PASS 100% 137 in → 414 out (394 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-m.1:low__json_schema PASS 100% 135 in → 251 out (213 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ]}
poolside/laguna-m.1:medium PASS 100% 138 in → 366 out (346 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-m.1:medium__json_schema PASS 100% 134 in → 454 out (415 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
poolside/laguna-m.1:none PASS 100% 134 in → 19 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-m.1:none__json_schema PASS 100% 137 in → 25 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
poolside/laguna-m.1:xhigh PASS 100% 135 in → 302 out (282 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-m.1:xhigh__json_schema PASS 100% 136 in → 471 out (445 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
poolside/laguna-xs-2.1:high PASS 100% 136 in → 316 out (298 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-xs-2.1:low PASS 100% 132 in → 376 out (358 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-xs-2.1:medium PASS 100% 138 in → 214 out (196 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-xs-2.1:none PASS 100% 132 in → 17 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-xs-2.1:xhigh PASS 100% 137 in → 250 out (232 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 133 in → 212 out (170 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
poolside/laguna-xs.2:high PASS 100% 174 in → 229 out (209 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-xs.2:high__json_schema PASS 100% 173 in → 270 out (243 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
poolside/laguna-xs.2:low PASS 100% 172 in → 235 out (215 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-xs.2:low__json_schema PASS 100% 173 in → 268 out (226 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
poolside/laguna-xs.2:medium PASS 100% 172 in → 219 out (199 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-xs.2:medium__json_schema PASS 100% 172 in → 224 out (198 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
poolside/laguna-xs.2:none PASS 100% 172 in → 19 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-xs.2:none__json_schema PASS 100% 176 in → 22 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
poolside/laguna-xs.2:xhigh PASS 100% 174 in → 201 out (181 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 172 in → 264 out (222 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
qwen/qwen3.6-27b:high PASS 100% 141 in → 1,173 out (1 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-27b:high__json_schema PASS 100% 140 in → 953 out (842 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
qwen/qwen3.6-27b:low PASS 100% 138 in → 825 out (717 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-27b:low__json_schema PASS 100% 139 in → 1,306 out (1,162 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
qwen/qwen3.6-27b:medium PASS 100% 137 in → 1,092 out (965 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-27b:medium__json_schema PASS 100% 136 in → 980 out (883 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
qwen/qwen3.6-27b:none PASS 100% 139 in → 21 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-27b:none__json_schema PASS 100% 139 in → 47 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3\n", "WIP: add line2\n", "WIP: add line1\n" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 139 in → 869 out (787 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-35b-a3b:high PASS 100% 138 in → 1,063 out (941 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 135 in → 1,049 out (921 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 139 in → 1,239 out (1,119 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 139 in → 1,081 out (936 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 137 in → 324 out (276 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 138 in → 957 out (804 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 135 in → 21 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 140 in → 29 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 139 in → 1,088 out (955 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 137 in → 1,072 out (961 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"] }
qwen/qwen3.6-flash:high PASS 100% 141 in → 1,513 out (1,488 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-flash:low PASS 100% 140 in → 352 out (327 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-flash:medium PASS 100% 139 in → 1,099 out (1,074 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-flash:none PASS 100% 140 in → 20 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.6-flash:xhigh PASS 100% 140 in → 1,735 out (1,710 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
qwen/qwen3.7-flash:high PASS 100% 136 in → 599 out (574 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-flash:low PASS 100% 137 in → 789 out (764 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-flash:max PASS 100% 139 in → 1,002 out (977 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-flash:medium PASS 100% 135 in → 509 out (484 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-flash:none PASS 100% 137 in → 20 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-flash:xhigh PASS 100% 140 in → 687 out (662 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-max:high PASS 100% 140 in → 240 out (215 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-max:low PASS 100% 137 in → 250 out (225 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-max:medium PASS 100% 139 in → 244 out (219 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-max:none PASS 100% 136 in → 20 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-max:xhigh PASS 100% 133 in → 412 out (387 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-plus:high PASS 100% 139 in → 1,152 out (1,127 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-plus:low PASS 100% 136 in → 730 out (705 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-plus:medium PASS 100% 138 in → 876 out (851 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-plus:none PASS 100% 141 in → 20 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
qwen/qwen3.7-plus:xhigh PASS 100% 137 in → 1,091 out (1,066 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
tencent/hy3:high PASS 100% 128 in → 442 out (420 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
tencent/hy3:high__json_schema PASS 100% 128 in → 612 out (533 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
tencent/hy3:low PASS 100% 126 in → 315 out (293 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
tencent/hy3:low__json_schema PASS 100% 130 in → 182 out (126 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
tencent/hy3:medium PASS 100% 127 in → 243 out (195 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
tencent/hy3:medium__json_schema PASS 100% 130 in → 254 out (192 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
tencent/hy3:none PASS 100% 134 in → 21 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
tencent/hy3:none__json_schema PASS 100% 135 in → 31 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"] }
tencent/hy3:xhigh PASS 100% 127 in → 457 out (435 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
tencent/hy3:xhigh__json_schema PASS 100% 127 in → 607 out (526 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
thinkingmachines/inkling-small:high PASS 100% 132 in → 507 out (481 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3
thinkingmachines/inkling-small:high__json_schema PASS 100% 131 in → 708 out (674 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
thinkingmachines/inkling-small:low PASS 100% 132 in → 150 out (109 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
thinkingmachines/inkling-small:low__json_schema PASS 100% 130 in → 78 out (44 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
thinkingmachines/inkling-small:max PASS 100% 133 in → 336 out (310 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
thinkingmachines/inkling-small:max__json_schema PASS 100% 137 in → 472 out (435 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
thinkingmachines/inkling-small:medium PASS 100% 129 in → 103 out (59 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
thinkingmachines/inkling-small:medium__json_schema PASS 100% 133 in → 262 out (220 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 139 in → 385 out (359 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 138 in → 270 out (228 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
thinkingmachines/inkling:high PASS 100% 133 in → 234 out (207 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
thinkingmachines/inkling:low PASS 100% 135 in → 64 out (38 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
thinkingmachines/inkling:low__json_schema PASS 100% 133 in → 63 out (32 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
thinkingmachines/inkling:max PASS 100% 137 in → 193 out (155 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
thinkingmachines/inkling:max__json_schema PASS 100% 136 in → 292 out (261 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
thinkingmachines/inkling:medium PASS 100% 131 in → 121 out (95 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
thinkingmachines/inkling:medium__json_schema PASS 100% 130 in → 148 out (106 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
thinkingmachines/inkling:xhigh PASS 100% 132 in → 230 out (203 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.3:high PASS 100% 296 in → 482 out (462 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.3:high__json_schema PASS 100% 363 in → 543 out (519 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
x-ai/grok-4.3:low PASS 100% 303 in → 309 out (289 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.3:low__json_schema PASS 100% 366 in → 447 out (411 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
x-ai/grok-4.3:max PASS 100% 296 in → 223 out (203 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.3:max__json_schema PASS 100% 360 in → 1,012 out (976 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
x-ai/grok-4.3:medium PASS 100% 303 in → 442 out (422 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.3:medium__json_schema PASS 100% 367 in → 509 out (473 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
x-ai/grok-4.3:xhigh PASS 100% 297 in → 426 out (406 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.3:xhigh__json_schema PASS 100% 363 in → 648 out (618 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"] }
x-ai/grok-4.5:high PASS 100% 316 in → 248 out (231 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.5:high__json_schema PASS 100% 398 in → 629 out (607 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["WIP: add line3","WIP: add line2","WIP: add line1"]}
x-ai/grok-4.5:low PASS 100% 316 in → 213 out (196 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.5:low__json_schema PASS 100% 400 in → 316 out (281 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
x-ai/grok-4.5:max PASS 100% 320 in → 283 out (266 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.5:max__json_schema PASS 100% 398 in → 370 out (335 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
x-ai/grok-4.5:medium PASS 100% 319 in → 293 out (276 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.5:medium__json_schema PASS 100% 398 in → 358 out (323 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
x-ai/grok-4.5:xhigh PASS 100% 319 in → 343 out (326 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
x-ai/grok-4.5:xhigh__json_schema PASS 100% 395 in → 332 out (297 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
z-ai/glm-4.7-flash:high PASS 100% 126 in → 528 out (441 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
z-ai/glm-4.7-flash:low PASS 100% 122 in → 544 out (448 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
z-ai/glm-4.7-flash:medium PASS 100% 126 in → 469 out (379 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
z-ai/glm-4.7-flash:none PASS 100% 122 in → 21 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
z-ai/glm-4.7-flash:none__json_schema PASS 100% 122 in → 29 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
z-ai/glm-4.7-flash:xhigh PASS 100% 124 in → 726 out (622 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
z-ai/glm-5.2:low PASS 100% 131 in → 343 out (321 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
z-ai/glm-5.2:low__json_schema PASS 100% 125 in → 358 out (280 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
z-ai/glm-5.2:medium PASS 100% 132 in → 1,274 out (1,252 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
z-ai/glm-5.2:medium__json_schema PASS 100% 123 in → 609 out (485 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
z-ai/glm-5.2:none PASS 100% 128 in → 21 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
z-ai/glm-5.2:none__json_schema PASS 100% 124 in → 30 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["WIP: add line3", "WIP: add line2", "WIP: add line1"]}
z-ai/glm-5.2:xhigh PASS 100% 132 in → 538 out (517 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
z-ai/glm-5.2:xhigh__json_schema PASS 100% 131 in → 572 out (497 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
arcee-ai/trinity-mini:low FAIL 66.7% 123 in → 317 out (276 reasoning)
WIP:add line3 WIP: add line2 WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line3']; Extra selected commit messages: ['WIP:add line3']
arcee-ai/trinity-mini:medium FAIL 66.7% 118 in → 506 out (526 reasoning)
WIP: addline3 WIP: add line2 WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line3']; Extra selected commit messages: ['WIP: addline3']
thinkingmachines/inkling-small:none FAIL 66.7% 134 in → 23 out (0 reasoning)
WIP: add line1 WIP: add line2 WIP: add line3<|end_message|>
Failure: Missing expected commit messages: ['WIP: add line3']; Extra selected commit messages: ['WIP: add line3']
thinkingmachines/inkling:none FAIL 66.7% 133 in → 23 out (0 reasoning)
WIP: add line3 WIP: add line2 WIP: add line1<|end_message|>
Failure: Missing expected commit messages: ['WIP: add line1']; Extra selected commit messages: ['WIP: add line1']
arcee-ai/trinity-mini:high FAIL 33.3% 121 in → 374 out (366 reasoning)
WIP: add line3WIP: add line2 WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3']
anthropic/claude-haiku-4.5:none FAIL 0% 134 in → 42 out (0 reasoning)
9e2096f WIP: add line3 55b6d93 WIP: add line2 d3a5a85 WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 312 in → 22 out (0 reasoning)
548488a cdff371 dd636bd
JSON Schema Structured Output
(raw) {"commits": ["548488a", "cdff371", "dd636bd"]}
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
anthropic/claude-opus-4.8:high FAIL 0% 174 in → 43 out (0 reasoning)
eab1267 WIP: add line1 ab501c0 WIP: add line2 dc4ea7b WIP: add line3
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']
anthropic/claude-sonnet-5:none FAIL 0% 170 in → 41 out (0 reasoning)
ddc05a7 WIP: add line3 329526f WIP: add line2 f94aa36 WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
anthropic/claude-sonnet-5:xhigh FAIL 0% 172 in → 46 out (0 reasoning)
``` b2a4880 WIP: add line3 f98d8bd WIP: add line2 a59eeec WIP: add line1 ```
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['```', 'WIP: add line3', 'WIP: add line2', 'WIP: add line1', '```']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium FAIL 0% 118 in → 566 out (560 reasoning)
bee07cc WIP: add line3 e05c850 WIP: add line2 9269bad WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high__json_schema FAIL 0% 119 in → 466 out (418 reasoning)
d465614 Complete feature 69f97dd WIP: add line3 b90afd9 WIP: add line2 0abf993 WIP: add line1 b8a272b Initial feature
JSON Schema Structured Output
(raw) { "commits": [ "d465614 Complete feature", "69f97dd WIP: add line3", "b90afd9 WIP: add line2", "0abf993 WIP: add line1", "b8a272b Initial feature" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['Complete feature', 'WIP: add line3', 'WIP: add line2', 'WIP: add line1', 'Initial feature']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 120 in → 548 out (506 reasoning)
be20663 Complete feature 507870e WIP: add line3 9f79801 WIP: add line2 11891d5 WIP: add line1 a60a7e9 Initial feature
JSON Schema Structured Output
(raw) { "commits": [ "be20663 Complete feature", "507870e WIP: add line3", "9f79801 WIP: add line2", "11891d5 WIP: add line1", "a60a7e9 Initial feature" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['Complete feature', 'WIP: add line3', 'WIP: add line2', 'WIP: add line1', 'Initial feature']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 122 in → 590 out (510 reasoning)
8d7ba9e Complete feature f56f36b WIP: add line3 d486624 WIP: add line2 b65033d WIP: add line1 51d0388 Initial feature
JSON Schema Structured Output
(raw) { "commits": [ "8d7ba9e Complete feature", "f56f36b WIP: add line3", "d486624 WIP: add line2", "b65033d WIP: add line1", "51d0388 Initial feature" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['Complete feature', 'WIP: add line3', 'WIP: add line2', 'WIP: add line1', 'Initial feature']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 120 in → 409 out (377 reasoning)
4bc567e WIP: add line3 c887eb1 WIP: add line2 c5b78fa WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "4bc567e WIP: add line3", "c887eb1 WIP: add line2", "c5b78fa WIP: add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
deepseek/deepseek-v4-flash-0731:high__json_schema FAIL 0% 204 in → 135 out (120 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "commits": [] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']
deepseek/deepseek-v4-flash-0731:low FAIL 0% 120 in → 77 out (44 reasoning)
1815d13 WIP: add line3 1cf2fff WIP: add line2 e4fe416 WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
deepseek/deepseek-v4-flash-0731:medium FAIL 0% 129 in → 176 out (128 reasoning)
2141596 WIP: add line3 e3c9cd7 WIP: add line2 e5826aa WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 118 in → 30 out (0 reasoning)
f317933 WIP: add line3 8040410 WIP: add line2 3c80881 WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 123 in → 44 out (0 reasoning)
c700a4a WIP: add line3 a9ae6d1 WIP: add line2 fe09640 WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["c700a4a WIP: add line3","a9ae6d1 WIP: add line2","fe09640 WIP: add line1"]}
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 119 in → 97 out (56 reasoning)
52e5891 WIP: add line3 fe2db3f WIP: add line2 d689338 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": ["52e5891 WIP: add line3", "fe2db3f WIP: add line2", "d689338 WIP: add line1"] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 125 in → 51 out (0 reasoning)
1a0d0e9 WIP: add line3 d87b9b9 WIP: add line2 b4aab87 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "1a0d0e9 WIP: add line3", "d87b9b9 WIP: add line2", "b4aab87 WIP: add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 117 in → 48 out (0 reasoning)
bc40512 WIP: add line1 0553860 WIP: add line2 10a6045 WIP: add line3
JSON Schema Structured Output
(raw) { "commits": [ "bc40512 WIP: add line1", "0553860 WIP: add line2", "10a6045 WIP: add line3" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']
google/gemini-3.1-flash-lite-preview:xhigh__json_schema FAIL 0% 124 in → 1,124 out (1,082 reasoning)
16b3444 WIP: add line3 a1efe56 WIP: add line2 410de93 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["16b3444 WIP: add line3", "a1efe56 WIP: add line2", "410de93 WIP: add line1"]}
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
google/gemini-3.5-flash-lite:low FAIL 0% 124 in → 36 out (0 reasoning)
e6b4f22 WIP: add line3 73c64db WIP: add line2 298bbd2 WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 124 in → 58 out (0 reasoning)
c249a94 WIP: add line3 719b39e WIP: add line2 fd9fe44 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "c249a94 WIP: add line3", "719b39e WIP: add line2", "fd9fe44 WIP: add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
google/gemini-3.6-flash:low__json_schema FAIL 0% 201 in → 59 out (0 reasoning)
2f145d0 WIP: add line3 4bd2f4c WIP: add line2 d47c88a WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "2f145d0 WIP: add line3", "4bd2f4c WIP: add line2", "d47c88a WIP: add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
inclusionai/ling-2.6-flash FAIL 0% 146 in → 38 out
c388a1f WIP: add line3 c28fbc2 WIP: add line2 ec92cca WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
liquid/lfm-2-24b-a2b FAIL 0% 132 in → 17 out
f5e5b7d 3570948 ab89c16
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: 8db63f7 WIP: add line3 2c78e46 WIP: add line2 7d8aced WIP: add line1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
minimax/minimax-m2.7:low__json_schema FAIL 0% 155 in → 71 out (71 reasoning)
77c2822 Complete feature 4d3d957 WIP: add line3 cde709d WIP: add line2 d9e864d WIP: add line1 08e8ca8 Initial feature
JSON Schema Structured Output
(raw) { "commits": [ "77c2822 Complete feature", "4d3d957 WIP: add line3", "cde709d WIP: add line2", "d9e864d WIP: add line1", "08e8ca8 Initial feature" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['Complete feature', 'WIP: add line3', 'WIP: add line2', 'WIP: add line1', 'Initial feature']
minimax/minimax-m2.7:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high__json_schema FAIL 0% 239 in → 46 out (45 reasoning)
783be12 WIP: add line3 e0c5093 WIP: add line2 7c9592c WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "783be12 WIP: add line3", "e0c5093 WIP: add line2", "7c9592c WIP: add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
minimax/minimax-m3:low__json_schema FAIL 0% 277 in → 51 out (0 reasoning)
c9e42d0 WIP: add line3 9eff59a WIP: add line2 26487ae WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "c9e42d0 WIP: add line3", "9eff59a WIP: add line2", "26487ae WIP: add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
mistralai/devstral-2512 FAIL 0% 132 in → 39 out
6c927c4 WIP: add line3 7f7ba0b WIP: add line2 bbbb60b WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
moonshotai/kimi-k2.6:medium__json_schema FAIL 0% 129 in → 75 out (0 reasoning)
7a93f50 Complete feature 446ab2d WIP: add line3 b6e0a5e WIP: add line2 f0ce6e9 WIP: add line1 0cd5c37 Initial feature
JSON Schema Structured Output
(raw) { "commits": [ "7a93f50 Complete feature", "446ab2d WIP: add line3", "b6e0a5e WIP: add line2", "f0ce6e9 WIP: add line1", "0cd5c37 Initial feature" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['Complete feature', 'WIP: add line3', 'WIP: add line2', 'WIP: add line1', 'Initial feature']
moonshotai/kimi-k2.7-code:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 0% 146 in → 513 out (472 reasoning)
add line3 add line2 add line1
JSON Schema Structured Output
(raw) { "commits": [ "add line3", "add line2", "add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['add line3', 'add line2', 'add line1']
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 144 in → 524 out (515 reasoning)
add line3 add line2 add line1
JSON Schema Structured Output
(raw) { "commits": [ "add line3", "add line2", "add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['add line3', 'add line2', 'add line1']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 145 in → 38 out (0 reasoning)
Fixup! Complete feature Squash! Complete feature
JSON Schema Structured Output
(raw) { "commits": [ "Fixup! Complete feature", "Squash! Complete feature" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['Complete feature', 'Complete feature']
openai/gpt-5.4-nano:none FAIL 0% 121 in → 37 out (0 reasoning)
f650059 WIP: add line3 c997b97 WIP: add line2 f4a7de0 WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
openai/gpt-oss-120b:high__json_schema FAIL 0% 186 in → 420 out (413 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"commits":[ ]}
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']
openai/gpt-oss-120b:xhigh__json_schema FAIL 0% 184 in → 34 out (0 reasoning)
47d9898 110f315 0ae1d55 e9f7878 c7eb3cf
JSON Schema Structured Output
(raw) {"commits":["47d9898","110f315","0ae1d55","e9f7878","c7eb3cf"]}
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['Complete feature', 'WIP: add line3', 'WIP: add line2', 'WIP: add line1', 'Initial feature']
poolside/laguna-xs-2.1:high__json_schema FAIL 0% 136 in → 326 out (280 reasoning)
c5d7691 WIP: add line3 d0a78d5 WIP: add line2 91cb267 WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["c5d7691 WIP: add line3", "d0a78d5 WIP: add line2", "91cb267 WIP: add line1"]}
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
poolside/laguna-xs-2.1:low__json_schema FAIL 0% 136 in → 248 out (187 reasoning)
b6401ad WIP: add line3 e6f307c WIP: add line2 8b8f3a3 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "b6401ad WIP: add line3", "e6f307c WIP: add line2", "8b8f3a3 WIP: add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
poolside/laguna-xs-2.1:medium__json_schema FAIL 0% 135 in → 320 out (259 reasoning)
b43f40c WIP: add line3 ff2f24c WIP: add line2 d249aa5 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "b43f40c WIP: add line3", "ff2f24c WIP: add line2", "d249aa5 WIP: add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 136 in → 62 out (0 reasoning)
2b42a03 WIP: add line3 b84184c WIP: add line2 21d4b35 WIP: add line1
JSON Schema Structured Output
(raw) { "commits": [ "2b42a03 WIP: add line3", "b84184c WIP: add line2", "21d4b35 WIP: add line1" ] }
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
qwen/qwen3.6-27b:xhigh__json_schema FAIL 0%
Invalid JSON. Output: WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7b947952-5993-9b4b-80f1-531f2ca5f7e9","request_id":"7b947952-5993-9b4b-80f1-531f2ca5f7e9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9beadf37-9c45-954d-aeff-420db7ec8223","request_id":"9beadf37-9c45-954d-aeff-420db7ec8223"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-090fb935-7f7f-902c-9051-6d3c1158e8b7","request_id":"090fb935-7f7f-902c-9051-6d3c1158e8b7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e39d0b12-f10e-9c0e-83ec-c7dd28d3a425","request_id":"e39d0b12-f10e-9c0e-83ec-c7dd28d3a425"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-abd7f2ca-59b8-99e3-814f-a580117ed3d2","request_id":"abd7f2ca-59b8-99e3-814f-a580117ed3d2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e267d136-1fb8-9775-85a5-44daebbcbf08"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3bf225f9-a795-918e-9725-7c6463ac4001"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-18c3ddac-6936-9170-89a9-a4af937fea4f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a8fc9d5b-e6cc-9b53-a34a-4977f4e16dd0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3a142c3a-1c5c-9bbc-979a-70a26ed7e41a"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ee91e2b9-304d-9b18-bab0-f90e569bb5ae"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b6b76285-6427-9476-8cea-217609145ba4","request_id":"b6b76285-6427-9476-8cea-217609145ba4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d324aafc-5682-902b-a3de-02a7e1d52ec3","request_id":"d324aafc-5682-902b-a3de-02a7e1d52ec3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7507ccc4-9f74-94af-83f1-fffb94de0169","request_id":"7507ccc4-9f74-94af-83f1-fffb94de0169"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-045d9d83-665a-9b50-9dc3-10b1354f76f9","request_id":"045d9d83-665a-9b50-9dc3-10b1354f76f9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a005322f-37e1-9242-b61a-bbf68a1188c2","request_id":"a005322f-37e1-9242-b61a-bbf68a1188c2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ff04d0e5-f56f-95c1-8bb7-d5af150ebe03","request_id":"ff04d0e5-f56f-95c1-8bb7-d5af150ebe03"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d5055cf6-6381-961a-af0b-eaf2595dd960","request_id":"d5055cf6-6381-961a-af0b-eaf2595dd960"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c0a405ea-c9d4-9ead-9374-8724e8bdce71","request_id":"c0a405ea-c9d4-9ead-9374-8724e8bdce71"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0f7459cc-cce0-9fff-9ac0-fa23ede3af72","request_id":"0f7459cc-cce0-9fff-9ac0-fa23ede3af72"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5fb35bba-eadf-9383-8aae-688b28c17f8a","request_id":"5fb35bba-eadf-9383-8aae-688b28c17f8a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: WIP: add line1 WIP: add line2 WIP: add line3{ "commits": [ "WIP: add line1", "WIP: add line2", "WIP: add line3" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: WIP: add line3 WIP: add line2 WIP: add line1{ "commits": [ "WIP: add line3", "WIP: add line2", "WIP: add line1" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:xhigh__json_schema FAIL 0%
Invalid JSON. Output: WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none FAIL 0% 293 in → 29 out (0 reasoning)
8885ae6 WIP: add line3 07b993b WIP: add line2 ed2794c WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
x-ai/grok-4.3:none__json_schema FAIL 0% 365 in → 35 out (0 reasoning)
30e4ef4 WIP: add line3 f26e22c WIP: add line2 57da5de WIP: add line1
JSON Schema Structured Output
(raw) {"commits":["30e4ef4 WIP: add line3","f26e22c WIP: add line2","57da5de WIP: add line1"]}
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output: WIP: add line3 WIP: add line2 WIP: add line1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:high FAIL 0% 132 in → 381 out (283 reasoning)
e78e8d8 WIP: add line3 a0bf95f WIP: add line2 965be21 WIP: add line1
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']
z-ai/glm-5.2:high__json_schema FAIL 0% 132 in → 405 out (361 reasoning)
86a0902 WIP: add line3 2646962 WIP: add line2 e92e31f WIP: add line1
JSON Schema Structured Output
(raw) {"commits": ["86a0902 WIP: add line3", "2646962 WIP: add line2", "e92e31f WIP: add line1"]}
Failure: Missing expected commit messages: ['WIP: add line1', 'WIP: add line2', 'WIP: add line3']; Extra selected commit messages: ['WIP: add line3', 'WIP: add line2', 'WIP: add line1']