f008 — commit_squash

← commit_squash
Config changes spread across commits
Tests ability to squash config changes spread across multiple commits. Evaluates recognizing scattered configuration updates.
medium commit-squash config scattered-changes squash
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo '{}' > config.json
  5. 05 git add config.json
  6. 06 git commit -m 'Initial config'
  7. 07 echo '{"debug": true}' > config.json
  8. 08 git add config.json
  9. 09 git commit -m 'WIP: add debug flag'
  10. 10 echo '{"debug": true, "port": 8080}' > config.json
  11. 11 git add config.json
  12. 12 git commit -m 'WIP: add port'
  13. 13 echo '{"debug": true, "port": 8080, "host": "localhost"}' > config.json
  14. 14 git add config.json
  15. 15 git commit -m 'Finalize config'
Prompt
These commits show incremental configuration changes. Identify the WIP commits that should be marked as squash/fixup to combine all config work into a single meaningful commit. Do not include the initial or final config anchor commit. Output ONLY the selected commit subject lines, one per line, with no explanation.
Expected
WIP: add debug flag WIP: add port
Fixture Metrics

Scoped model quality, cost, API time, and token usage for commit_squash/f008.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 156 in → 38 out (15 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-fable-5:high__json_schema PASS 100% 396 in → 60 out (24 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
anthropic/claude-fable-5:low PASS 100% 162 in → 18 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-fable-5:low__json_schema PASS 100% 397 in → 29 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
anthropic/claude-fable-5:max PASS 100% 160 in → 402 out (70 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-fable-5:max__json_schema PASS 100% 400 in → 260 out (60 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
anthropic/claude-fable-5:medium PASS 100% 158 in → 30 out (10 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-fable-5:medium__json_schema PASS 100% 395 in → 29 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
anthropic/claude-fable-5:xhigh PASS 100% 161 in → 95 out (35 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 397 in → 60 out (24 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
anthropic/claude-haiku-4.5:high PASS 100% 148 in → 333 out (283 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 338 in → 1,033 out (532 reasoning)
WIP: add debug flag WIP: add port
JSON Schema Structured Output
(raw) {"commits": ["WIP: add debug flag", "WIP: add port"]}
anthropic/claude-haiku-4.5:low PASS 100% 148 in → 345 out (292 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 331 in → 1,072 out (652 reasoning)
WIP: add debug flag WIP: add port
JSON Schema Structured Output
(raw) {"commits": ["WIP: add debug flag", "WIP: add port"]}
anthropic/claude-haiku-4.5:medium PASS 100% 147 in → 303 out (286 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 335 in → 627 out (531 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
anthropic/claude-haiku-4.5:none PASS 100% 120 in → 15 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-haiku-4.5:xhigh PASS 100% 146 in → 1,212 out (714 reasoning)
WIP: add debug flag WIP: add port
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 337 in → 4,752 out (1,558 reasoning)
WIP: add debug flag WIP: add port
JSON Schema Structured Output
(raw) {"commits": ["WIP: add debug flag", "WIP: add port"]}
anthropic/claude-opus-4.6:high PASS 100% 120 in → 96 out (59 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.6:high__json_schema PASS 100% 306 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.6:low PASS 100% 119 in → 15 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.6:low__json_schema PASS 100% 306 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.6:max PASS 100% 122 in → 180 out (129 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.6:max__json_schema PASS 100% 303 in → 192 out (143 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.6:medium PASS 100% 119 in → 15 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 305 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.6:none PASS 100% 117 in → 15 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.6:none__json_schema PASS 100% 304 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 122 in → 85 out (55 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 308 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.7:high PASS 100% 169 in → 20 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.7:high__json_schema PASS 100% 401 in → 29 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.7:low PASS 100% 163 in → 20 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.7:low__json_schema PASS 100% 407 in → 29 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.7:max PASS 100% 166 in → 181 out (54 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.7:max__json_schema PASS 100% 406 in → 29 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.7:medium PASS 100% 165 in → 20 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 401 in → 29 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.7:none PASS 100% 166 in → 20 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.7:none__json_schema PASS 100% 406 in → 29 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 163 in → 20 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 408 in → 29 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.8:high PASS 100% 158 in → 85 out (52 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.8:high__json_schema PASS 100% 400 in → 27 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.8:low PASS 100% 158 in → 49 out (29 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.8:low__json_schema PASS 100% 398 in → 27 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.8:max PASS 100% 160 in → 422 out (76 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.8:max__json_schema PASS 100% 403 in → 260 out (73 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
anthropic/claude-opus-4.8:medium PASS 100% 161 in → 60 out (38 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 398 in → 27 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.8:none PASS 100% 158 in → 18 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.8:none__json_schema PASS 100% 398 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 159 in → 141 out (83 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 394 in → 27 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-sonnet-4.6:high PASS 100% 118 in → 118 out (86 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 306 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-sonnet-4.6:low PASS 100% 117 in → 15 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 308 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-sonnet-4.6:max PASS 100% 119 in → 185 out (162 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 307 in → 187 out (136 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 118 in → 15 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 306 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-sonnet-4.6:none PASS 100% 117 in → 15 out (0 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 306 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 118 in → 104 out (69 reasoning)
WIP: add port WIP: add debug flag
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 307 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
anthropic/claude-sonnet-5:high PASS 100% 161 in → 18 out (0 reasoning)
WIP: add debug flag WIP: add port
anthropic/claude-sonnet-5:medium PASS 100% 158 in → 18 out (0 reasoning)
WIP: add debug flag WIP: add port
arcee-ai/trinity-large-thinking:high PASS 100% 106 in → 1,035 out (1,083 reasoning)
WIP: add port WIP: add debug flag
arcee-ai/trinity-large-thinking:low PASS 100% 112 in → 862 out (891 reasoning)
WIP: add port WIP: add debug flag
arcee-ai/trinity-large-thinking:medium PASS 100% 109 in → 545 out (560 reasoning)
WIP: add port WIP: add debug flag
arcee-ai/trinity-large-thinking:xhigh PASS 100% 108 in → 467 out (475 reasoning)
WIP: add port WIP: add debug flag
arcee-ai/trinity-mini:medium PASS 100% 113 in → 263 out (271 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-flash-0731:high PASS 100% 109 in → 134 out (109 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 188 in → 13,089 out (13,499 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 110 in → 455 out (425 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 109 in → 59 out (31 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 190 in → 172 out (159 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 281 in → 230 out (209 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 194 in → 138 out (122 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 110 in → 52 out (35 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": ["WIP: add port", "WIP: add debug flag"] }
deepseek/deepseek-v4-flash:high PASS 100% 108 in → 213 out (206 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 111 in → 179 out (157 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
deepseek/deepseek-v4-flash:low PASS 100% 109 in → 132 out (122 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 112 in → 392 out (370 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
deepseek/deepseek-v4-flash:medium PASS 100% 105 in → 109 out (89 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 215 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
deepseek/deepseek-v4-flash:none PASS 100% 107 in → 13 out (0 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 110 in → 21 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": ["WIP: add port", "WIP: add debug flag"] }
deepseek/deepseek-v4-flash:xhigh PASS 100% 187 in → 543 out (589 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-pro:high PASS 100% 109 in → 263 out (249 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-pro:high__json_schema FAIL 100% 109 in → 308 out (262 reasoning)
WIP: add port WIP: add debug flag exclude_initial_and_final_anchors_for_squash_range
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag", "exclude_initial_and_final_anchors_for_squash_range" ] }
Failure: Extra selected commit messages: ['exclude_initial_and_final_anchors_for_squash_range']
deepseek/deepseek-v4-pro:low PASS 100% 108 in → 207 out (193 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-pro:medium PASS 100% 107 in → 204 out (190 reasoning)
WIP: add port WIP: add debug flag
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 312 in → 295 out (273 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
google/gemini-3-flash-preview:high PASS 100% 110 in → 423 out (411 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3-flash-preview:high__json_schema PASS 100% 187 in → 467 out (451 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3-flash-preview:low PASS 100% 109 in → 351 out (339 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3-flash-preview:medium PASS 100% 112 in → 368 out (356 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3-flash-preview:medium__json_schema PASS 100% 190 in → 464 out (448 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3-flash-preview:xhigh PASS 100% 111 in → 294 out (283 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 190 in → 995 out (979 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 111 in → 979 out (967 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 189 in → 408 out (390 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 112 in → 200 out (188 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 114 in → 149 out (118 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 112 in → 316 out (304 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 108 in → 239 out (221 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 110 in → 1,194 out (1,182 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 189 in → 316 out (289 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
google/gemini-3.1-pro-preview:high PASS 100% 112 in → 373 out (361 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 192 in → 551 out (535 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.1-pro-preview:low PASS 100% 110 in → 229 out (217 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 192 in → 343 out (327 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.1-pro-preview:medium PASS 100% 108 in → 435 out (423 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 188 in → 481 out (465 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 111 in → 465 out (453 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 187 in → 523 out (507 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.5-flash-lite:high PASS 100% 110 in → 1,028 out (1,016 reasoning)
WIP: add debug flag WIP: add port
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 189 in → 703 out (687 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 111 in → 31 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
google/gemini-3.5-flash-lite:max PASS 100% 112 in → 817 out (805 reasoning)
WIP: add debug flag WIP: add port
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 109 in → 765 out (749 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.5-flash-lite:medium PASS 100% 111 in → 764 out (752 reasoning)
WIP: add debug flag WIP: add port
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 110 in → 583 out (565 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 108 in → 852 out (840 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 189 in → 782 out (766 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.5-flash:high PASS 100% 110 in → 442 out (430 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.5-flash:high__json_schema PASS 100% 187 in → 628 out (612 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.5-flash:low PASS 100% 112 in → 403 out (391 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.5-flash:low__json_schema PASS 100% 189 in → 172 out (141 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
google/gemini-3.5-flash:medium PASS 100% 110 in → 613 out (601 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.5-flash:medium__json_schema PASS 100% 191 in → 567 out (549 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
google/gemini-3.5-flash:xhigh PASS 100% 112 in → 512 out (500 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 192 in → 604 out (588 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.6-flash:high PASS 100% 110 in → 855 out (843 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.6-flash:high__json_schema PASS 100% 109 in → 1,018 out (1,002 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.6-flash:low PASS 100% 111 in → 380 out (368 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.6-flash:max PASS 100% 110 in → 709 out (697 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.6-flash:max__json_schema PASS 100% 111 in → 676 out (660 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.6-flash:medium PASS 100% 111 in → 794 out (783 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.6-flash:medium__json_schema PASS 100% 190 in → 595 out (579 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemini-3.6-flash:xhigh PASS 100% 113 in → 607 out (595 reasoning)
WIP: add port WIP: add debug flag
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 106 in → 1,108 out (1,092 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemma-4-26b-a4b-it:high PASS 100% 127 in → 440 out (343 reasoning)
WIP: add port WIP: add debug flag
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 123 in → 384 out (307 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
google/gemma-4-26b-a4b-it:low PASS 100% 125 in → 320 out (280 reasoning)
WIP: add port WIP: add debug flag
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 125 in → 409 out (314 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 126 in → 492 out (420 reasoning)
WIP: add port WIP: add debug flag
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 127 in → 290 out (217 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 123 in → 13 out (0 reasoning)
WIP: add port WIP: add debug flag
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 122 in → 28 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 127 in → 829 out (816 reasoning)
WIP: add port WIP: add debug flag
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 128 in → 299 out (230 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": ["WIP: add port", "WIP: add debug flag"] }
google/gemma-4-31b-it:high PASS 100% 129 in → 340 out (253 reasoning)
WIP: add port WIP: add debug flag
google/gemma-4-31b-it:high__json_schema PASS 100% 129 in → 315 out (1 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
google/gemma-4-31b-it:low PASS 100% 123 in → 212 out (165 reasoning)
WIP: add port WIP: add debug flag
google/gemma-4-31b-it:low__json_schema PASS 100% 127 in → 239 out (162 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
google/gemma-4-31b-it:medium PASS 100% 125 in → 303 out (1 reasoning)
WIP: add port WIP: add debug flag
google/gemma-4-31b-it:medium__json_schema PASS 100% 130 in → 233 out (215 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
google/gemma-4-31b-it:none PASS 100% 123 in → 12 out (0 reasoning)
WIP: add port WIP: add debug flag
google/gemma-4-31b-it:none__json_schema PASS 100% 122 in → 20 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
google/gemma-4-31b-it:xhigh PASS 100% 126 in → 272 out (213 reasoning)
WIP: add port WIP: add debug flag
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 125 in → 304 out (226 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
ibm-granite/granite-4.1-8b PASS 100% 115 in → 13 out
WIP: add debug flag WIP: add port
ibm-granite/granite-4.1-8b__json_schema PASS 100% 114 in → 20 out
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
inclusionai/ling-2.6-flash__json_schema PASS 100% 134 in → 21 out
WIP: add debug flag WIP: add port
JSON Schema Structured Output
(raw) {"commits":["WIP: add debug flag","WIP: add port"]}
minimax/minimax-m2.5:high PASS 100% 149 in → 609 out (595 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m2.5:high__json_schema PASS 100% 146 in → 287 out (259 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
minimax/minimax-m2.5:low PASS 100% 139 in → 577 out (563 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m2.5:low__json_schema PASS 100% 139 in → 455 out (450 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
minimax/minimax-m2.5:medium PASS 100% 140 in → 368 out (354 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m2.5:medium__json_schema PASS 100% 145 in → 410 out (382 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
minimax/minimax-m2.5:xhigh PASS 100% 142 in → 265 out (251 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 143 in → 170 out (142 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
minimax/minimax-m2.7:high PASS 100% 143 in → 319 out (304 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m2.7:low PASS 100% 143 in → 277 out (262 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m2.7:low__json_schema PASS 100% 292 in → 605 out (585 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
minimax/minimax-m2.7:medium PASS 100% 140 in → 480 out (486 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m2.7:xhigh PASS 100% 146 in → 521 out (507 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m3:high PASS 100% 279 in → 148 out (128 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m3:low PASS 100% 283 in → 140 out (120 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m3:medium PASS 100% 278 in → 130 out (120 reasoning)
WIP: add port WIP: add debug flag
minimax/minimax-m3:xhigh PASS 100% 278 in → 139 out (127 reasoning)
WIP: add port WIP: add debug flag
mistralai/devstral-2512__json_schema PASS 100% 118 in → 21 out
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
mistralai/mistral-medium-3-5:high PASS 100% 128 in → 4,500 out (4,717 reasoning)
WIP: add port WIP: add debug flag
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 117 in → 3,776 out (3,339 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
mistralai/mistral-medium-3-5:low PASS 100% 128 in → 5,995 out (5,233 reasoning)
WIP: add debug flag WIP: add port
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 115 in → 965 out (988 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
mistralai/mistral-medium-3-5:medium PASS 100% 127 in → 289 out (266 reasoning)
WIP: add port WIP: add debug flag
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 116 in → 329 out (281 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
mistralai/mistral-medium-3-5:none PASS 100% 130 in → 13 out (0 reasoning)
WIP: add port WIP: add debug flag
mistralai/mistral-medium-3-5:xhigh PASS 100% 131 in → 2,427 out (2,209 reasoning)
WIP: add port WIP: add debug flag
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 119 in → 327 out (267 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
moonshotai/kimi-k2.6:high PASS 100% 112 in → 1,006 out (992 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k2.6:high__json_schema PASS 100% 110 in → 890 out (870 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
moonshotai/kimi-k2.6:low PASS 100% 113 in → 1,705 out (1,717 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k2.6:low__json_schema PASS 100% 112 in → 1,060 out (834 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
moonshotai/kimi-k2.6:medium PASS 100% 113 in → 942 out (929 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k2.6:xhigh PASS 100% 111 in → 1,229 out (1,303 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k2.7-code:high PASS 100% 112 in → 156 out (142 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 110 in → 325 out (326 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": ["WIP: add port", "WIP: add debug flag"] }
moonshotai/kimi-k2.7-code:low PASS 100% 112 in → 194 out (180 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 113 in → 1,056 out (1,154 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
moonshotai/kimi-k2.7-code:medium PASS 100% 111 in → 230 out (216 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 205 in → 139 out (118 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 112 in → 200 out (191 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 204 in → 322 out (303 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
moonshotai/kimi-k3:high PASS 100% 188 in → 166 out (139 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k3:high__json_schema PASS 100% 294 in → 1,057 out (1,025 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
moonshotai/kimi-k3:low PASS 100% 188 in → 209 out (182 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k3:low__json_schema PASS 100% 294 in → 834 out (802 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
moonshotai/kimi-k3:max PASS 100% 193 in → 388 out (361 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k3:max__json_schema PASS 100% 297 in → 323 out (291 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
moonshotai/kimi-k3:medium PASS 100% 188 in → 235 out (208 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k3:medium__json_schema PASS 100% 185 in → 172 out (145 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k3:xhigh PASS 100% 187 in → 171 out (144 reasoning)
WIP: add port WIP: add debug flag
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 186 in → 168 out (141 reasoning)
WIP: add port WIP: add debug flag
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 131 in → 205 out (203 reasoning)
WIP: add port WIP: add debug flag
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 131 in → 301 out (298 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 128 in → 188 out (168 reasoning)
WIP: add port WIP: add debug flag
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 129 in → 258 out (234 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 129 in → 504 out (492 reasoning)
WIP: add port WIP: add debug flag
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 130 in → 397 out (377 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 130 in → 13 out (0 reasoning)
WIP: add port WIP: add debug flag
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 132 in → 28 out (0 reasoning)
WIP: add debug flag WIP: add port
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add debug flag", "WIP: add port" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 131 in → 354 out (300 reasoning)
WIP: add port WIP: add debug flag
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 128 in → 269 out (235 reasoning)
WIP: add port WIP: add debug flag
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 129 in → 240 out (198 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 133 in → 326 out (286 reasoning)
WIP: add port WIP: add debug flag
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 129 in → 200 out (160 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 131 in → 215 out (187 reasoning)
WIP: add port WIP: add debug flag
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 132 in → 218 out (176 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 130 in → 28 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 127 in → 397 out (366 reasoning)
WIP: add debug flag WIP: add port
openai/gpt-5.4-mini:high PASS 100% 111 in → 150 out (132 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4-mini:high__json_schema PASS 100% 155 in → 151 out (123 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4-mini:low PASS 100% 111 in → 52 out (34 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4-mini:low__json_schema PASS 100% 160 in → 50 out (22 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4-mini:medium PASS 100% 111 in → 101 out (83 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4-mini:medium__json_schema PASS 100% 157 in → 80 out (52 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4-mini:none PASS 100% 110 in → 22 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4-mini:none__json_schema PASS 100% 153 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4-mini:xhigh PASS 100% 113 in → 175 out (157 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 158 in → 217 out (189 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4-nano:high PASS 100% 112 in → 73 out (55 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4-nano:high__json_schema PASS 100% 157 in → 81 out (53 reasoning)
WIP: add debug flag WIP: add port
JSON Schema Structured Output
(raw) {"commits":["WIP: add debug flag","WIP: add port"]}
openai/gpt-5.4-nano:low__json_schema PASS 100% 159 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4-nano:medium PASS 100% 111 in → 51 out (33 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4-nano:medium__json_schema PASS 100% 158 in → 72 out (44 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4-nano:none__json_schema PASS 100% 159 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4-nano:xhigh PASS 100% 108 in → 233 out (215 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 160 in → 244 out (216 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4:high PASS 100% 111 in → 99 out (81 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4:high__json_schema PASS 100% 160 in → 88 out (60 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4:low PASS 100% 111 in → 41 out (23 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4:low__json_schema PASS 100% 158 in → 46 out (18 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4:medium PASS 100% 110 in → 45 out (27 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4:medium__json_schema PASS 100% 158 in → 70 out (42 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4:none PASS 100% 113 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4:none__json_schema PASS 100% 157 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.4:xhigh PASS 100% 107 in → 366 out (348 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.4:xhigh__json_schema PASS 100% 157 in → 161 out (133 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.5:high PASS 100% 107 in → 77 out (59 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.5:high__json_schema PASS 100% 159 in → 85 out (57 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.5:low PASS 100% 111 in → 38 out (20 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.5:low__json_schema PASS 100% 157 in → 51 out (23 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.5:medium PASS 100% 111 in → 56 out (38 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.5:medium__json_schema PASS 100% 155 in → 62 out (34 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.5:none PASS 100% 108 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.5:none__json_schema PASS 100% 157 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.5:xhigh PASS 100% 111 in → 332 out (314 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.5:xhigh__json_schema PASS 100% 161 in → 182 out (154 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-luna:high PASS 100% 111 in → 44 out (26 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-luna:high__json_schema PASS 100% 156 in → 79 out (51 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-luna:low PASS 100% 108 in → 34 out (16 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-luna:low__json_schema PASS 100% 158 in → 40 out (12 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-luna:max PASS 100% 112 in → 69 out (51 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-luna:max__json_schema PASS 100% 161 in → 120 out (92 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-luna:medium PASS 100% 110 in → 36 out (18 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-luna:medium__json_schema PASS 100% 158 in → 160 out (132 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-luna:none PASS 100% 114 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-luna:none__json_schema PASS 100% 156 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-luna:xhigh PASS 100% 111 in → 75 out (57 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 157 in → 67 out (39 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-sol:high PASS 100% 109 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-sol:high__json_schema PASS 100% 159 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-sol:low PASS 100% 109 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-sol:low__json_schema PASS 100% 159 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-sol:max PASS 100% 111 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-sol:max__json_schema PASS 100% 158 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-sol:medium PASS 100% 109 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-sol:medium__json_schema PASS 100% 156 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-sol:none PASS 100% 108 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-sol:none__json_schema PASS 100% 158 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-sol:xhigh PASS 100% 110 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 158 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-terra:high PASS 100% 110 in → 33 out (15 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-terra:high__json_schema PASS 100% 158 in → 45 out (17 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-terra:low PASS 100% 111 in → 39 out (21 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-terra:low__json_schema PASS 100% 156 in → 45 out (17 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-terra:max PASS 100% 107 in → 121 out (103 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-terra:max__json_schema PASS 100% 156 in → 81 out (53 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-terra:medium PASS 100% 110 in → 39 out (21 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-terra:medium__json_schema PASS 100% 157 in → 41 out (13 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-terra:none PASS 100% 114 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-terra:none__json_schema PASS 100% 161 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-5.6-terra:xhigh PASS 100% 112 in → 29 out (11 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 159 in → 44 out (16 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-oss-120b:high PASS 100% 170 in → 273 out (253 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-oss-120b:high__json_schema PASS 100% 171 in → 293 out (310 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-oss-120b:low PASS 100% 171 in → 85 out (64 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-oss-120b:low__json_schema PASS 100% 159 in → 139 out (84 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
openai/gpt-oss-120b:medium__json_schema PASS 100% 168 in → 180 out (140 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
openai/gpt-oss-120b:xhigh PASS 100% 158 in → 776 out (815 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 322 in → 886 out (858 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-oss-20b:high PASS 100% 160 in → 413 out (417 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-oss-20b:high__json_schema PASS 100% 175 in → 407 out (375 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-oss-20b:low PASS 100% 173 in → 66 out (45 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-oss-20b:low__json_schema PASS 100% 156 in → 101 out (71 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
openai/gpt-oss-20b:medium PASS 100% 168 in → 225 out (199 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-oss-20b:medium__json_schema PASS 100% 173 in → 280 out (253 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
openai/gpt-oss-20b:xhigh PASS 100% 171 in → 462 out (470 reasoning)
WIP: add port WIP: add debug flag
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 170 in → 565 out (542 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
poolside/laguna-m.1:high PASS 100% 120 in → 242 out (229 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-m.1:high__json_schema PASS 100% 121 in → 232 out (202 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
poolside/laguna-m.1:low PASS 100% 119 in → 280 out (267 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-m.1:low__json_schema PASS 100% 123 in → 269 out (251 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
poolside/laguna-m.1:medium PASS 100% 121 in → 330 out (317 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-m.1:medium__json_schema PASS 100% 121 in → 386 out (359 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
poolside/laguna-m.1:none PASS 100% 122 in → 12 out (0 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-m.1:none__json_schema PASS 100% 122 in → 30 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
poolside/laguna-m.1:xhigh PASS 100% 120 in → 282 out (269 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-m.1:xhigh__json_schema PASS 100% 123 in → 338 out (320 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
poolside/laguna-xs-2.1:high PASS 100% 122 in → 236 out (225 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-xs-2.1:high__json_schema PASS 100% 123 in → 339 out (308 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
poolside/laguna-xs-2.1:low PASS 100% 121 in → 322 out (311 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-xs-2.1:medium PASS 100% 121 in → 211 out (200 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-xs-2.1:none PASS 100% 121 in → 10 out (0 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-xs-2.1:none__json_schema FAIL 100% 118 in → 42 out (0 reasoning)
Finalize config WIP: add port WIP: add debug flag Initial config
JSON Schema Structured Output
(raw) { "commits": [ "Finalize config", "WIP: add port", "WIP: add debug flag", "Initial config" ] }
Failure: Extra selected commit messages: ['Finalize config', 'Initial config']
poolside/laguna-xs-2.1:xhigh PASS 100% 120 in → 175 out (164 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-xs.2:high PASS 100% 162 in → 240 out (227 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-xs.2:high__json_schema PASS 100% 158 in → 276 out (258 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
poolside/laguna-xs.2:low PASS 100% 159 in → 152 out (139 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-xs.2:low__json_schema PASS 100% 157 in → 235 out (217 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
poolside/laguna-xs.2:medium PASS 100% 158 in → 249 out (236 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-xs.2:medium__json_schema PASS 100% 161 in → 184 out (165 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
poolside/laguna-xs.2:none PASS 100% 156 in → 12 out (0 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-xs.2:none__json_schema PASS 100% 155 in → 31 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
poolside/laguna-xs.2:xhigh PASS 100% 158 in → 245 out (232 reasoning)
WIP: add port WIP: add debug flag
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 157 in → 197 out (178 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
qwen/qwen3.6-27b:high PASS 100% 124 in → 970 out (864 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-27b:high__json_schema PASS 100% 120 in → 838 out (730 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
qwen/qwen3.6-27b:low PASS 100% 124 in → 805 out (758 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-27b:low__json_schema PASS 100% 124 in → 1,283 out (1 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
qwen/qwen3.6-27b:medium PASS 100% 121 in → 663 out (1 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-27b:medium__json_schema PASS 100% 123 in → 836 out (1 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
qwen/qwen3.6-27b:none PASS 100% 125 in → 13 out (0 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-27b:xhigh PASS 100% 123 in → 1,245 out (1,182 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 127 in → 33 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 123 in → 1,250 out (1,170 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 124 in → 336 out (301 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 121 in → 942 out (884 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 126 in → 1,048 out (970 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": ["WIP: add port", "WIP: add debug flag"] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 125 in → 943 out (905 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 123 in → 993 out (903 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 125 in → 13 out (0 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 122 in → 1,443 out (1,344 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 121 in → 1,300 out (1,228 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
qwen/qwen3.6-flash:high PASS 100% 125 in → 1,008 out (991 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-flash:low PASS 100% 124 in → 864 out (847 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-flash:medium PASS 100% 122 in → 450 out (433 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.6-flash:none PASS 100% 124 in → 12 out (0 reasoning)
WIP: add debug flag WIP: add port
qwen/qwen3.6-flash:xhigh PASS 100% 123 in → 1,129 out (1,112 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-flash:high PASS 100% 124 in → 923 out (906 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-flash:low PASS 100% 124 in → 385 out (368 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-flash:max PASS 100% 124 in → 1,079 out (1,062 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-flash:medium PASS 100% 122 in → 1,440 out (1,423 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-flash:none PASS 100% 126 in → 12 out (0 reasoning)
WIP: add debug flag WIP: add port
qwen/qwen3.7-flash:xhigh PASS 100% 124 in → 1,109 out (1,092 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-max:high PASS 100% 120 in → 372 out (355 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-max:low PASS 100% 126 in → 957 out (940 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-max:medium PASS 100% 124 in → 399 out (382 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-max:none PASS 100% 124 in → 12 out (0 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-max:xhigh PASS 100% 123 in → 282 out (265 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-plus:high PASS 100% 126 in → 307 out (290 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-plus:low PASS 100% 124 in → 431 out (414 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-plus:medium PASS 100% 121 in → 690 out (673 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-plus:none PASS 100% 123 in → 12 out (0 reasoning)
WIP: add port WIP: add debug flag
qwen/qwen3.7-plus:xhigh PASS 100% 122 in → 925 out (908 reasoning)
WIP: add port WIP: add debug flag
tencent/hy3:high PASS 100% 116 in → 259 out (245 reasoning)
WIP: add port WIP: add debug flag
tencent/hy3:high__json_schema PASS 100% 117 in → 790 out (755 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
tencent/hy3:low PASS 100% 116 in → 347 out (333 reasoning)
WIP: add port WIP: add debug flag
tencent/hy3:low__json_schema PASS 100% 114 in → 359 out (348 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": ["WIP: add port", "WIP: add debug flag"] }
tencent/hy3:medium PASS 100% 116 in → 508 out (494 reasoning)
WIP: add port WIP: add debug flag
tencent/hy3:medium__json_schema PASS 100% 114 in → 421 out (390 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": ["WIP: add port", "WIP: add debug flag"] }
tencent/hy3:none PASS 100% 122 in → 13 out (0 reasoning)
WIP: add port WIP: add debug flag
tencent/hy3:none__json_schema PASS 100% 118 in → 26 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
tencent/hy3:xhigh PASS 100% 117 in → 597 out (583 reasoning)
WIP: add port WIP: add debug flag
tencent/hy3:xhigh__json_schema PASS 100% 116 in → 539 out (486 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": ["WIP: add port", "WIP: add debug flag"] }
thinkingmachines/inkling-small:high PASS 100% 120 in → 231 out (213 reasoning)
WIP: add port WIP: add debug flag
thinkingmachines/inkling-small:high__json_schema PASS 100% 120 in → 190 out (162 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
thinkingmachines/inkling-small:low PASS 100% 119 in → 51 out (29 reasoning)
WIP: add debug flag WIP: add port
thinkingmachines/inkling-small:low__json_schema PASS 100% 119 in → 54 out (26 reasoning)
WIP: add debug flag WIP: add port
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add debug flag", "WIP: add port" ] }
thinkingmachines/inkling-small:max PASS 100% 119 in → 380 out (377 reasoning)
WIP: add port WIP: add debug flag
thinkingmachines/inkling-small:max__json_schema PASS 100% 119 in → 185 out (153 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
thinkingmachines/inkling-small:medium PASS 100% 121 in → 310 out (292 reasoning)
WIP: add port WIP: add debug flag
thinkingmachines/inkling-small:medium__json_schema PASS 100% 116 in → 258 out (232 reasoning)
WIP: add debug flag WIP: add port
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add debug flag", "WIP: add port" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 119 in → 234 out (209 reasoning)
WIP: add port WIP: add debug flag
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 122 in → 128 out (96 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
thinkingmachines/inkling:high PASS 100% 117 in → 213 out (194 reasoning)
WIP: add port WIP: add debug flag
thinkingmachines/inkling:high__json_schema PASS 100% 120 in → 288 out (256 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
thinkingmachines/inkling:low PASS 100% 120 in → 49 out (31 reasoning)
WIP: add port WIP: add debug flag
thinkingmachines/inkling:low__json_schema PASS 100% 120 in → 56 out (23 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
thinkingmachines/inkling:max PASS 100% 118 in → 273 out (279 reasoning)
WIP: add port WIP: add debug flag
thinkingmachines/inkling:max__json_schema PASS 100% 119 in → 228 out (202 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": ["WIP: add port", "WIP: add debug flag"] }
thinkingmachines/inkling:medium PASS 100% 118 in → 80 out (61 reasoning)
WIP: add port WIP: add debug flag
thinkingmachines/inkling:none PASS 100% 116 in → 15 out (0 reasoning)
WIP: add debug flag WIP: add port
thinkingmachines/inkling:xhigh PASS 100% 119 in → 192 out (173 reasoning)
WIP: add port WIP: add debug flag
thinkingmachines/inkling:xhigh__json_schema PASS 100% 119 in → 301 out (269 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
x-ai/grok-4.3:high PASS 100% 284 in → 624 out (612 reasoning)
WIP: add port WIP: add debug flag
x-ai/grok-4.3:high__json_schema PASS 100% 352 in → 817 out (791 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
x-ai/grok-4.3:low PASS 100% 291 in → 249 out (237 reasoning)
WIP: add port WIP: add debug flag
x-ai/grok-4.3:low__json_schema PASS 100% 361 in → 418 out (402 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
x-ai/grok-4.3:max PASS 100% 284 in → 299 out (287 reasoning)
WIP: add port WIP: add debug flag
x-ai/grok-4.3:max__json_schema PASS 100% 350 in → 495 out (479 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
x-ai/grok-4.3:medium FAIL 100% 291 in → 905 out (822 reasoning)
WIP: add port WIP: add debug flag The provided history contains exactly two non-anchor commits, both clearly marked WIP and representing incremental config changes. These are the only subjects that qualify for squash/fixup; the two anchor commits (Initial config, Finalize config) are excluded by the problem statement. The lines are emitted verbatim and in the order they appear in the supplied newest-first list.
Failure: Extra selected commit messages: ['Finalize config']
x-ai/grok-4.3:medium__json_schema PASS 100% 359 in → 779 out (753 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "WIP: add port", "WIP: add debug flag" ] }
x-ai/grok-4.3:none__json_schema PASS 100% 351 in → 16 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
x-ai/grok-4.3:xhigh__json_schema PASS 100% 351 in → 574 out (558 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
x-ai/grok-4.5:high PASS 100% 303 in → 267 out (257 reasoning)
WIP: add port WIP: add debug flag
x-ai/grok-4.5:high__json_schema PASS 100% 383 in → 454 out (439 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
x-ai/grok-4.5:low PASS 100% 307 in → 179 out (169 reasoning)
WIP: add port WIP: add debug flag
x-ai/grok-4.5:low__json_schema PASS 100% 383 in → 323 out (308 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
x-ai/grok-4.5:max PASS 100% 307 in → 162 out (152 reasoning)
WIP: add port WIP: add debug flag
x-ai/grok-4.5:max__json_schema PASS 100% 389 in → 457 out (442 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
x-ai/grok-4.5:medium PASS 100% 308 in → 295 out (285 reasoning)
WIP: add port WIP: add debug flag
x-ai/grok-4.5:medium__json_schema PASS 100% 386 in → 399 out (384 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
x-ai/grok-4.5:xhigh PASS 100% 307 in → 227 out (217 reasoning)
WIP: add port WIP: add debug flag
x-ai/grok-4.5:xhigh__json_schema PASS 100% 384 in → 474 out (459 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
z-ai/glm-4.7-flash:high PASS 100% 112 in → 479 out (399 reasoning)
WIP: add port WIP: add debug flag
z-ai/glm-4.7-flash:low PASS 100% 109 in → 452 out (405 reasoning)
WIP: add port WIP: add debug flag
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 113 in → 408 out (342 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": ["WIP: add port", "WIP: add debug flag"] }
z-ai/glm-4.7-flash:none PASS 100% 110 in → 13 out (0 reasoning)
WIP: add port WIP: add debug flag
z-ai/glm-4.7-flash:none__json_schema PASS 100% 111 in → 20 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
z-ai/glm-4.7-flash:xhigh PASS 100% 113 in → 418 out (358 reasoning)
WIP: add port WIP: add debug flag
z-ai/glm-5.2:high PASS 100% 120 in → 225 out (211 reasoning)
WIP: add port WIP: add debug flag
z-ai/glm-5.2:low PASS 100% 117 in → 309 out (296 reasoning)
WIP: add port WIP: add debug flag
z-ai/glm-5.2:medium PASS 100% 118 in → 171 out (165 reasoning)
WIP: add port WIP: add debug flag
z-ai/glm-5.2:medium__json_schema PASS 100% 303 in → 19 out (0 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["WIP: add port","WIP: add debug flag"]}
z-ai/glm-5.2:none PASS 100% 115 in → 13 out (0 reasoning)
WIP: add port WIP: add debug flag
z-ai/glm-5.2:xhigh PASS 100% 120 in → 577 out (557 reasoning)
WIP: add port WIP: add debug flag
z-ai/glm-5.2:xhigh__json_schema PASS 100% 117 in → 174 out (153 reasoning)
WIP: add port WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["WIP: add port", "WIP: add debug flag"]}
openai/gpt-oss-120b:medium FAIL 50% 178 in → 161 out (140 reasoning)
WIP: add port WIP:
Failure: Missing expected commit messages: ['WIP: add debug flag']; Extra selected commit messages: ['WIP:']
thinkingmachines/inkling-small:none FAIL 50% 123 in → 15 out (0 reasoning)
WIP: add debug flag WIP: add port<|end_message|>
Failure: Missing expected commit messages: ['WIP: add port']; Extra selected commit messages: ['WIP: add port']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 306 in → 33 out (0 reasoning)
8f39d7e WIP: add port b7e577a WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["8f39d7e WIP: add port", "b7e577a WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
anthropic/claude-sonnet-5:high__json_schema FAIL 0% 399 in → 39 out (0 reasoning)
41cc2c0 WIP: add port 43eda79 WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["41cc2c0 WIP: add port","43eda79 WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
anthropic/claude-sonnet-5:low FAIL 0% 157 in → 29 out (0 reasoning)
1ad24ce WIP: add port 7d098fa WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 398 in → 39 out (0 reasoning)
322efd9 WIP: add port 3ba0c1a WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["322efd9 WIP: add port","3ba0c1a WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 395 in → 38 out (0 reasoning)
26c575b WIP: add port b4e27d1 WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["26c575b WIP: add port","b4e27d1 WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
anthropic/claude-sonnet-5:none FAIL 0% 154 in → 30 out (0 reasoning)
``` 2716812 WIP: add port 931b8da WIP: add debug flag ```
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['```', 'WIP: add port', 'WIP: add debug flag', '```']
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 394 in → 34 out (0 reasoning)
60e362e WIP: add port 376c086 WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["60e362e WIP: add port","376c086 WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
anthropic/claude-sonnet-5:xhigh FAIL 0% 160 in → 34 out (0 reasoning)
``` 6f26b49 WIP: add debug flag 8561cb6 WIP: add port ```
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['```', 'WIP: add debug flag', 'WIP: add port', '```']
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 397 in → 39 out (0 reasoning)
5f2cd01 WIP: add port 4584b9f WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["5f2cd01 WIP: add port","4584b9f WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 109 in → 378 out (381 reasoning)
WIP: add portWIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port']
arcee-ai/trinity-mini:high__json_schema FAIL 0% 113 in → 372 out (345 reasoning)
0d289c2 Finalize config 7d5b361 WIP: add port 3dd113d WIP: add debug flag 0e5bf0d Initial config
JSON Schema Structured Output
(raw) { "commits": [ "0d289c2 Finalize config", "7d5b361 WIP: add port", "3dd113d WIP: add debug flag", "0e5bf0d Initial config" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['Finalize config', 'WIP: add port', 'WIP: add debug flag', 'Initial config']
arcee-ai/trinity-mini:low FAIL 0% 109 in → 341 out (344 reasoning)
edd932a WIP: add port 1f156a4 WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 110 in → 455 out (419 reasoning)
a30594a Finalize config 1508ce1 WIP: add port 9014e4d WIP: add debug flag 06ebce7 Initial config
JSON Schema Structured Output
(raw) { "commits": [ "a30594a Finalize config", "1508ce1 WIP: add port", "9014e4d WIP: add debug flag", "06ebce7 Initial config" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['Finalize config', 'WIP: add port', 'WIP: add debug flag', 'Initial config']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 108 in → 297 out (260 reasoning)
403fcb5 Finalize config b0366a9 WIP: add port c2aaebf WIP: add debug flag 718d255 Initial config
JSON Schema Structured Output
(raw) { "commits": [ "403fcb5 Finalize config", "b0366a9 WIP: add port", "c2aaebf WIP: add debug flag", "718d255 Initial config" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['Finalize config', 'WIP: add port', 'WIP: add debug flag', 'Initial config']
arcee-ai/trinity-mini:xhigh FAIL 0% 106 in → 306 out (328 reasoning)
add portadd debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['add portadd debug flag']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 112 in → 384 out (339 reasoning)
6582b3a Finalize config 8505a6b WIP: add port 03da33f WIP: add debug flag 35b2d06 Initial config
JSON Schema Structured Output
(raw) { "commits": [ "6582b3a Finalize config", "8505a6b WIP: add port", "03da33f WIP: add debug flag", "35b2d06 Initial config" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['Finalize config', 'WIP: add port', 'WIP: add debug flag', 'Initial config']
deepseek/deepseek-v4-flash-0731:medium FAIL 0% 106 in → 242 out (221 reasoning)
92a678c WIP: add port 40efac3 WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
deepseek/deepseek-v4-flash-0731:medium__json_schema FAIL 0% 108 in → 108 out (71 reasoning)
b92522b WIP: add port 9411816 WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "b92522b WIP: add port", "9411816 WIP: add debug flag" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 104 in → 20 out (0 reasoning)
c687251 WIP: add debug flag 760c62e WIP: add port
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add debug flag', 'WIP: add port']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 111 in → 29 out (0 reasoning)
29ef0e6 WIP: add port 8126d09 WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["29ef0e6 WIP: add port","8126d09 WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output: WIP: add port WIP: add debug flag
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 105 in → 35 out (0 reasoning)
3837729 WIP: add port 10c5623 WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "3837729 WIP: add port", "10c5623 WIP: add debug flag" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
deepseek/deepseek-v4-pro:none FAIL 0% 107 in → 20 out (0 reasoning)
c45569c WIP: add port bc7c264 WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 110 in → 37 out (0 reasoning)
1a878d0 WIP: add port da1c171 WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "1a878d0 WIP: add port", "da1c171 WIP: add debug flag" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
deepseek/deepseek-v4-pro:xhigh FAIL 0% 107 in → 358 out (337 reasoning)
269007f WIP: add port ad48f88 WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 386 in → 1,778 out (1,748 reasoning)
a51db92 WIP: add port 530ca5e WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["a51db92 WIP: add port", "530ca5e WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
google/gemini-3-flash-preview:low__json_schema FAIL 0% 111 in → 917 out (888 reasoning)
8756815 WIP: add port 9fce09f WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["8756815 WIP: add port", "9fce09f WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
google/gemini-3.5-flash-lite:low FAIL 0% 107 in → 21 out (0 reasoning)
1799eda WIP: add port aee038b WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
google/gemini-3.6-flash:low__json_schema FAIL 0% 112 in → 18 out (0 reasoning)
816d826 WIP: add port b3fe376 WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "816d826 WIP: add port", "b3fe376 WIP: add debug flag" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
inclusionai/ling-2.6-flash FAIL 0% 133 in → 26 out
1792186 WIP: add port b6e6ffb WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
liquid/lfm-2-24b-a2b FAIL 0% 120 in → 12 out
f9f27c2 aa746e0
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: e544945 31227a1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:high__json_schema FAIL 0% 141 in → 55 out (55 reasoning)
90286b4 Finalize config 369a017 WIP: add port 3bdb6df WIP: add debug flag 33f26fc Initial config
JSON Schema Structured Output
(raw) { "commits": [ "90286b4 Finalize config", "369a017 WIP: add port", "3bdb6df WIP: add debug flag", "33f26fc Initial config" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['Finalize config', 'WIP: add port', 'WIP: add debug flag', 'Initial config']
minimax/minimax-m2.7:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high__json_schema FAIL 0% 226 in → 17 out (13 reasoning)
hash
JSON Schema Structured Output
(raw) { "commits": [ "hash" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['hash']
minimax/minimax-m3:low__json_schema FAIL 0% 231 in → 50 out (49 reasoning)
a7c52b6 Finalize config 74156bd WIP: add port 0f0fa2b WIP: add debug flag 599a0c1 Initial config
JSON Schema Structured Output
(raw) {"commits": ["a7c52b6 Finalize config", "74156bd WIP: add port", "0f0fa2b WIP: add debug flag", "599a0c1 Initial config"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['Finalize config', 'WIP: add port', 'WIP: add debug flag', 'Initial config']
minimax/minimax-m3:medium__json_schema FAIL 0% 231 in → 59 out (58 reasoning)
678a29b Finalize config 39a70f6 WIP: add port c4e95b0 WIP: add debug flag a0238c1 Initial config
JSON Schema Structured Output
(raw) { "commits": [ "678a29b Finalize config", "39a70f6 WIP: add port", "c4e95b0 WIP: add debug flag", "a0238c1 Initial config" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['Finalize config', 'WIP: add port', 'WIP: add debug flag', 'Initial config']
minimax/minimax-m3:xhigh__json_schema FAIL 0% 229 in → 33 out (32 reasoning)
0726b5a 9f7a97a bd28894 c47db12
JSON Schema Structured Output
(raw) { "commits": ["0726b5a", "9f7a97a", "bd28894", "c47db12"] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag', 'Finalize config', 'Initial config']
mistralai/devstral-2512 FAIL 0% 117 in → 25 out
92415ee WIP: add port 7ca835e WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
mistralai/mistral-medium-3-5:none__json_schema FAIL 0% 118 in → 34 out (0 reasoning)
cb45399 WIP: add debug flag b52578c WIP: add port
JSON Schema Structured Output
(raw) {"commits": ["cb45399 WIP: add debug flag", "b52578c WIP: add port"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add debug flag', 'WIP: add port']
moonshotai/kimi-k2.6:medium__json_schema FAIL 0% 108 in → 313 out (171 reasoning)
daded50 WIP: add port f4fcf10 WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "daded50 WIP: add port", "f4fcf10 WIP: add debug flag" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0%
Invalid JSON. Output: WIP: add port WIP: add debug flag
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 0% 133 in → 437 out (404 reasoning)
add port add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "add port", "add debug flag" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['add port', 'add debug flag']
nvidia/nemotron-3-super-120b-a12b:none FAIL 0% 132 in → 26 out (0 reasoning)
9d5eaf9 WIP: add port a28315a WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema FAIL 0% 130 in → 6,080 out (5,696 reasoning)
962fcf7 Finalize config 69bd226 WIP: add port 2b72e62 WIP: add debug flag f1858ca Initial config
JSON Schema Structured Output
(raw) { "commits": [ "962fcf7 Finalize config", "69bd226 WIP: add port", "2b72e62 WIP: add debug flag", "f1858ca Initial config" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['Finalize config', 'WIP: add port', 'WIP: add debug flag', 'Initial config']
openai/gpt-5.4-nano:low FAIL 0% 114 in → 26 out (0 reasoning)
7f5398a WIP: add port 31d0af2 WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
openai/gpt-5.4-nano:none FAIL 0% 113 in → 27 out (0 reasoning)
15e1a74 WIP: add port d5a2d59 WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
poolside/laguna-xs-2.1:low__json_schema FAIL 0% 124 in → 291 out (258 reasoning)
d7383a4 WIP: add port 187e24d WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["d7383a4 WIP: add port", "187e24d WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
poolside/laguna-xs-2.1:medium__json_schema FAIL 0% 120 in → 268 out (224 reasoning)
6668878 WIP: add port 9ffcc81 WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "6668878 WIP: add port", "9ffcc81 WIP: add debug flag" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
poolside/laguna-xs-2.1:xhigh__json_schema FAIL 0% 124 in → 211 out (166 reasoning)
91188a1 WIP: add port ea77f26 WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "91188a1 WIP: add port", "ea77f26 WIP: add debug flag" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 127 in → 47 out (0 reasoning)
2501e4a WIP: add debug flag 8b154f0 WIP: add port
JSON Schema Structured Output
(raw) { "commits": [ "2501e4a WIP: add debug flag", "8b154f0 WIP: add port" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add debug flag', 'WIP: add port']
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 126 in → 32 out (0 reasoning)
2d78742 WIP: add port e65b26e WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits":["2d78742 WIP: add port","e65b26e WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dcf40e33-43a9-9c8a-87c0-00b90e8d3901","request_id":"dcf40e33-43a9-9c8a-87c0-00b90e8d3901"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-96b6aa83-760e-975b-9565-3c0c508bd3d2","request_id":"96b6aa83-760e-975b-9565-3c0c508bd3d2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ce37662f-c74a-9148-8668-b6d912f27671","request_id":"ce37662f-c74a-9148-8668-b6d912f27671"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-23f366f0-cba2-9952-9004-c6cad81f684e","request_id":"23f366f0-cba2-9952-9004-c6cad81f684e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-54d27b36-50d4-9118-84c5-a1862dd312f3","request_id":"54d27b36-50d4-9118-84c5-a1862dd312f3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c371c7b0-28ee-961a-a823-a48b47c5cae6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-76adb0dc-d496-9aa9-9ae3-ec4c354e2189"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-48b1c2b1-91cb-93ad-9a3d-3fb6b705bb9e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c624ccb6-de29-904c-98a9-97b741d3cc13"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c7490517-70d0-9e02-b1e0-49d469a4d276"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e4e00aee-e152-9b23-ba57-83fcf928e0ef"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f8eeaf19-f8d6-9062-8fe1-906e485349ef","request_id":"f8eeaf19-f8d6-9062-8fe1-906e485349ef"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8c56a144-ba38-97f5-88d8-4295acc171b4","request_id":"8c56a144-ba38-97f5-88d8-4295acc171b4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-af9ede11-b082-9ceb-abf1-6db8b99e6677","request_id":"af9ede11-b082-9ceb-abf1-6db8b99e6677"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fc1801ea-6a7c-9bb4-97c4-0fea4d8c4e40","request_id":"fc1801ea-6a7c-9bb4-97c4-0fea4d8c4e40"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-27a6c798-4dd8-9f53-8e78-0d9f9ad33ad1","request_id":"27a6c798-4dd8-9f53-8e78-0d9f9ad33ad1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-10f22e20-317f-969a-a9de-e6c6278f6ff2","request_id":"10f22e20-317f-969a-a9de-e6c6278f6ff2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fbefc16f-275d-9bbb-bfb1-d0d4fb649687","request_id":"fbefc16f-275d-9bbb-bfb1-d0d4fb649687"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e098d72b-5aa1-93d2-be3a-0174334cec8f","request_id":"e098d72b-5aa1-93d2-be3a-0174334cec8f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f48e07dd-ae31-9687-987c-6c6fc8a8e009","request_id":"f48e07dd-ae31-9687-987c-6c6fc8a8e009"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1d53b8a2-88f5-99ad-a3cf-d68911feb0c0","request_id":"1d53b8a2-88f5-99ad-a3cf-d68911feb0c0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: WIP: add debug flag WIP: add port{"commits": ["e3bc5bd WIP: add debug flag", "ae86e76 WIP: add port"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: WIP: add port WIP: add debug flag
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: {
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 32840 column 1 (char 196488)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 32840 column 1 (char 196488)
x-ai/grok-4.3:none FAIL 0% 283 in → 17 out (0 reasoning)
e793b84 WIP: add port b065094 WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
x-ai/grok-4.3:xhigh FAIL 0% 286 in → 201 out (182 reasoning)
c4cf25b WIP: add port 66e391c WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium FAIL 0% 113 in → 378 out (320 reasoning)
fd3367a WIP: add port 70415a0 WIP: add debug flag
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:high__json_schema FAIL 0% 116 in → 184 out (148 reasoning)
a7d33fb WIP: add port bad9efe WIP: add debug flag
JSON Schema Structured Output
(raw) {"commits": ["a7d33fb WIP: add port", "bad9efe WIP: add debug flag"]}
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
z-ai/glm-5.2:low__json_schema FAIL 0% 306 in → 40 out (0 reasoning)
b06d9d0 WIP: add port b7958fc WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "b06d9d0 WIP: add port", "b7958fc WIP: add debug flag" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']
z-ai/glm-5.2:none__json_schema FAIL 0% 112 in → 32 out (0 reasoning)
87eda0f WIP: add port 400323d WIP: add debug flag
JSON Schema Structured Output
(raw) { "commits": [ "87eda0f WIP: add port", "400323d WIP: add debug flag" ] }
Failure: Missing expected commit messages: ['WIP: add debug flag', 'WIP: add port']; Extra selected commit messages: ['WIP: add port', 'WIP: add debug flag']