f001 — branch_cleanup

← branch_cleanup
Single merged branch
Tests ability to identify a single merged branch that can be safely deleted. Evaluates basic git branch --merged comprehension.
easy branch-management merged-branches cleanup basic
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Hello' > greeting.txt
  5. 05 git add greeting.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 git checkout -b fix-typo
  8. 08 echo 'Hello fixed' > greeting.txt
  9. 09 git add greeting.txt
  10. 10 git commit -m 'Fix typo'
  11. 11 git checkout main
  12. 12 git merge fix-typo
  13. 13 git checkout -b feature-login
  14. 14 echo 'Login code' > login.py
  15. 15 git add login.py
  16. 16 git commit -m 'Add login'
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
fix-typo
Fixture Metrics

Scoped model quality, cost, API time, and token usage for branch_cleanup/f001.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 131 in → 68 out (45 reasoning)
fix-typo
anthropic/claude-fable-5:high__json_schema PASS 100% 370 in → 69 out (33 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
anthropic/claude-fable-5:low PASS 100% 133 in → 41 out (26 reasoning)
fix-typo
anthropic/claude-fable-5:low__json_schema PASS 100% 374 in → 53 out (34 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
anthropic/claude-fable-5:max PASS 100% 132 in → 305 out (69 reasoning)
fix-typo
anthropic/claude-fable-5:max__json_schema PASS 100% 375 in → 188 out (58 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
anthropic/claude-fable-5:medium FAIL 100% 134 in → 159 out (54 reasoning)
fix-typo The `fix-typo` branch points to the same commit as `main` (6b6718e) and appears in the merged list, so it's safe to delete. `main` itself should never be deleted, and `feature-login` is not merged (its commit 201fdb2 doesn't appear in main's history).
Failure: Extra: ["The `fix-typo` branch points to the same commit as `main` (6b6718e) and appears in the merged list, so it's safe to delete. `main` itself should never be deleted, and `feature-login` is not merged (its commit 201fdb2 doesn't appear in main's history)."]
anthropic/claude-fable-5:medium__json_schema PASS 100% 372 in → 51 out (20 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
anthropic/claude-fable-5:xhigh PASS 100% 137 in → 76 out (49 reasoning)
fix-typo
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 374 in → 87 out (63 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
anthropic/claude-haiku-4.5:high PASS 100% 138 in → 229 out (199 reasoning)
fix-typo
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 325 in → 239 out (205 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
anthropic/claude-haiku-4.5:low PASS 100% 136 in → 201 out (162 reasoning)
fix-typo
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 329 in → 290 out (249 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
anthropic/claude-haiku-4.5:medium PASS 100% 137 in → 206 out (176 reasoning)
fix-typo
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 324 in → 496 out (454 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 293 in → 17 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 139 in → 157 out (134 reasoning)
fix-typo
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 332 in → 256 out (211 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
anthropic/claude-opus-4.6:high PASS 100% 106 in → 98 out (75 reasoning)
fix-typo
anthropic/claude-opus-4.6:high__json_schema PASS 100% 300 in → 75 out (41 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.6:low PASS 100% 109 in → 7 out (0 reasoning)
fix-typo
anthropic/claude-opus-4.6:low__json_schema PASS 100% 300 in → 17 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.6:max PASS 100% 112 in → 120 out (97 reasoning)
fix-typo
anthropic/claude-opus-4.6:max__json_schema PASS 100% 295 in → 148 out (124 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.6:medium PASS 100% 108 in → 7 out (0 reasoning)
fix-typo
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 299 in → 17 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.6:none PASS 100% 113 in → 7 out (0 reasoning)
fix-typo
anthropic/claude-opus-4.6:none__json_schema PASS 100% 294 in → 17 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 108 in → 83 out (55 reasoning)
fix-typo
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 301 in → 70 out (37 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.7:high PASS 100% 142 in → 9 out (0 reasoning)
fix-typo
anthropic/claude-opus-4.7:high__json_schema PASS 100% 384 in → 23 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.7:low PASS 100% 140 in → 9 out (0 reasoning)
fix-typo
anthropic/claude-opus-4.7:low__json_schema PASS 100% 374 in → 23 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.7:max PASS 100% 135 in → 193 out (67 reasoning)
fix-typo
anthropic/claude-opus-4.7:max__json_schema PASS 100% 374 in → 24 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.7:medium PASS 100% 137 in → 9 out (0 reasoning)
fix-typo
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 378 in → 23 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.7:none PASS 100% 142 in → 9 out (0 reasoning)
fix-typo
anthropic/claude-opus-4.7:none__json_schema PASS 100% 380 in → 23 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 138 in → 9 out (0 reasoning)
fix-typo
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 375 in → 23 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.8:high PASS 100% 136 in → 142 out (57 reasoning)
fix-typo
anthropic/claude-opus-4.8:high__json_schema PASS 100% 373 in → 21 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.8:low PASS 100% 134 in → 46 out (42 reasoning)
fix-typo
anthropic/claude-opus-4.8:low__json_schema PASS 100% 372 in → 21 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.8:max__json_schema PASS 100% 375 in → 181 out (65 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.8:medium PASS 100% 131 in → 50 out (23 reasoning)
fix-typo
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 369 in → 21 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.8:none PASS 100% 133 in → 7 out (0 reasoning)
fix-typo
anthropic/claude-opus-4.8:none__json_schema PASS 100% 373 in → 20 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 134 in → 205 out (67 reasoning)
fix-typo
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 376 in → 21 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-4.6:high PASS 100% 110 in → 99 out (80 reasoning)
fix-typo
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 297 in → 151 out (112 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-4.6:low PASS 100% 105 in → 7 out (0 reasoning)
fix-typo
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 293 in → 17 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-4.6:max PASS 100% 107 in → 185 out (154 reasoning)
fix-typo
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 300 in → 171 out (134 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 110 in → 7 out (0 reasoning)
fix-typo
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 300 in → 17 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-4.6:none PASS 100% 110 in → 7 out (0 reasoning)
fix-typo
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 295 in → 17 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 108 in → 128 out (108 reasoning)
fix-typo
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 298 in → 144 out (102 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-5:high PASS 100% 133 in → 89 out (54 reasoning)
fix-typo
anthropic/claude-sonnet-5:high__json_schema PASS 100% 372 in → 21 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-5:low PASS 100% 133 in → 7 out (0 reasoning)
fix-typo
anthropic/claude-sonnet-5:low__json_schema PASS 100% 377 in → 21 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-5:medium PASS 100% 132 in → 85 out (58 reasoning)
fix-typo
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 374 in → 21 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-5:none PASS 100% 136 in → 7 out (0 reasoning)
fix-typo
anthropic/claude-sonnet-5:none__json_schema PASS 100% 372 in → 20 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 130 in → 87 out (40 reasoning)
fix-typo
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 369 in → 22 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
arcee-ai/trinity-large-thinking:high PASS 100% 101 in → 760 out (738 reasoning)
fix-typo
arcee-ai/trinity-large-thinking:low PASS 100% 101 in → 944 out (922 reasoning)
fix-typo
arcee-ai/trinity-large-thinking:medium PASS 100% 102 in → 768 out (758 reasoning)
fix-typo
arcee-ai/trinity-large-thinking:xhigh PASS 100% 99 in → 1,017 out (1,005 reasoning)
fix-typo
arcee-ai/trinity-mini:high FAIL 100% 99 in → 676 out (726 reasoning)
fix-typo main
Failure: Extra: ['main']
arcee-ai/trinity-mini:high__json_schema PASS 100% 101 in → 796 out (849 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
arcee-ai/trinity-mini:low PASS 100% 99 in → 458 out (492 reasoning)
fix-typo
arcee-ai/trinity-mini:low__json_schema FAIL 100% 104 in → 836 out (936 reasoning)
fix-typo main
JSON Schema Structured Output
(raw) { "branches_to_delete":[ "fix-typo", "main" ] }
Failure: Extra: ['main']
arcee-ai/trinity-mini:medium PASS 100% 101 in → 520 out (552 reasoning)
fix-typo
arcee-ai/trinity-mini:medium__json_schema FAIL 100% 101 in → 639 out (675 reasoning)
fix-typo main
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo", "main" ] }
Failure: Extra: ['main']
arcee-ai/trinity-mini:xhigh PASS 100% 100 in → 545 out (591 reasoning)
fix-typo
arcee-ai/trinity-mini:xhigh__json_schema FAIL 100% 101 in → 1,129 out (1,259 reasoning)
fix-typo main
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo", "main" ] }
Failure: Extra: ['main']
deepseek/deepseek-v4-flash-0731:high PASS 100% 106 in → 72 out (82 reasoning)
fix-typo
deepseek/deepseek-v4-flash-0731:low PASS 100% 101 in → 346 out (340 reasoning)
fix-typo
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 99 in → 314 out (288 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
deepseek/deepseek-v4-flash-0731:max PASS 100% 183 in → 245 out (232 reasoning)
fix-typo
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 182 in → 398 out (397 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 101 in → 105 out (111 reasoning)
fix-typo
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 101 in → 433 out (422 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 102 in → 20 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 104 in → 156 out (196 reasoning)
fix-typo
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 187 in → 186 out (156 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
deepseek/deepseek-v4-flash:high PASS 100% 99 in → 127 out (123 reasoning)
fix-typo
deepseek/deepseek-v4-flash:high__json_schema FAIL 100% 101 in → 177 out (147 reasoning)
fix-typo main
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo", "main" ] }
Failure: Extra: ['main']
deepseek/deepseek-v4-flash:low PASS 100% 100 in → 217 out (212 reasoning)
fix-typo
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 102 in → 183 out (162 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
deepseek/deepseek-v4-flash:medium PASS 100% 98 in → 100 out (96 reasoning)
fix-typo
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 106 in → 129 out (108 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
deepseek/deepseek-v4-flash:none PASS 100% 101 in → 5 out (0 reasoning)
fix-typo
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 210 in → 14 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
deepseek/deepseek-v4-flash:xhigh PASS 100% 101 in → 204 out (187 reasoning)
fix-typo
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 181 in → 145 out (124 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
deepseek/deepseek-v4-pro:high PASS 100% 100 in → 192 out (186 reasoning)
fix-typo
deepseek/deepseek-v4-pro:low PASS 100% 98 in → 238 out (232 reasoning)
fix-typo
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 300 in → 434 out (419 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
deepseek/deepseek-v4-pro:medium PASS 100% 102 in → 291 out (285 reasoning)
fix-typo
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 101 in → 16 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
deepseek/deepseek-v4-pro:none PASS 100% 100 in → 4 out (0 reasoning)
fix-typo
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 103 in → 17 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 184 in → 553 out (547 reasoning)
fix-typo
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 377 in → 589 out (572 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
google/gemini-3-flash-preview:high PASS 100% 99 in → 243 out (239 reasoning)
fix-typo
google/gemini-3-flash-preview:high__json_schema PASS 100% 178 in → 306 out (294 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3-flash-preview:low PASS 100% 99 in → 266 out (262 reasoning)
fix-typo
google/gemini-3-flash-preview:low__json_schema PASS 100% 179 in → 214 out (202 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3-flash-preview:medium PASS 100% 102 in → 270 out (266 reasoning)
fix-typo
google/gemini-3-flash-preview:medium__json_schema PASS 100% 183 in → 374 out (361 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
google/gemini-3-flash-preview:xhigh PASS 100% 98 in → 220 out (216 reasoning)
fix-typo
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 183 in → 344 out (332 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 102 in → 399 out (395 reasoning)
fix-typo
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 101 in → 492 out (476 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
google/gemini-3.1-flash-lite-preview:low PASS 100% 96 in → 142 out (138 reasoning)
fix-typo
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 102 in → 121 out (97 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 102 in → 278 out (274 reasoning)
fix-typo
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 183 in → 275 out (251 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 99 in → 423 out (419 reasoning)
fix-typo
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 180 in → 443 out (430 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
google/gemini-3.1-pro-preview:high PASS 100% 101 in → 333 out (329 reasoning)
fix-typo
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 181 in → 326 out (314 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.1-pro-preview:low PASS 100% 99 in → 174 out (170 reasoning)
fix-typo
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 177 in → 69 out (57 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.1-pro-preview:medium PASS 100% 102 in → 327 out (323 reasoning)
fix-typo
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 181 in → 284 out (272 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 101 in → 361 out (357 reasoning)
fix-typo
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 101 in → 380 out (368 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.5-flash-lite:high PASS 100% 101 in → 444 out (440 reasoning)
fix-typo
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 100 in → 297 out (285 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 183 in → 24 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemini-3.5-flash-lite:max PASS 100% 100 in → 433 out (429 reasoning)
fix-typo
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 97 in → 487 out (474 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
google/gemini-3.5-flash-lite:medium PASS 100% 97 in → 346 out (342 reasoning)
fix-typo
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 99 in → 452 out (439 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 100 in → 299 out (295 reasoning)
fix-typo
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 100 in → 346 out (333 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
google/gemini-3.5-flash:high PASS 100% 100 in → 368 out (364 reasoning)
fix-typo
google/gemini-3.5-flash:high__json_schema PASS 100% 179 in → 341 out (329 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.5-flash:low PASS 100% 99 in → 251 out (247 reasoning)
fix-typo
google/gemini-3.5-flash:low__json_schema PASS 100% 179 in → 183 out (171 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.5-flash:medium PASS 100% 102 in → 350 out (346 reasoning)
fix-typo
google/gemini-3.5-flash:medium__json_schema PASS 100% 183 in → 328 out (316 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.5-flash:xhigh PASS 100% 101 in → 431 out (427 reasoning)
fix-typo
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 181 in → 492 out (480 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.6-flash:high PASS 100% 103 in → 348 out (344 reasoning)
fix-typo
google/gemini-3.6-flash:high__json_schema PASS 100% 175 in → 348 out (336 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.6-flash:low PASS 100% 97 in → 153 out (149 reasoning)
fix-typo
google/gemini-3.6-flash:low__json_schema PASS 100% 183 in → 24 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemini-3.6-flash:max PASS 100% 96 in → 367 out (363 reasoning)
fix-typo
google/gemini-3.6-flash:max__json_schema PASS 100% 179 in → 501 out (489 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.6-flash:medium PASS 100% 102 in → 399 out (395 reasoning)
fix-typo
google/gemini-3.6-flash:medium__json_schema PASS 100% 179 in → 344 out (332 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemini-3.6-flash:xhigh PASS 100% 101 in → 338 out (334 reasoning)
fix-typo
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 100 in → 329 out (318 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
google/gemma-4-26b-a4b-it:high PASS 100% 115 in → 370 out (314 reasoning)
fix-typo
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 117 in → 495 out (395 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 116 in → 563 out (505 reasoning)
fix-typo
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 116 in → 316 out (237 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 117 in → 357 out (352 reasoning)
fix-typo
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 118 in → 316 out (245 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 114 in → 5 out (0 reasoning)
fix-typo
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 110 in → 22 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 117 in → 292 out (258 reasoning)
fix-typo
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 114 in → 295 out (227 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemma-4-31b-it:high PASS 100% 117 in → 253 out (203 reasoning)
fix-typo
google/gemma-4-31b-it:high__json_schema PASS 100% 117 in → 207 out (169 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
google/gemma-4-31b-it:low PASS 100% 118 in → 206 out (201 reasoning)
fix-typo
google/gemma-4-31b-it:low__json_schema PASS 100% 118 in → 254 out (208 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
google/gemma-4-31b-it:medium PASS 100% 115 in → 198 out (167 reasoning)
fix-typo
google/gemma-4-31b-it:medium__json_schema PASS 100% 116 in → 284 out (213 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemma-4-31b-it:none PASS 100% 113 in → 5 out (0 reasoning)
fix-typo
google/gemma-4-31b-it:none__json_schema PASS 100% 115 in → 18 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
google/gemma-4-31b-it:xhigh PASS 100% 117 in → 250 out (1 reasoning)
fix-typo
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 119 in → 229 out (215 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
inclusionai/ling-2.6-flash__json_schema PASS 100% 125 in → 14 out
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
minimax/minimax-m2.5:high PASS 100% 132 in → 832 out (826 reasoning)
fix-typo
minimax/minimax-m2.5:high__json_schema PASS 100% 130 in → 337 out (317 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
minimax/minimax-m2.5:low PASS 100% 131 in → 249 out (244 reasoning)
fix-typo
minimax/minimax-m2.5:low__json_schema PASS 100% 135 in → 411 out (385 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
minimax/minimax-m2.5:medium PASS 100% 132 in → 319 out (346 reasoning)
fix-typo
minimax/minimax-m2.5:medium__json_schema PASS 100% 136 in → 896 out (907 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
minimax/minimax-m2.5:xhigh PASS 100% 133 in → 573 out (631 reasoning)
fix-typo
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 133 in → 456 out (481 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
minimax/minimax-m2.7:high PASS 100% 132 in → 321 out (314 reasoning)
fix-typo
minimax/minimax-m2.7:high__json_schema PASS 100% 132 in → 16 out (15 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
minimax/minimax-m2.7:low PASS 100% 134 in → 338 out (332 reasoning)
fix-typo
minimax/minimax-m2.7:low__json_schema PASS 100% 284 in → 425 out (411 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
minimax/minimax-m2.7:medium PASS 100% 132 in → 370 out (375 reasoning)
fix-typo
minimax/minimax-m2.7:medium__json_schema PASS 100% 281 in → 505 out (491 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
minimax/minimax-m2.7:xhigh PASS 100% 134 in → 403 out (397 reasoning)
fix-typo
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 280 in → 880 out (866 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
minimax/minimax-m3:high PASS 100% 272 in → 126 out (111 reasoning)
fix-typo
minimax/minimax-m3:high__json_schema PASS 100% 258 in → 16 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
minimax/minimax-m3:low PASS 100% 272 in → 178 out (186 reasoning)
fix-typo
minimax/minimax-m3:low__json_schema PASS 100% 260 in → 15 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
minimax/minimax-m3:medium PASS 100% 271 in → 472 out (484 reasoning)
fix-typo
minimax/minimax-m3:medium__json_schema PASS 100% 257 in → 15 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
minimax/minimax-m3:xhigh PASS 100% 271 in → 244 out (238 reasoning)
fix-typo
minimax/minimax-m3:xhigh__json_schema PASS 100% 257 in → 15 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
mistralai/devstral-2512 PASS 100% 106 in → 5 out
fix-typo
mistralai/devstral-2512__json_schema PASS 100% 105 in → 14 out
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
mistralai/mistral-medium-3-5:high PASS 100% 113 in → 476 out (476 reasoning)
fix-typo
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 104 in → 585 out (596 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
mistralai/mistral-medium-3-5:low PASS 100% 115 in → 395 out (390 reasoning)
fix-typo
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 104 in → 211 out (184 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
mistralai/mistral-medium-3-5:medium PASS 100% 116 in → 166 out (144 reasoning)
fix-typo
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 103 in → 802 out (780 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
mistralai/mistral-medium-3-5:none PASS 100% 119 in → 5 out (0 reasoning)
fix-typo
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 104 in → 14 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 115 in → 638 out (635 reasoning)
fix-typo
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 104 in → 268 out (251 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
moonshotai/kimi-k2.6:high PASS 100% 102 in → 324 out (318 reasoning)
fix-typo
moonshotai/kimi-k2.6:high__json_schema PASS 100% 99 in → 496 out (289 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
moonshotai/kimi-k2.6:low PASS 100% 101 in → 667 out (659 reasoning)
fix-typo
moonshotai/kimi-k2.6:low__json_schema PASS 100% 99 in → 418 out (426 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
moonshotai/kimi-k2.6:medium PASS 100% 103 in → 283 out (277 reasoning)
fix-typo
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 103 in → 803 out (562 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
moonshotai/kimi-k2.6:xhigh PASS 100% 96 in → 485 out (479 reasoning)
fix-typo
moonshotai/kimi-k2.7-code:high PASS 100% 98 in → 105 out (99 reasoning)
fix-typo
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 195 in → 238 out (224 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
moonshotai/kimi-k2.7-code:low PASS 100% 102 in → 153 out (147 reasoning)
fix-typo
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 99 in → 601 out (200 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
moonshotai/kimi-k2.7-code:medium PASS 100% 101 in → 207 out (203 reasoning)
fix-typo
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 99 in → 181 out (186 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 98 in → 127 out (121 reasoning)
fix-typo
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 101 in → 160 out (145 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
moonshotai/kimi-k3:high PASS 100% 173 in → 151 out (132 reasoning)
fix-typo
moonshotai/kimi-k3:high__json_schema PASS 100% 176 in → 181 out (162 reasoning)
fix-typo
moonshotai/kimi-k3:low PASS 100% 177 in → 98 out (79 reasoning)
fix-typo
moonshotai/kimi-k3:low__json_schema PASS 100% 179 in → 134 out (115 reasoning)
fix-typo
moonshotai/kimi-k3:max PASS 100% 176 in → 118 out (99 reasoning)
fix-typo
moonshotai/kimi-k3:max__json_schema PASS 100% 177 in → 139 out (120 reasoning)
fix-typo
moonshotai/kimi-k3:medium PASS 100% 175 in → 162 out (143 reasoning)
fix-typo
moonshotai/kimi-k3:medium__json_schema PASS 100% 177 in → 125 out (106 reasoning)
fix-typo
moonshotai/kimi-k3:xhigh PASS 100% 176 in → 146 out (127 reasoning)
fix-typo
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 285 in → 149 out (122 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 115 in → 270 out (257 reasoning)
fix-typo
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 116 in → 298 out (298 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 119 in → 299 out (305 reasoning)
fix-typo
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 116 in → 407 out (408 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 115 in → 265 out (279 reasoning)
fix-typo
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 118 in → 297 out (288 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 114 in → 11,112 out (11,918 reasoning)
fix-typo
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 118 in → 150 out (127 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": [ "fix-typo" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 120 in → 168 out (143 reasoning)
fix-typo
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 113 in → 144 out (121 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 119 in → 132 out (119 reasoning)
fix-typo
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 116 in → 167 out (138 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 116 in → 103 out (101 reasoning)
fix-typo
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 118 in → 161 out (137 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
nvidia/nemotron-3-super-120b-a12b:none__json_schema FAIL 100% 117 in → 24 out (0 reasoning)
fix-typo main
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo", "main" ] }
Failure: Extra: ['main']
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 118 in → 181 out (170 reasoning)
fix-typo
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 117 in → 189 out (156 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
openai/gpt-5.4-mini:high PASS 100% 94 in → 161 out (151 reasoning)
fix-typo
openai/gpt-5.4-mini:high__json_schema PASS 100% 137 in → 152 out (131 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4-mini:low PASS 100% 100 in → 56 out (46 reasoning)
fix-typo
openai/gpt-5.4-mini:low__json_schema PASS 100% 143 in → 78 out (57 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4-mini:medium PASS 100% 98 in → 134 out (124 reasoning)
fix-typo
openai/gpt-5.4-mini:medium__json_schema PASS 100% 143 in → 152 out (131 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4-mini:none PASS 100% 95 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.4-mini:none__json_schema PASS 100% 141 in → 19 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4-mini:xhigh PASS 100% 94 in → 647 out (637 reasoning)
fix-typo
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 140 in → 1,363 out (1,342 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4-nano:high PASS 100% 98 in → 84 out (74 reasoning)
fix-typo
openai/gpt-5.4-nano:high__json_schema PASS 100% 144 in → 97 out (76 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4-nano:low PASS 100% 100 in → 71 out (61 reasoning)
fix-typo
openai/gpt-5.4-nano:low__json_schema PASS 100% 144 in → 50 out (29 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4-nano:medium PASS 100% 97 in → 74 out (64 reasoning)
fix-typo
openai/gpt-5.4-nano:medium__json_schema PASS 100% 142 in → 79 out (58 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4-nano:none PASS 100% 97 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.4-nano:xhigh PASS 100% 95 in → 229 out (219 reasoning)
fix-typo
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 140 in → 386 out (365 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4:high PASS 100% 99 in → 141 out (131 reasoning)
fix-typo
openai/gpt-5.4:high__json_schema PASS 100% 140 in → 188 out (167 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4:low PASS 100% 98 in → 68 out (58 reasoning)
fix-typo
openai/gpt-5.4:low__json_schema PASS 100% 139 in → 53 out (32 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4:medium PASS 100% 99 in → 69 out (59 reasoning)
fix-typo
openai/gpt-5.4:medium__json_schema PASS 100% 141 in → 103 out (82 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4:none PASS 100% 100 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.4:none__json_schema PASS 100% 137 in → 19 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.4:xhigh PASS 100% 98 in → 134 out (124 reasoning)
fix-typo
openai/gpt-5.4:xhigh__json_schema PASS 100% 146 in → 266 out (245 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.5:high PASS 100% 99 in → 99 out (89 reasoning)
fix-typo
openai/gpt-5.5:high__json_schema PASS 100% 147 in → 213 out (192 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.5:low PASS 100% 97 in → 52 out (42 reasoning)
fix-typo
openai/gpt-5.5:low__json_schema PASS 100% 141 in → 19 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.5:medium PASS 100% 97 in → 57 out (47 reasoning)
fix-typo
openai/gpt-5.5:medium__json_schema PASS 100% 144 in → 85 out (64 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.5:none PASS 100% 97 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.5:none__json_schema PASS 100% 138 in → 19 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.5:xhigh PASS 100% 97 in → 129 out (119 reasoning)
fix-typo
openai/gpt-5.5:xhigh__json_schema PASS 100% 139 in → 191 out (170 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-luna:high PASS 100% 95 in → 112 out (102 reasoning)
fix-typo
openai/gpt-5.6-luna:high__json_schema PASS 100% 137 in → 122 out (101 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-luna:low PASS 100% 100 in → 45 out (35 reasoning)
fix-typo
openai/gpt-5.6-luna:low__json_schema PASS 100% 140 in → 123 out (102 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-luna:max PASS 100% 99 in → 159 out (149 reasoning)
fix-typo
openai/gpt-5.6-luna:max__json_schema PASS 100% 139 in → 537 out (516 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-luna:medium PASS 100% 92 in → 103 out (93 reasoning)
fix-typo
openai/gpt-5.6-luna:medium__json_schema PASS 100% 139 in → 77 out (56 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-luna:none PASS 100% 97 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.6-luna:none__json_schema PASS 100% 139 in → 19 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-luna:xhigh PASS 100% 100 in → 78 out (68 reasoning)
fix-typo
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 142 in → 357 out (336 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-sol:high PASS 100% 101 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.6-sol:high__json_schema PASS 100% 144 in → 19 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-sol:low PASS 100% 95 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.6-sol:low__json_schema PASS 100% 142 in → 19 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-sol:max PASS 100% 96 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.6-sol:max__json_schema PASS 100% 141 in → 104 out (83 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-sol:medium PASS 100% 96 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.6-sol:medium__json_schema PASS 100% 137 in → 65 out (44 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-sol:none PASS 100% 100 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.6-sol:none__json_schema PASS 100% 139 in → 19 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-sol:xhigh PASS 100% 99 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 142 in → 70 out (49 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-terra:high PASS 100% 98 in → 59 out (49 reasoning)
fix-typo
openai/gpt-5.6-terra:high__json_schema PASS 100% 142 in → 19 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-terra:low PASS 100% 97 in → 45 out (35 reasoning)
fix-typo
openai/gpt-5.6-terra:low__json_schema PASS 100% 140 in → 61 out (40 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-terra:max PASS 100% 97 in → 67 out (57 reasoning)
fix-typo
openai/gpt-5.6-terra:max__json_schema PASS 100% 143 in → 114 out (93 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-terra:medium PASS 100% 101 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.6-terra:medium__json_schema PASS 100% 140 in → 54 out (33 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-terra:none PASS 100% 104 in → 8 out (0 reasoning)
fix-typo
openai/gpt-5.6-terra:none__json_schema PASS 100% 145 in → 19 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-5.6-terra:xhigh PASS 100% 96 in → 68 out (58 reasoning)
fix-typo
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 143 in → 58 out (37 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-oss-120b:high PASS 100% 149 in → 367 out (400 reasoning)
fix-typo
openai/gpt-oss-120b:high__json_schema PASS 100% 165 in → 417 out (415 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
openai/gpt-oss-120b:low__json_schema PASS 100% 157 in → 82 out (56 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
openai/gpt-oss-120b:medium PASS 100% 161 in → 164 out (160 reasoning)
fix-typo
openai/gpt-oss-120b:medium__json_schema PASS 100% 157 in → 155 out (141 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-oss-120b:xhigh PASS 100% 144 in → 277 out (274 reasoning)
fix-typo
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 303 in → 361 out (340 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-oss-20b:high PASS 100% 158 in → 399 out (392 reasoning)
fix-typo
openai/gpt-oss-20b:high__json_schema PASS 100% 159 in → 661 out (635 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-oss-20b:low PASS 100% 165 in → 80 out (67 reasoning)
fix-typo
openai/gpt-oss-20b:low__json_schema PASS 100% 158 in → 84 out (60 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-oss-20b:medium PASS 100% 157 in → 249 out (233 reasoning)
fix-typo
openai/gpt-oss-20b:medium__json_schema PASS 100% 145 in → 281 out (284 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
openai/gpt-oss-20b:xhigh PASS 100% 155 in → 532 out (560 reasoning)
fix-typo
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 155 in → 568 out (551 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
poolside/laguna-m.1:high PASS 100% 113 in → 342 out (336 reasoning)
fix-typo
poolside/laguna-m.1:high__json_schema PASS 100% 113 in → 231 out (215 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
poolside/laguna-m.1:low PASS 100% 113 in → 448 out (442 reasoning)
fix-typo
poolside/laguna-m.1:low__json_schema PASS 100% 115 in → 475 out (463 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
poolside/laguna-m.1:medium PASS 100% 108 in → 298 out (292 reasoning)
fix-typo
poolside/laguna-m.1:medium__json_schema PASS 100% 113 in → 334 out (322 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
poolside/laguna-m.1:none PASS 100% 114 in → 5 out (0 reasoning)
fix-typo
poolside/laguna-m.1:none__json_schema FAIL 100% 109 in → 14 out (0 reasoning)
fix-typo main
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo", "main"]}
Failure: Extra: ['main']
poolside/laguna-m.1:xhigh PASS 100% 115 in → 231 out (225 reasoning)
fix-typo
poolside/laguna-m.1:xhigh__json_schema PASS 100% 111 in → 272 out (252 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
poolside/laguna-xs-2.1:high PASS 100% 109 in → 151 out (147 reasoning)
fix-typo
poolside/laguna-xs-2.1:high__json_schema PASS 100% 113 in → 149 out (137 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
poolside/laguna-xs-2.1:low PASS 100% 113 in → 321 out (317 reasoning)
fix-typo
poolside/laguna-xs-2.1:low__json_schema PASS 100% 108 in → 315 out (303 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
poolside/laguna-xs-2.1:medium PASS 100% 112 in → 144 out (140 reasoning)
fix-typo
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 112 in → 305 out (293 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
poolside/laguna-xs-2.1:none PASS 100% 108 in → 3 out (0 reasoning)
fix-typo
poolside/laguna-xs-2.1:none__json_schema PASS 100% 113 in → 22 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
poolside/laguna-xs-2.1:xhigh PASS 100% 113 in → 521 out (517 reasoning)
fix-typo
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 112 in → 330 out (307 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
poolside/laguna-xs.2:high PASS 100% 152 in → 123 out (117 reasoning)
fix-typo
poolside/laguna-xs.2:high__json_schema PASS 100% 145 in → 137 out (125 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
poolside/laguna-xs.2:low PASS 100% 143 in → 194 out (188 reasoning)
fix-typo
poolside/laguna-xs.2:low__json_schema PASS 100% 148 in → 190 out (178 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
poolside/laguna-xs.2:medium PASS 100% 150 in → 351 out (345 reasoning)
fix-typo
poolside/laguna-xs.2:medium__json_schema PASS 100% 147 in → 208 out (185 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
poolside/laguna-xs.2:none FAIL 100% 147 in → 8 out (0 reasoning)
feature-login fix-typo
Failure: Extra: ['feature-login']
poolside/laguna-xs.2:none__json_schema FAIL 100% 147 in → 15 out (0 reasoning)
feature-login fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-login", "fix-typo"]}
Failure: Extra: ['feature-login']
poolside/laguna-xs.2:xhigh PASS 100% 145 in → 154 out (148 reasoning)
fix-typo
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 151 in → 253 out (241 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
qwen/qwen3.6-27b:high PASS 100% 112 in → 828 out (728 reasoning)
fix-typo
qwen/qwen3.6-27b:high__json_schema PASS 100% 109 in → 770 out (708 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
qwen/qwen3.6-27b:low PASS 100% 110 in → 724 out (645 reasoning)
fix-typo
qwen/qwen3.6-27b:low__json_schema PASS 100% 112 in → 909 out (836 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
qwen/qwen3.6-27b:medium PASS 100% 110 in → 769 out (1 reasoning)
fix-typo
qwen/qwen3.6-27b:medium__json_schema PASS 100% 110 in → 933 out (1 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
qwen/qwen3.6-27b:none PASS 100% 111 in → 4 out (0 reasoning)
fix-typo
qwen/qwen3.6-27b:none__json_schema PASS 100% 113 in → 22 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 106 in → 656 out (648 reasoning)
fix-typo
qwen/qwen3.6-35b-a3b:high PASS 100% 110 in → 877 out (822 reasoning)
fix-typo
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 107 in → 1,193 out (1,101 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
qwen/qwen3.6-35b-a3b:low PASS 100% 110 in → 944 out (900 reasoning)
fix-typo
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 113 in → 977 out (856 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 107 in → 1,082 out (1,036 reasoning)
fix-typo
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 109 in → 887 out (790 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 111 in → 4 out (0 reasoning)
fix-typo
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 111 in → 22 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 113 in → 1,090 out (1,082 reasoning)
fix-typo
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 109 in → 914 out (842 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
qwen/qwen3.6-flash:high PASS 100% 111 in → 1,027 out (1,019 reasoning)
fix-typo
qwen/qwen3.6-flash:low PASS 100% 112 in → 994 out (986 reasoning)
fix-typo
qwen/qwen3.6-flash:medium PASS 100% 105 in → 874 out (866 reasoning)
fix-typo
qwen/qwen3.6-flash:none PASS 100% 110 in → 3 out (0 reasoning)
fix-typo
qwen/qwen3.6-flash:xhigh PASS 100% 108 in → 1,106 out (1,098 reasoning)
fix-typo
qwen/qwen3.7-flash:high PASS 100% 108 in → 958 out (950 reasoning)
fix-typo
qwen/qwen3.7-flash:low PASS 100% 107 in → 812 out (804 reasoning)
fix-typo
qwen/qwen3.7-flash:max PASS 100% 111 in → 889 out (881 reasoning)
fix-typo
qwen/qwen3.7-flash:medium PASS 100% 111 in → 588 out (580 reasoning)
fix-typo
qwen/qwen3.7-flash:none PASS 100% 114 in → 3 out (0 reasoning)
fix-typo
qwen/qwen3.7-flash:none__json_schema PASS 100% 114 in → 3 out (0 reasoning)
fix-typo
qwen/qwen3.7-flash:xhigh PASS 100% 107 in → 448 out (440 reasoning)
fix-typo
qwen/qwen3.7-max:high PASS 100% 106 in → 538 out (530 reasoning)
fix-typo
qwen/qwen3.7-max:low PASS 100% 110 in → 219 out (211 reasoning)
fix-typo
qwen/qwen3.7-max:medium PASS 100% 111 in → 227 out (219 reasoning)
fix-typo
qwen/qwen3.7-max:none PASS 100% 111 in → 3 out (0 reasoning)
fix-typo
qwen/qwen3.7-max:xhigh PASS 100% 109 in → 200 out (192 reasoning)
fix-typo
qwen/qwen3.7-plus:high PASS 100% 110 in → 691 out (684 reasoning)
fix-typo
qwen/qwen3.7-plus:low PASS 100% 108 in → 504 out (496 reasoning)
fix-typo
qwen/qwen3.7-plus:medium PASS 100% 109 in → 703 out (695 reasoning)
fix-typo
qwen/qwen3.7-plus:none PASS 100% 115 in → 3 out (0 reasoning)
fix-typo
qwen/qwen3.7-plus:xhigh PASS 100% 109 in → 706 out (698 reasoning)
fix-typo
tencent/hy3:high PASS 100% 109 in → 480 out (474 reasoning)
fix-typo
tencent/hy3:low PASS 100% 107 in → 347 out (341 reasoning)
fix-typo
tencent/hy3:low__json_schema PASS 100% 112 in → 316 out (258 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
tencent/hy3:medium PASS 100% 112 in → 357 out (351 reasoning)
fix-typo
tencent/hy3:medium__json_schema PASS 100% 106 in → 527 out (500 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
tencent/hy3:none PASS 100% 111 in → 5 out (0 reasoning)
fix-typo
tencent/hy3:none__json_schema PASS 100% 111 in → 20 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
tencent/hy3:xhigh PASS 100% 107 in → 501 out (495 reasoning)
fix-typo
tencent/hy3:xhigh__json_schema PASS 100% 110 in → 742 out (664 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
thinkingmachines/inkling-small:high PASS 100% 109 in → 191 out (187 reasoning)
fix-typo
thinkingmachines/inkling-small:high__json_schema PASS 100% 108 in → 215 out (191 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
thinkingmachines/inkling-small:low PASS 100% 110 in → 73 out (72 reasoning)
fix-typo
thinkingmachines/inkling-small:low__json_schema PASS 100% 106 in → 45 out (27 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-typo"]}
thinkingmachines/inkling-small:max PASS 100% 105 in → 297 out (287 reasoning)
fix-typo
thinkingmachines/inkling-small:max__json_schema PASS 100% 110 in → 137 out (119 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
thinkingmachines/inkling-small:medium PASS 100% 108 in → 164 out (154 reasoning)
fix-typo
thinkingmachines/inkling-small:medium__json_schema PASS 100% 104 in → 190 out (166 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
thinkingmachines/inkling-small:none PASS 100% 103 in → 7 out (0 reasoning)
fix-typo
thinkingmachines/inkling-small:xhigh PASS 100% 109 in → 176 out (166 reasoning)
fix-typo
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 107 in → 123 out (104 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
thinkingmachines/inkling:high PASS 100% 104 in → 196 out (185 reasoning)
fix-typo
thinkingmachines/inkling:low PASS 100% 106 in → 105 out (104 reasoning)
fix-typo
thinkingmachines/inkling:max PASS 100% 108 in → 211 out (201 reasoning)
fix-typo
thinkingmachines/inkling:medium PASS 100% 107 in → 167 out (156 reasoning)
fix-typo
thinkingmachines/inkling:medium__json_schema PASS 100% 108 in → 173 out (150 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
thinkingmachines/inkling:xhigh PASS 100% 108 in → 429 out (418 reasoning)
fix-typo
thinkingmachines/inkling:xhigh__json_schema PASS 100% 104 in → 233 out (214 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
x-ai/grok-4.3:high PASS 100% 274 in → 512 out (509 reasoning)
fix-typo
x-ai/grok-4.3:high__json_schema PASS 100% 340 in → 595 out (586 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
x-ai/grok-4.3:low PASS 100% 278 in → 319 out (316 reasoning)
fix-typo
x-ai/grok-4.3:low__json_schema PASS 100% 344 in → 394 out (385 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
x-ai/grok-4.3:max PASS 100% 269 in → 578 out (575 reasoning)
fix-typo
x-ai/grok-4.3:max__json_schema PASS 100% 338 in → 479 out (470 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
x-ai/grok-4.3:medium PASS 100% 278 in → 486 out (483 reasoning)
fix-typo
x-ai/grok-4.3:medium__json_schema PASS 100% 345 in → 666 out (657 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
x-ai/grok-4.3:xhigh PASS 100% 272 in → 245 out (242 reasoning)
fix-typo
x-ai/grok-4.3:xhigh__json_schema PASS 100% 343 in → 695 out (686 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
x-ai/grok-4.5:high PASS 100% 302 in → 561 out (557 reasoning)
fix-typo
x-ai/grok-4.5:high__json_schema PASS 100% 386 in → 414 out (401 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
x-ai/grok-4.5:low PASS 100% 300 in → 278 out (274 reasoning)
fix-typo
x-ai/grok-4.5:low__json_schema PASS 100% 384 in → 290 out (277 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
x-ai/grok-4.5:max PASS 100% 302 in → 400 out (396 reasoning)
fix-typo
x-ai/grok-4.5:max__json_schema PASS 100% 385 in → 522 out (509 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
x-ai/grok-4.5:medium PASS 100% 299 in → 483 out (479 reasoning)
fix-typo
x-ai/grok-4.5:medium__json_schema PASS 100% 384 in → 334 out (321 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
x-ai/grok-4.5:xhigh PASS 100% 305 in → 342 out (338 reasoning)
fix-typo
x-ai/grok-4.5:xhigh__json_schema PASS 100% 384 in → 477 out (464 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
z-ai/glm-4.7-flash:high PASS 100% 97 in → 690 out (701 reasoning)
fix-typo
z-ai/glm-4.7-flash:low PASS 100% 96 in → 1,407 out (1,319 reasoning)
fix-typo
z-ai/glm-4.7-flash:medium PASS 100% 102 in → 489 out (481 reasoning)
fix-typo
z-ai/glm-4.7-flash:xhigh PASS 100% 97 in → 535 out (480 reasoning)
fix-typo
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 98 in → 520 out (461 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
z-ai/glm-5.2:high PASS 100% 105 in → 283 out (270 reasoning)
fix-typo
z-ai/glm-5.2:high__json_schema PASS 100% 287 in → 228 out (217 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
z-ai/glm-5.2:low PASS 100% 107 in → 209 out (187 reasoning)
fix-typo
z-ai/glm-5.2:low__json_schema PASS 100% 103 in → 166 out (139 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ] }
z-ai/glm-5.2:medium PASS 100% 107 in → 204 out (190 reasoning)
fix-typo
z-ai/glm-5.2:medium__json_schema PASS 100% 100 in → 393 out (325 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-typo"] }
z-ai/glm-5.2:none PASS 100% 101 in → 3 out (0 reasoning)
fix-typo
z-ai/glm-5.2:none__json_schema PASS 100% 285 in → 10 out (0 reasoning)
fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-typo"]}
z-ai/glm-5.2:xhigh PASS 100% 107 in → 379 out (328 reasoning)
fix-typo
z-ai/glm-5.2:xhigh__json_schema PASS 100% 107 in → 354 out (311 reasoning)
fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-typo" ]}
anthropic/claude-haiku-4.5:none FAIL 0% 110 in → 6 out (0 reasoning)
feature-login
Failure: Missing: ['fix-typo'] Extra: ['feature-login']
anthropic/claude-opus-4.8:max FAIL 0% 136 in → 683 out (51 reasoning)
**fix-typo** Reasoning: - `fix-typo` — merged into main (same commit `b38c1c1`) and is not the main branch itself → safe to delete. - `main` — although it appears in the merged list, it *is* the target branch, so it should never be deleted. - `feature-login` — not in the merged list (points to `6401386 Add login`, which isn't in main) → keep it.
Failure: Missing: ['fix-typo'] Extra: ['**fix-typo**', "- `feature-login` — not in the merged list (points to `6401386 Add login`, which isn't in main) → keep it.", '- `fix-typo` — merged into main (same commit `b38c1c1`) and is not the main branch itself → safe to delete.', '- `main` — although it appears in the merged list, it *is* the target branch, so it should never be deleted.', 'Reasoning:']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash-0731:high__json_schema FAIL 0%
Invalid JSON. Output: { { "branches_to_delete": ["fix-typo"] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)
deepseek/deepseek-v4-flash-0731:none FAIL 0% 101 in → 4 out (0 reasoning)
feature-login
Failure: Missing: ['fix-typo'] Extra: ['feature-login']
deepseek/deepseek-v4-pro:high__json_schema FAIL 0%
Invalid JSON. Output: fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
google/gemini-3.5-flash-lite:low FAIL 0% 102 in → 3 out (0 reasoning)
feature-login
Failure: Missing: ['fix-typo'] Extra: ['feature-login']
ibm-granite/granite-4.1-8b FAIL 0% 108 in → 4 out
feature-Login
Failure: Missing: ['fix-typo'] Extra: ['feature-Login']
ibm-granite/granite-4.1-8b__json_schema FAIL 0% 110 in → 19 out
feature-login
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-login" ] }
Failure: Missing: ['fix-typo'] Extra: ['feature-login']
inclusionai/ling-2.6-flash FAIL 0% 122 in → 6 out
feature-login main
Failure: Missing: ['fix-typo'] Extra: ['feature-login', 'main']
liquid/lfm-2-24b-a2b FAIL 0% 112 in → 5 out
feature-login
Failure: Missing: ['fix-typo'] Extra: ['feature-login']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: feature-login
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0%
Invalid JSON. Output: fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 114 in → 2 out (0 reasoning)
none
Failure: Missing: ['fix-typo'] Extra: ['none']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 120 in → 19 out (0 reasoning)
feature-login
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-login" ] }
Failure: Missing: ['fix-typo'] Extra: ['feature-login']
nvidia/nemotron-3-super-120b-a12b:none FAIL 0% 116 in → 2 out (0 reasoning)
none
Failure: Missing: ['fix-typo'] Extra: ['none']
openai/gpt-5.4-nano:none__json_schema FAIL 0% 141 in → 17 out (0 reasoning)
feature-login
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-login"]}
Failure: Missing: ['fix-typo'] Extra: ['feature-login']
openai/gpt-oss-120b:low FAIL 0% 161 in → 85 out (73 reasoning)
fix
Failure: Missing: ['fix-typo'] Extra: ['fix']
qwen/qwen3.6-27b:xhigh__json_schema FAIL 0%
Invalid JSON. Output: fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-11114c9f-b272-9778-9c13-10f46d38dc90","request_id":"11114c9f-b272-9778-9c13-10f46d38dc90"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f9845841-4f26-9b40-a3f4-1bc2019f7c9b","request_id":"f9845841-4f26-9b40-a3f4-1bc2019f7c9b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-71e11df6-1d3a-9f1b-849d-c276ff44cdca","request_id":"71e11df6-1d3a-9f1b-849d-c276ff44cdca"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3eaca4ba-d15c-9b14-959e-4fd172765708","request_id":"3eaca4ba-d15c-9b14-959e-4fd172765708"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-23519c3c-153b-9230-b5f7-b83713c5c5ad","request_id":"23519c3c-153b-9230-b5f7-b83713c5c5ad"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-772722ea-b59a-994e-bba7-fc5c71e20652"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-46b8dfde-9abb-9928-bc2f-d82c2fe4e014"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1c03cb15-6c4b-9dd8-88dd-3b7645000e55"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-94944ba4-bea3-9ba4-8bfd-bd6817882942"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-73bce3e6-7194-9128-af4d-08ab6e4dc8f5"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-801afa75-d063-9323-84a4-02e666b14513","request_id":"801afa75-d063-9323-84a4-02e666b14513"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b62d2456-9b6c-9191-8cbf-6325f06a8144","request_id":"b62d2456-9b6c-9191-8cbf-6325f06a8144"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-10501a75-a6d0-9ebe-a8d6-a0c9bbeacd13","request_id":"10501a75-a6d0-9ebe-a8d6-a0c9bbeacd13"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e50c14a6-1c5c-9134-a49f-f82b6f3a5728","request_id":"e50c14a6-1c5c-9134-a49f-f82b6f3a5728"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2c7e7489-c919-9a32-8105-a86cfca2f8ac","request_id":"2c7e7489-c919-9a32-8105-a86cfca2f8ac"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fd02fe38-1b70-90ae-98c3-8818db9a50c5","request_id":"fd02fe38-1b70-90ae-98c3-8818db9a50c5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-239d7713-33b2-91b3-80f2-8da6b1b5720d","request_id":"239d7713-33b2-91b3-80f2-8da6b1b5720d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9cbf7e4e-32c2-9177-b952-e4cdfaef324b","request_id":"9cbf7e4e-32c2-9177-b952-e4cdfaef324b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-616095cd-decc-952f-9410-6b51110fcf80","request_id":"616095cd-decc-952f-9410-6b51110fcf80"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c407cdeb-b6b3-9a5a-9d87-d9a982cbd651","request_id":"c407cdeb-b6b3-9a5a-9d87-d9a982cbd651"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: OpenAI-compatible response contained provider error: Provider returned error
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: fix-typo{"branches_to_delete": ["fix-typo"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:max__json_schema FAIL 0%
Invalid JSON. Output: fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 102 in → 6 out (0 reasoning)
feature-login
Failure: Missing: ['fix-typo'] Extra: ['feature-login']
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: feature-login{ "branches_to_delete": [ "feature-login" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none FAIL 0% 273 in → 1 out (0 reasoning)
none
Failure: Missing: ['fix-typo'] Extra: ['none']
x-ai/grok-4.3:none__json_schema FAIL 0% 337 in → 6 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['fix-typo']
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 99 in → 2 out (0 reasoning)
feature-login
Failure: Missing: ['fix-typo'] Extra: ['feature-login']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 96 in → 17 out (0 reasoning)
feature-login
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-login" ] }
Failure: Missing: ['fix-typo'] Extra: ['feature-login']