Multiple merged branches
Tests ability to identify multiple merged branches for deletion. Evaluates git branch --merged output parsing with several branches.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'v1' > version.txt
  5. 05 git add version.txt
  6. 06 git commit -m 'Initial version'
  7. 07 git checkout -b fix-a
  8. 08 echo 'fix a' > a.txt
  9. 09 git add a.txt
  10. 10 git commit -m 'Fix A'
  11. 11 git checkout main
  12. 12 git merge fix-a
  13. 13 git checkout -b fix-b
  14. 14 echo 'fix b' > b.txt
  15. 15 git add b.txt
  16. 16 git commit -m 'Fix B'
  17. 17 git checkout main
  18. 18 git merge fix-b
  19. 19 git checkout -b feature-new
  20. 20 echo 'new feature' > new.txt
  21. 21 git add new.txt
  22. 22 git commit -m 'New feature'
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
fix-a fix-b

Scoped model quality, cost, API time, and token usage for branch_cleanup/f002.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high FAIL 100% 156 in → 267 out (40 reasoning)
fix-a fix-b **Reasoning:** The `git branch --merged main` output shows `fix-a`, `fix-b`, and `main` are fully merged. Of these: - **fix-a** — safe to delete (merged, tip commit `4411094` is reachable from main) - **fix-b** — safe to delete (its tip `1e0ffe8` is the same commit as main) - **main** — should *not* be deleted; it's the main branch itself - **feature-new** — should *not* be deleted; it has an unmerged commit (`5948f1c`) and is also the currently checked-out branch (marked with `*`)
Failure: Extra: ['**Reasoning:** The `git branch --merged main` output shows `fix-a`, `fix-b`, and `main` are fully merged. Of these:', '- **feature-new** — should *not* be deleted; it has an unmerged commit (`5948f1c`) and is also the currently checked-out branch (marked with `*`)', '- **fix-a** — safe to delete (merged, tip commit `4411094` is reachable from main)', '- **fix-b** — safe to delete (its tip `1e0ffe8` is the same commit as main)', "- **main** — should *not* be deleted; it's the main branch itself"]
anthropic/claude-fable-5:high__json_schema PASS 100% 396 in → 60 out (24 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
anthropic/claude-fable-5:low PASS 100% 152 in → 23 out (13 reasoning)
fix-a fix-b
anthropic/claude-fable-5:low__json_schema PASS 100% 396 in → 25 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-fable-5:max PASS 100% 157 in → 180 out (43 reasoning)
fix-a fix-b
anthropic/claude-fable-5:max__json_schema PASS 100% 391 in → 183 out (63 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
anthropic/claude-fable-5:medium PASS 100% 154 in → 55 out (38 reasoning)
fix-a fix-b
anthropic/claude-fable-5:medium__json_schema PASS 100% 390 in → 45 out (18 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
anthropic/claude-fable-5:xhigh PASS 100% 154 in → 84 out (59 reasoning)
fix-a fix-b
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 393 in → 109 out (65 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
anthropic/claude-haiku-4.5:high PASS 100% 152 in → 193 out (155 reasoning)
fix-a fix-b
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 346 in → 450 out (375 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
anthropic/claude-haiku-4.5:low PASS 100% 147 in → 195 out (158 reasoning)
fix-a fix-b
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 341 in → 380 out (311 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
anthropic/claude-haiku-4.5:medium PASS 100% 151 in → 214 out (176 reasoning)
fix-a fix-b
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 340 in → 340 out (294 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
anthropic/claude-haiku-4.5:none PASS 100% 127 in → 10 out (0 reasoning)
fix-a fix-b
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 313 in → 21 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 150 in → 269 out (229 reasoning)
fix-a fix-b
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 342 in → 333 out (283 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
anthropic/claude-opus-4.6:high PASS 100% 120 in → 127 out (106 reasoning)
fix-a fix-b
anthropic/claude-opus-4.6:high__json_schema PASS 100% 315 in → 102 out (63 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.6:low PASS 100% 120 in → 10 out (0 reasoning)
fix-a fix-b
anthropic/claude-opus-4.6:low__json_schema PASS 100% 308 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.6:max PASS 100% 121 in → 160 out (140 reasoning)
fix-a fix-b
anthropic/claude-opus-4.6:max__json_schema PASS 100% 307 in → 118 out (88 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.6:medium PASS 100% 121 in → 59 out (32 reasoning)
fix-a fix-b
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 311 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.6:none PASS 100% 122 in → 10 out (0 reasoning)
fix-a fix-b
anthropic/claude-opus-4.6:none__json_schema PASS 100% 313 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 123 in → 160 out (129 reasoning)
fix-a fix-b
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 311 in → 69 out (35 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.7:high PASS 100% 159 in → 13 out (0 reasoning)
fix-a fix-b
anthropic/claude-opus-4.7:high__json_schema PASS 100% 395 in → 27 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.7:low PASS 100% 153 in → 13 out (0 reasoning)
fix-a fix-b
anthropic/claude-opus-4.7:low__json_schema PASS 100% 396 in → 27 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.7:max PASS 100% 159 in → 211 out (43 reasoning)
fix-a fix-b
anthropic/claude-opus-4.7:max__json_schema PASS 100% 391 in → 118 out (49 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 393 in → 27 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.7:none PASS 100% 160 in → 13 out (0 reasoning)
fix-a fix-b
anthropic/claude-opus-4.7:none__json_schema PASS 100% 396 in → 27 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 155 in → 13 out (0 reasoning)
fix-a fix-b
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 395 in → 27 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.8:high PASS 100% 152 in → 94 out (37 reasoning)
fix-a fix-b
anthropic/claude-opus-4.8:high__json_schema PASS 100% 391 in → 25 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.8:low PASS 100% 149 in → 69 out (41 reasoning)
fix-a fix-b
anthropic/claude-opus-4.8:low__json_schema PASS 100% 394 in → 25 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.8:max FAIL 100% 149 in → 926 out (86 reasoning)
Looking at the branches, I need to identify those that are: 1. Fully merged into main, AND 2. Not the `main` branch itself (never delete main) **Analysis of merged branches:** - `fix-a` — merged into main ✓, not main → **delete** - `fix-b` — merged into main ✓, not main (points to same commit as main, `b82480d`) → **delete** - `main` — this is the main branch itself → **keep** - `feature-new` — NOT in the merged list (has unmerged commit `2b251ba`) → **keep** **Branches to delete:** ``` fix-a fix-b ```
Failure: Extra: ['**Analysis of merged branches:**', '**Branches to delete:**', '- `feature-new` — NOT in the merged list (has unmerged commit `2b251ba`) → **keep**', '- `fix-a` — merged into main ✓, not main → **delete**', '- `fix-b` — merged into main ✓, not main (points to same commit as main, `b82480d`) → **delete**', '- `main` — this is the main branch itself → **keep**', '1. Fully merged into main, AND', '2. Not the `main` branch itself (never delete main)', 'Looking at the branches, I need to identify those that are:', '```']
anthropic/claude-opus-4.8:max__json_schema PASS 100% 391 in → 262 out (57 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.8:medium PASS 100% 155 in → 58 out (52 reasoning)
fix-a fix-b
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 392 in → 25 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.8:none FAIL 100% 148 in → 158 out (0 reasoning)
Looking at the branches merged into main: - `fix-a` is listed as merged into main - `fix-b` has the same commit hash (c952494) as main, so it's fully merged However, I need to verify each against the merge list: - `fix-a` → in merged list ✓ - `fix-b` → in merged list ✓ - `feature-new` → NOT in merged list (has unmerged commit 296b501) Branches to delete: ``` fix-a fix-b ```
Failure: Extra: ['- `feature-new` → NOT in merged list (has unmerged commit 296b501)', '- `fix-a` is listed as merged into main', '- `fix-a` → in merged list ✓', "- `fix-b` has the same commit hash (c952494) as main, so it's fully merged", '- `fix-b` → in merged list ✓', 'Branches to delete:', 'However, I need to verify each against the merge list:', 'Looking at the branches merged into main:', '```']
anthropic/claude-opus-4.8:none__json_schema PASS 100% 393 in → 24 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 150 in → 212 out (56 reasoning)
fix-a fix-b
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 390 in → 104 out (51 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-4.6:high PASS 100% 124 in → 110 out (86 reasoning)
fix-a fix-b
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 312 in → 93 out (55 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-4.6:low PASS 100% 121 in → 10 out (0 reasoning)
fix-a fix-b
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 319 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-4.6:max PASS 100% 123 in → 154 out (122 reasoning)
fix-a fix-b
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 311 in → 135 out (103 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 123 in → 10 out (0 reasoning)
fix-a fix-b
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 313 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-4.6:none PASS 100% 122 in → 10 out (0 reasoning)
fix-a fix-b
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 312 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 129 in → 130 out (102 reasoning)
fix-a fix-b
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 313 in → 77 out (37 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-5:high PASS 100% 154 in → 11 out (0 reasoning)
fix-a fix-b
anthropic/claude-sonnet-5:high__json_schema PASS 100% 395 in → 25 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-5:low PASS 100% 154 in → 11 out (0 reasoning)
fix-a fix-b
anthropic/claude-sonnet-5:low__json_schema PASS 100% 394 in → 25 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-5:medium PASS 100% 151 in → 11 out (0 reasoning)
fix-a fix-b
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 389 in → 25 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-5:none PASS 100% 155 in → 11 out (0 reasoning)
fix-a fix-b
anthropic/claude-sonnet-5:none__json_schema PASS 100% 394 in → 24 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 151 in → 139 out (54 reasoning)
fix-a fix-b
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 392 in → 25 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
arcee-ai/trinity-large-thinking:high PASS 100% 114 in → 975 out (970 reasoning)
fix-a fix-b
arcee-ai/trinity-large-thinking:low PASS 100% 111 in → 524 out (516 reasoning)
fix-a fix-b
arcee-ai/trinity-large-thinking:medium PASS 100% 115 in → 1,358 out (1,356 reasoning)
fix-a fix-b
arcee-ai/trinity-large-thinking:xhigh PASS 100% 117 in → 675 out (671 reasoning)
fix-a fix-b
arcee-ai/trinity-mini:high__json_schema PASS 100% 113 in → 460 out (489 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
arcee-ai/trinity-mini:low PASS 100% 115 in → 349 out (375 reasoning)
fix-a fix-b
arcee-ai/trinity-mini:low__json_schema PASS 100% 115 in → 583 out (633 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
arcee-ai/trinity-mini:medium PASS 100% 117 in → 433 out (469 reasoning)
fix-a fix-b
arcee-ai/trinity-mini:medium__json_schema PASS 100% 113 in → 405 out (412 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 113 in → 312 out (330 reasoning)
fix-a fix-b
arcee-ai/trinity-mini:xhigh__json_schema FAIL 100% 111 in → 431 out (446 reasoning)
fix-a fix-b main
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b", "main" ] }
Failure: Extra: ['main']
deepseek/deepseek-v4-flash-0731:high PASS 100% 191 in → 341 out (324 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 187 in → 247 out (225 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 111 in → 60 out (54 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 115 in → 83 out (72 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 205 in → 373 out (367 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 190 in → 365 out (336 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 109 in → 207 out (200 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 113 in → 446 out (423 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 108 in → 6 out (0 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 120 in → 17 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 185 in → 714 out (708 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 115 in → 205 out (181 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
deepseek/deepseek-v4-flash:high PASS 100% 110 in → 105 out (99 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 187 in → 85 out (73 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
deepseek/deepseek-v4-flash:low PASS 100% 108 in → 93 out (86 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 221 in → 16 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
deepseek/deepseek-v4-flash:medium PASS 100% 110 in → 127 out (120 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 114 in → 126 out (102 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
deepseek/deepseek-v4-flash:none PASS 100% 109 in → 5 out (0 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 114 in → 28 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
deepseek/deepseek-v4-flash:xhigh PASS 100% 190 in → 226 out (219 reasoning)
fix-a fix-b
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 189 in → 349 out (336 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
deepseek/deepseek-v4-pro:high PASS 100% 109 in → 432 out (425 reasoning)
fix-a fix-b
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 308 in → 377 out (361 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
deepseek/deepseek-v4-pro:low PASS 100% 115 in → 387 out (380 reasoning)
fix-a fix-b
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 306 in → 77 out (60 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
deepseek/deepseek-v4-pro:medium PASS 100% 107 in → 549 out (542 reasoning)
fix-a fix-b
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 309 in → 314 out (297 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
deepseek/deepseek-v4-pro:none PASS 100% 110 in → 5 out (0 reasoning)
fix-a fix-b
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 109 in → 23 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 192 in → 202 out (195 reasoning)
fix-a fix-b
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 100% 193 in → 716 out (684 reasoning)
feature-new fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-new", "fix-a", "fix-b" ] }
Failure: Extra: ['feature-new']
google/gemini-3-flash-preview:high PASS 100% 118 in → 444 out (437 reasoning)
fix-a fix-b
google/gemini-3-flash-preview:high__json_schema PASS 100% 195 in → 350 out (333 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
google/gemini-3-flash-preview:low PASS 100% 116 in → 313 out (306 reasoning)
fix-a fix-b
google/gemini-3-flash-preview:low__json_schema PASS 100% 195 in → 436 out (421 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3-flash-preview:medium PASS 100% 118 in → 284 out (277 reasoning)
fix-a fix-b
google/gemini-3-flash-preview:medium__json_schema PASS 100% 195 in → 330 out (315 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3-flash-preview:xhigh PASS 100% 117 in → 310 out (303 reasoning)
fix-a fix-b
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 199 in → 353 out (338 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 119 in → 497 out (490 reasoning)
fix-a fix-b
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 115 in → 318 out (301 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 115 in → 119 out (112 reasoning)
fix-a fix-b
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 117 in → 151 out (121 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 112 in → 269 out (262 reasoning)
fix-a fix-b
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 196 in → 307 out (277 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 119 in → 435 out (428 reasoning)
fix-a fix-b
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 199 in → 368 out (347 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
google/gemini-3.1-pro-preview:high PASS 100% 115 in → 360 out (353 reasoning)
fix-a fix-b
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 197 in → 465 out (450 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.1-pro-preview:low PASS 100% 116 in → 250 out (243 reasoning)
fix-a fix-b
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 196 in → 225 out (210 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.1-pro-preview:medium PASS 100% 118 in → 424 out (417 reasoning)
fix-a fix-b
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 201 in → 195 out (180 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 118 in → 363 out (356 reasoning)
fix-a fix-b
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 198 in → 412 out (397 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.5-flash-lite:high PASS 100% 118 in → 733 out (726 reasoning)
fix-a fix-b
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 197 in → 319 out (302 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
google/gemini-3.5-flash-lite:low PASS 100% 117 in → 7 out (0 reasoning)
fix-a fix-b
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 198 in → 17 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
google/gemini-3.5-flash-lite:max PASS 100% 121 in → 415 out (408 reasoning)
fix-a fix-b
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 201 in → 514 out (499 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.5-flash-lite:medium PASS 100% 114 in → 376 out (369 reasoning)
fix-a fix-b
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 200 in → 397 out (380 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 116 in → 454 out (447 reasoning)
fix-a fix-b
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 198 in → 345 out (328 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
google/gemini-3.5-flash:high PASS 100% 115 in → 485 out (478 reasoning)
fix-a fix-b
google/gemini-3.5-flash:high__json_schema PASS 100% 197 in → 510 out (495 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.5-flash:low PASS 100% 118 in → 268 out (261 reasoning)
fix-a fix-b
google/gemini-3.5-flash:low__json_schema PASS 100% 198 in → 233 out (218 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.5-flash:medium PASS 100% 120 in → 433 out (426 reasoning)
fix-a fix-b
google/gemini-3.5-flash:medium__json_schema PASS 100% 199 in → 388 out (371 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
google/gemini-3.5-flash:xhigh PASS 100% 116 in → 530 out (523 reasoning)
fix-a fix-b
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 194 in → 551 out (536 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.6-flash:high PASS 100% 118 in → 398 out (391 reasoning)
fix-a fix-b
google/gemini-3.6-flash:high__json_schema PASS 100% 200 in → 270 out (255 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.6-flash:low PASS 100% 118 in → 158 out (151 reasoning)
fix-a fix-b
google/gemini-3.6-flash:low__json_schema PASS 100% 195 in → 99 out (84 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.6-flash:max PASS 100% 118 in → 347 out (340 reasoning)
fix-a fix-b
google/gemini-3.6-flash:max__json_schema PASS 100% 197 in → 490 out (475 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.6-flash:medium PASS 100% 121 in → 431 out (424 reasoning)
fix-a fix-b
google/gemini-3.6-flash:medium__json_schema PASS 100% 199 in → 381 out (366 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemini-3.6-flash:xhigh PASS 100% 119 in → 414 out (407 reasoning)
fix-a fix-b
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 117 in → 416 out (401 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemma-4-26b-a4b-it:high PASS 100% 130 in → 336 out (291 reasoning)
fix-a fix-b
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 131 in → 371 out (279 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 132 in → 728 out (634 reasoning)
fix-a fix-b
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 135 in → 756 out (582 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 131 in → 333 out (246 reasoning)
fix-a fix-b
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 134 in → 482 out (343 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 133 in → 8 out (0 reasoning)
fix-a fix-b
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 127 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 128 in → 394 out (310 reasoning)
fix-a fix-b
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 128 in → 486 out (361 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
google/gemma-4-31b-it:high PASS 100% 136 in → 308 out (1 reasoning)
fix-a fix-b
google/gemma-4-31b-it:high__json_schema PASS 100% 135 in → 233 out (165 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
google/gemma-4-31b-it:low PASS 100% 133 in → 286 out (1 reasoning)
fix-a fix-b
google/gemma-4-31b-it:low__json_schema PASS 100% 137 in → 264 out (247 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
google/gemma-4-31b-it:medium PASS 100% 135 in → 266 out (210 reasoning)
fix-a fix-b
google/gemma-4-31b-it:medium__json_schema PASS 100% 134 in → 308 out (202 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
google/gemma-4-31b-it:none PASS 100% 132 in → 8 out (0 reasoning)
fix-a fix-b
google/gemma-4-31b-it:none__json_schema PASS 100% 133 in → 31 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
google/gemma-4-31b-it:xhigh PASS 100% 135 in → 241 out (1 reasoning)
fix-a fix-b
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 135 in → 235 out (173 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
ibm-granite/granite-4.1-8b__json_schema PASS 100% 120 in → 24 out
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
inclusionai/ling-2.6-flash PASS 100% 139 in → 6 out
fix-a fix-b
inclusionai/ling-2.6-flash__json_schema PASS 100% 140 in → 28 out
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
minimax/minimax-m2.5:high PASS 100% 142 in → 624 out (672 reasoning)
fix-a fix-b
minimax/minimax-m2.5:high__json_schema PASS 100% 146 in → 540 out (530 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
minimax/minimax-m2.5:low PASS 100% 147 in → 418 out (411 reasoning)
fix-a fix-b
minimax/minimax-m2.5:low__json_schema PASS 100% 145 in → 440 out (422 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
minimax/minimax-m2.5:medium PASS 100% 143 in → 710 out (766 reasoning)
fix-a fix-b
minimax/minimax-m2.5:medium__json_schema PASS 100% 144 in → 444 out (456 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
minimax/minimax-m2.5:xhigh PASS 100% 156 in → 530 out (523 reasoning)
fix-a fix-b
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 141 in → 473 out (474 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
minimax/minimax-m2.7:high PASS 100% 148 in → 228 out (222 reasoning)
fix-a fix-b
minimax/minimax-m2.7:low PASS 100% 149 in → 356 out (350 reasoning)
fix-a fix-b
minimax/minimax-m2.7:low__json_schema PASS 100% 297 in → 666 out (651 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
minimax/minimax-m2.7:medium PASS 100% 128 in → 133 out (122 reasoning)
fix-a fix-b
minimax/minimax-m2.7:medium__json_schema PASS 100% 294 in → 485 out (466 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
minimax/minimax-m2.7:xhigh PASS 100% 144 in → 173 out (167 reasoning)
fix-a fix-b
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 295 in → 570 out (555 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
minimax/minimax-m3:high PASS 100% 282 in → 166 out (181 reasoning)
fix-a fix-b
minimax/minimax-m3:high__json_schema PASS 100% 265 in → 18 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
minimax/minimax-m3:low PASS 100% 282 in → 194 out (184 reasoning)
fix-a fix-b
minimax/minimax-m3:low__json_schema PASS 100% 270 in → 17 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
minimax/minimax-m3:medium PASS 100% 284 in → 260 out (269 reasoning)
fix-a fix-b
minimax/minimax-m3:xhigh PASS 100% 278 in → 150 out (126 reasoning)
fix-a fix-b
minimax/minimax-m3:xhigh__json_schema PASS 100% 268 in → 18 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
mistralai/devstral-2512 PASS 100% 114 in → 6 out
fix-a fix-b
mistralai/devstral-2512__json_schema PASS 100% 120 in → 16 out
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
mistralai/mistral-medium-3-5:high PASS 100% 130 in → 180 out (155 reasoning)
fix-a fix-b
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 122 in → 256 out (235 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
mistralai/mistral-medium-3-5:low PASS 100% 132 in → 258 out (245 reasoning)
fix-a fix-b
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 115 in → 197 out (164 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
mistralai/mistral-medium-3-5:medium PASS 100% 133 in → 174 out (142 reasoning)
fix-a fix-b
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 122 in → 251 out (221 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
mistralai/mistral-medium-3-5:none PASS 100% 132 in → 6 out (0 reasoning)
fix-a fix-b
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 117 in → 16 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 132 in → 464 out (457 reasoning)
fix-a fix-b
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 117 in → 183 out (161 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
moonshotai/kimi-k2.6:high PASS 100% 113 in → 245 out (238 reasoning)
fix-a fix-b
moonshotai/kimi-k2.6:high__json_schema PASS 100% 110 in → 251 out (232 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
moonshotai/kimi-k2.6:low PASS 100% 112 in → 418 out (411 reasoning)
fix-a fix-b
moonshotai/kimi-k2.6:low__json_schema PASS 100% 113 in → 371 out (357 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
moonshotai/kimi-k2.6:medium PASS 100% 108 in → 565 out (568 reasoning)
fix-a fix-b
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 107 in → 429 out (281 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
moonshotai/kimi-k2.6:xhigh PASS 100% 116 in → 335 out (328 reasoning)
fix-a fix-b
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 113 in → 313 out (276 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
moonshotai/kimi-k2.7-code:high PASS 100% 109 in → 88 out (81 reasoning)
fix-a fix-b
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 110 in → 114 out (103 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
moonshotai/kimi-k2.7-code:low PASS 100% 111 in → 232 out (227 reasoning)
fix-a fix-b
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 112 in → 480 out (462 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
moonshotai/kimi-k2.7-code:medium PASS 100% 111 in → 155 out (168 reasoning)
fix-a fix-b
moonshotai/kimi-k2.7-code:xhigh PASS 100% 115 in → 174 out (167 reasoning)
fix-a fix-b
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 106 in → 283 out (155 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
moonshotai/kimi-k3:high PASS 100% 185 in → 182 out (162 reasoning)
fix-a fix-b
moonshotai/kimi-k3:high__json_schema PASS 100% 186 in → 155 out (135 reasoning)
fix-a fix-b
moonshotai/kimi-k3:low PASS 100% 188 in → 208 out (188 reasoning)
fix-a fix-b
moonshotai/kimi-k3:low__json_schema PASS 100% 184 in → 172 out (152 reasoning)
fix-a fix-b
moonshotai/kimi-k3:max PASS 100% 187 in → 114 out (94 reasoning)
fix-a fix-b
moonshotai/kimi-k3:max__json_schema PASS 100% 189 in → 146 out (126 reasoning)
fix-a fix-b
moonshotai/kimi-k3:medium PASS 100% 190 in → 236 out (216 reasoning)
fix-a fix-b
moonshotai/kimi-k3:medium__json_schema PASS 100% 186 in → 162 out (142 reasoning)
fix-a fix-b
moonshotai/kimi-k3:xhigh PASS 100% 192 in → 167 out (147 reasoning)
fix-a fix-b
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 186 in → 175 out (155 reasoning)
fix-a fix-b
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 134 in → 267 out (250 reasoning)
fix-a fix-b
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 127 in → 300 out (280 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 135 in → 119 out (118 reasoning)
fix-a fix-b
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 131 in → 258 out (255 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 127 in → 245 out (235 reasoning)
fix-a fix-b
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 125 in → 304 out (317 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 129 in → 6 out (0 reasoning)
fix-a fix-b
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 130 in → 23 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 130 in → 232 out (228 reasoning)
fix-a fix-b
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 128 in → 201 out (170 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 126 in → 67 out (57 reasoning)
fix-a fix-b
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 135 in → 175 out (144 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 132 in → 204 out (193 reasoning)
fix-a fix-b
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 130 in → 81 out (55 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 128 in → 291 out (274 reasoning)
fix-a fix-b
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 133 in → 293 out (259 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
nvidia/nemotron-3-super-120b-a12b:none FAIL 100% 133 in → 8 out (0 reasoning)
fix-a fix-b main
Failure: Extra: ['main']
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 131 in → 16 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 133 in → 157 out (127 reasoning)
fix-a fix-b
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 134 in → 213 out (176 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
openai/gpt-5.4-mini:high PASS 100% 111 in → 111 out (100 reasoning)
fix-a fix-b
openai/gpt-5.4-mini:high__json_schema PASS 100% 154 in → 109 out (87 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4-mini:low PASS 100% 111 in → 54 out (43 reasoning)
fix-a fix-b
openai/gpt-5.4-mini:low__json_schema PASS 100% 152 in → 70 out (48 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4-mini:medium PASS 100% 113 in → 245 out (234 reasoning)
fix-a fix-b
openai/gpt-5.4-mini:medium__json_schema PASS 100% 152 in → 73 out (51 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4-mini:none PASS 100% 109 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.4-mini:none__json_schema PASS 100% 152 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4-mini:xhigh PASS 100% 107 in → 166 out (155 reasoning)
fix-a fix-b
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 151 in → 1,056 out (1,034 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4-nano:high PASS 100% 112 in → 99 out (88 reasoning)
fix-a fix-b
openai/gpt-5.4-nano:high__json_schema PASS 100% 157 in → 66 out (44 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4-nano:low PASS 100% 105 in → 50 out (39 reasoning)
fix-a fix-b
openai/gpt-5.4-nano:low__json_schema PASS 100% 157 in → 51 out (29 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4-nano:medium PASS 100% 111 in → 83 out (72 reasoning)
fix-a fix-b
openai/gpt-5.4-nano:medium__json_schema PASS 100% 151 in → 76 out (54 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4-nano:none PASS 100% 110 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.4-nano:none__json_schema PASS 100% 154 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4-nano:xhigh PASS 100% 110 in → 168 out (157 reasoning)
fix-a fix-b
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 151 in → 518 out (496 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4:high PASS 100% 106 in → 114 out (103 reasoning)
fix-a fix-b
openai/gpt-5.4:high__json_schema PASS 100% 152 in → 101 out (79 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4:low PASS 100% 110 in → 74 out (63 reasoning)
fix-a fix-b
openai/gpt-5.4:low__json_schema PASS 100% 157 in → 54 out (32 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4:medium PASS 100% 110 in → 111 out (100 reasoning)
fix-a fix-b
openai/gpt-5.4:medium__json_schema PASS 100% 154 in → 90 out (68 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4:none PASS 100% 109 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.4:none__json_schema PASS 100% 152 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.4:xhigh PASS 100% 109 in → 199 out (188 reasoning)
fix-a fix-b
openai/gpt-5.4:xhigh__json_schema PASS 100% 153 in → 178 out (156 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.5:high PASS 100% 110 in → 138 out (127 reasoning)
fix-a fix-b
openai/gpt-5.5:high__json_schema PASS 100% 154 in → 121 out (99 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.5:low PASS 100% 114 in → 75 out (64 reasoning)
fix-a fix-b
openai/gpt-5.5:low__json_schema PASS 100% 151 in → 53 out (31 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.5:medium PASS 100% 108 in → 113 out (102 reasoning)
fix-a fix-b
openai/gpt-5.5:medium__json_schema PASS 100% 153 in → 124 out (102 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.5:none PASS 100% 108 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.5:none__json_schema PASS 100% 151 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.5:xhigh PASS 100% 110 in → 162 out (151 reasoning)
fix-a fix-b
openai/gpt-5.5:xhigh__json_schema PASS 100% 151 in → 114 out (92 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-luna:high PASS 100% 109 in → 90 out (79 reasoning)
fix-a fix-b
openai/gpt-5.6-luna:high__json_schema PASS 100% 148 in → 301 out (279 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-luna:low PASS 100% 109 in → 50 out (39 reasoning)
fix-a fix-b
openai/gpt-5.6-luna:low__json_schema PASS 100% 153 in → 94 out (72 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-luna:max PASS 100% 109 in → 106 out (95 reasoning)
fix-a fix-b
openai/gpt-5.6-luna:max__json_schema PASS 100% 156 in → 284 out (262 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-luna:medium PASS 100% 114 in → 47 out (36 reasoning)
fix-a fix-b
openai/gpt-5.6-luna:medium__json_schema PASS 100% 153 in → 62 out (40 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-luna:none PASS 100% 109 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.6-luna:none__json_schema PASS 100% 154 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-luna:xhigh PASS 100% 110 in → 85 out (74 reasoning)
fix-a fix-b
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 150 in → 391 out (369 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-sol:high PASS 100% 105 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.6-sol:high__json_schema PASS 100% 152 in → 85 out (63 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-sol:low PASS 100% 106 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.6-sol:low__json_schema PASS 100% 151 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-sol:max PASS 100% 110 in → 99 out (88 reasoning)
fix-a fix-b
openai/gpt-5.6-sol:max__json_schema PASS 100% 156 in → 139 out (117 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-sol:medium PASS 100% 111 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.6-sol:medium__json_schema PASS 100% 149 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-sol:none PASS 100% 111 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.6-sol:none__json_schema PASS 100% 157 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-sol:xhigh PASS 100% 110 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 155 in → 102 out (80 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-terra:high PASS 100% 110 in → 39 out (28 reasoning)
fix-a fix-b
openai/gpt-5.6-terra:high__json_schema PASS 100% 153 in → 69 out (47 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-terra:low PASS 100% 108 in → 45 out (34 reasoning)
fix-a fix-b
openai/gpt-5.6-terra:low__json_schema PASS 100% 153 in → 43 out (21 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-terra:max PASS 100% 105 in → 83 out (72 reasoning)
fix-a fix-b
openai/gpt-5.6-terra:max__json_schema PASS 100% 151 in → 80 out (58 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-terra:medium PASS 100% 108 in → 40 out (29 reasoning)
fix-a fix-b
openai/gpt-5.6-terra:medium__json_schema PASS 100% 155 in → 49 out (27 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-terra:none PASS 100% 109 in → 9 out (0 reasoning)
fix-a fix-b
openai/gpt-5.6-terra:none__json_schema PASS 100% 152 in → 20 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-5.6-terra:xhigh PASS 100% 108 in → 33 out (22 reasoning)
fix-a fix-b
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 150 in → 67 out (45 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
openai/gpt-oss-120b:high PASS 100% 157 in → 485 out (496 reasoning)
fix-a fix-b
openai/gpt-oss-120b:high__json_schema PASS 100% 169 in → 22 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
openai/gpt-oss-120b:low PASS 100% 171 in → 164 out (152 reasoning)
fix-a fix-b
openai/gpt-oss-120b:low__json_schema FAIL 100% 157 in → 94 out (57 reasoning)
fix-a fix-b main
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b", "main" ] }
Failure: Extra: ['main']
openai/gpt-oss-120b:medium__json_schema PASS 100% 169 in → 201 out (176 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
openai/gpt-oss-120b:xhigh PASS 100% 176 in → 311 out (324 reasoning)
fix-a fix-b
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 174 in → 288 out (266 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
openai/gpt-oss-20b:high PASS 100% 171 in → 155 out (150 reasoning)
fix-a fix-b
openai/gpt-oss-20b:high__json_schema PASS 100% 164 in → 459 out (458 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
openai/gpt-oss-20b:low PASS 100% 171 in → 76 out (60 reasoning)
fix-a fix-b
openai/gpt-oss-20b:medium PASS 100% 164 in → 192 out (179 reasoning)
fix-a fix-b
openai/gpt-oss-20b:medium__json_schema PASS 100% 173 in → 404 out (380 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
openai/gpt-oss-20b:xhigh PASS 100% 171 in → 673 out (715 reasoning)
fix-a fix-b
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 166 in → 412 out (416 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
poolside/laguna-m.1:high PASS 100% 134 in → 127 out (119 reasoning)
fix-a fix-b
poolside/laguna-m.1:high__json_schema PASS 100% 133 in → 266 out (251 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
poolside/laguna-m.1:low PASS 100% 129 in → 188 out (180 reasoning)
fix-a fix-b
poolside/laguna-m.1:low__json_schema PASS 100% 133 in → 261 out (236 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
poolside/laguna-m.1:medium PASS 100% 133 in → 258 out (250 reasoning)
fix-a fix-b
poolside/laguna-m.1:medium__json_schema PASS 100% 132 in → 322 out (294 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
poolside/laguna-m.1:none PASS 100% 129 in → 7 out (0 reasoning)
fix-a fix-b
poolside/laguna-m.1:none__json_schema PASS 100% 134 in → 18 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
poolside/laguna-m.1:xhigh PASS 100% 136 in → 198 out (190 reasoning)
fix-a fix-b
poolside/laguna-m.1:xhigh__json_schema PASS 100% 132 in → 231 out (212 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
poolside/laguna-xs-2.1:high PASS 100% 133 in → 244 out (238 reasoning)
fix-a fix-b
poolside/laguna-xs-2.1:high__json_schema PASS 100% 135 in → 247 out (232 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
poolside/laguna-xs-2.1:low PASS 100% 133 in → 292 out (286 reasoning)
fix-a fix-b
poolside/laguna-xs-2.1:low__json_schema PASS 100% 133 in → 354 out (339 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
poolside/laguna-xs-2.1:medium PASS 100% 134 in → 601 out (595 reasoning)
fix-a fix-b
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 129 in → 242 out (227 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
poolside/laguna-xs-2.1:none PASS 100% 131 in → 5 out (0 reasoning)
fix-a fix-b
poolside/laguna-xs-2.1:none__json_schema PASS 100% 128 in → 14 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
poolside/laguna-xs-2.1:xhigh PASS 100% 129 in → 249 out (243 reasoning)
fix-a fix-b
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 132 in → 476 out (448 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
poolside/laguna-xs.2:high PASS 100% 168 in → 151 out (143 reasoning)
fix-a fix-b
poolside/laguna-xs.2:high__json_schema PASS 100% 166 in → 306 out (291 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
poolside/laguna-xs.2:low PASS 100% 172 in → 208 out (200 reasoning)
fix-a fix-b
poolside/laguna-xs.2:low__json_schema PASS 100% 170 in → 195 out (180 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
poolside/laguna-xs.2:medium PASS 100% 166 in → 285 out (277 reasoning)
fix-a fix-b
poolside/laguna-xs.2:medium__json_schema PASS 100% 168 in → 230 out (215 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
poolside/laguna-xs.2:none__json_schema PASS 100% 167 in → 14 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
poolside/laguna-xs.2:xhigh PASS 100% 171 in → 286 out (278 reasoning)
fix-a fix-b
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 170 in → 265 out (250 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
qwen/qwen3.6-27b:high PASS 100% 126 in → 861 out (851 reasoning)
fix-a fix-b
qwen/qwen3.6-27b:high__json_schema PASS 100% 129 in → 652 out (573 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
qwen/qwen3.6-27b:low PASS 100% 127 in → 1,090 out (1,009 reasoning)
fix-a fix-b
qwen/qwen3.6-27b:low__json_schema FAIL 100% 130 in → 33 out (0 reasoning)
feature-new fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-new", "fix-a", "fix-b" ] }
Failure: Extra: ['feature-new']
qwen/qwen3.6-27b:medium PASS 100% 127 in → 928 out (836 reasoning)
fix-a fix-b
qwen/qwen3.6-27b:medium__json_schema PASS 100% 127 in → 1,075 out (942 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
qwen/qwen3.6-27b:none__json_schema FAIL 100% 130 in → 33 out (0 reasoning)
feature-new fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-new", "fix-a", "fix-b" ] }
Failure: Extra: ['feature-new']
qwen/qwen3.6-27b:xhigh PASS 100% 124 in → 851 out (801 reasoning)
fix-a fix-b
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 122 in → 790 out (717 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 120 in → 948 out (917 reasoning)
fix-a fix-b
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 125 in → 1,033 out (943 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 122 in → 736 out (671 reasoning)
fix-a fix-b
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 127 in → 879 out (797 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 125 in → 1,153 out (1,056 reasoning)
fix-a fix-b
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 124 in → 743 out (631 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 130 in → 6 out (0 reasoning)
fix-a fix-b
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 129 in → 14 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 126 in → 944 out (874 reasoning)
fix-a fix-b
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 125 in → 986 out (914 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
qwen/qwen3.6-flash:high PASS 100% 125 in → 1,475 out (1,465 reasoning)
fix-a fix-b
qwen/qwen3.6-flash:low PASS 100% 127 in → 1,103 out (1,093 reasoning)
fix-a fix-b
qwen/qwen3.6-flash:medium PASS 100% 127 in → 1,082 out (1,072 reasoning)
fix-a fix-b
qwen/qwen3.6-flash:none FAIL 100% 129 in → 8 out (0 reasoning)
feature-new fix-a fix-b
Failure: Extra: ['feature-new']
qwen/qwen3.6-flash:xhigh PASS 100% 124 in → 1,054 out (1,044 reasoning)
fix-a fix-b
qwen/qwen3.7-flash:high PASS 100% 118 in → 958 out (948 reasoning)
fix-a fix-b
qwen/qwen3.7-flash:low PASS 100% 120 in → 691 out (681 reasoning)
fix-a fix-b
qwen/qwen3.7-flash:max PASS 100% 123 in → 400 out (390 reasoning)
fix-a fix-b
qwen/qwen3.7-flash:medium PASS 100% 122 in → 751 out (741 reasoning)
fix-a fix-b
qwen/qwen3.7-flash:none PASS 100% 127 in → 5 out (0 reasoning)
fix-a fix-b
qwen/qwen3.7-flash:xhigh PASS 100% 124 in → 1,175 out (1,165 reasoning)
fix-a fix-b
qwen/qwen3.7-max:high PASS 100% 126 in → 269 out (259 reasoning)
fix-a fix-b
qwen/qwen3.7-max:low PASS 100% 127 in → 213 out (203 reasoning)
fix-a fix-b
qwen/qwen3.7-max:medium PASS 100% 124 in → 238 out (228 reasoning)
fix-a fix-b
qwen/qwen3.7-max:none PASS 100% 128 in → 5 out (0 reasoning)
fix-a fix-b
qwen/qwen3.7-max:xhigh PASS 100% 126 in → 219 out (209 reasoning)
fix-a fix-b
qwen/qwen3.7-plus:high PASS 100% 125 in → 281 out (271 reasoning)
fix-a fix-b
qwen/qwen3.7-plus:low PASS 100% 130 in → 486 out (476 reasoning)
fix-a fix-b
qwen/qwen3.7-plus:medium PASS 100% 127 in → 202 out (192 reasoning)
fix-a fix-b
qwen/qwen3.7-plus:none PASS 100% 124 in → 5 out (0 reasoning)
fix-a fix-b
qwen/qwen3.7-plus:xhigh PASS 100% 129 in → 358 out (348 reasoning)
fix-a fix-b
tencent/hy3:high PASS 100% 119 in → 579 out (572 reasoning)
fix-a fix-b
tencent/hy3:high__json_schema PASS 100% 119 in → 645 out (608 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
tencent/hy3:low PASS 100% 120 in → 614 out (607 reasoning)
fix-a fix-b
tencent/hy3:low__json_schema PASS 100% 115 in → 374 out (335 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
tencent/hy3:medium PASS 100% 118 in → 425 out (418 reasoning)
fix-a fix-b
tencent/hy3:medium__json_schema PASS 100% 119 in → 638 out (560 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
tencent/hy3:none PASS 100% 122 in → 6 out (0 reasoning)
fix-a fix-b
tencent/hy3:none__json_schema PASS 100% 123 in → 24 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
tencent/hy3:xhigh PASS 100% 115 in → 526 out (519 reasoning)
fix-a fix-b
tencent/hy3:xhigh__json_schema PASS 100% 120 in → 564 out (530 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
thinkingmachines/inkling-small:high PASS 100% 115 in → 140 out (129 reasoning)
fix-a fix-b
thinkingmachines/inkling-small:high__json_schema PASS 100% 119 in → 208 out (182 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
thinkingmachines/inkling-small:low PASS 100% 120 in → 36 out (25 reasoning)
fix-a fix-b
thinkingmachines/inkling-small:low__json_schema PASS 100% 119 in → 60 out (39 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
thinkingmachines/inkling-small:max PASS 100% 124 in → 288 out (277 reasoning)
fix-a fix-b
thinkingmachines/inkling-small:max__json_schema PASS 100% 116 in → 217 out (195 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
thinkingmachines/inkling-small:medium PASS 100% 122 in → 85 out (78 reasoning)
fix-a fix-b
thinkingmachines/inkling-small:medium__json_schema PASS 100% 119 in → 99 out (79 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
thinkingmachines/inkling-small:none PASS 100% 122 in → 8 out (0 reasoning)
fix-a fix-b
thinkingmachines/inkling-small:xhigh PASS 100% 125 in → 210 out (199 reasoning)
fix-a fix-b
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 120 in → 155 out (135 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
thinkingmachines/inkling:high PASS 100% 117 in → 211 out (200 reasoning)
fix-a fix-b
thinkingmachines/inkling:low PASS 100% 122 in → 56 out (52 reasoning)
fix-a fix-b
thinkingmachines/inkling:low__json_schema PASS 100% 121 in → 102 out (82 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
thinkingmachines/inkling:max PASS 100% 118 in → 295 out (284 reasoning)
fix-a fix-b
thinkingmachines/inkling:max__json_schema PASS 100% 119 in → 247 out (221 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
thinkingmachines/inkling:medium PASS 100% 119 in → 101 out (89 reasoning)
fix-a fix-b
thinkingmachines/inkling:none PASS 100% 116 in → 8 out (0 reasoning)
fix-a fix-b
thinkingmachines/inkling:xhigh PASS 100% 122 in → 216 out (201 reasoning)
fix-a fix-b
x-ai/grok-4.3:high PASS 100% 293 in → 643 out (638 reasoning)
fix-a fix-b
x-ai/grok-4.3:high__json_schema PASS 100% 358 in → 786 out (770 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
x-ai/grok-4.3:low PASS 100% 293 in → 444 out (439 reasoning)
fix-a fix-b
x-ai/grok-4.3:low__json_schema PASS 100% 364 in → 391 out (380 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
x-ai/grok-4.3:max PASS 100% 286 in → 512 out (507 reasoning)
fix-a fix-b
x-ai/grok-4.3:max__json_schema PASS 100% 350 in → 683 out (672 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
x-ai/grok-4.3:medium FAIL 100% 292 in → 623 out (480 reasoning)
fix-a fix-b The repository shows four branches with the following states: - `main` is the reference branch (at commit 28ed244). - `fix-b` points to the identical commit as `main`, so it is fully merged. - `fix-a` is explicitly reported under “Merged into main,” confirming its tip commit is an ancestor of `main`. - `feature-new` is not listed among the merged branches and its commit (738cb4e) differs from `main`, so it is not merged. Only `fix-a` and `fix-b` therefore qualify for deletion; `main` must be retained. The list contains exactly those two names, one per line.
Failure: Extra: ['- `feature-new` is not listed among the merged branches and its commit (738cb4e) differs from `main`, so it is not merged.', '- `fix-a` is explicitly reported under “Merged into main,” confirming its tip commit is an ancestor of `main`.', '- `fix-b` points to the identical commit as `main`, so it is fully merged.', '- `main` is the reference branch (at commit 28ed244).', 'Only `fix-a` and `fix-b` therefore qualify for deletion; `main` must be retained. The list contains exactly those two names, one per line.', 'The repository shows four branches with the following states:']
x-ai/grok-4.3:medium__json_schema PASS 100% 359 in → 565 out (544 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
x-ai/grok-4.3:none__json_schema PASS 100% 351 in → 11 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
x-ai/grok-4.3:xhigh PASS 100% 294 in → 310 out (305 reasoning)
fix-a fix-b
x-ai/grok-4.3:xhigh__json_schema PASS 100% 360 in → 952 out (936 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
x-ai/grok-4.5:high PASS 100% 316 in → 450 out (443 reasoning)
fix-a fix-b
x-ai/grok-4.5:high__json_schema PASS 100% 402 in → 412 out (396 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
x-ai/grok-4.5:low PASS 100% 314 in → 337 out (330 reasoning)
fix-a fix-b
x-ai/grok-4.5:low__json_schema PASS 100% 396 in → 343 out (327 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
x-ai/grok-4.5:max PASS 100% 317 in → 548 out (541 reasoning)
fix-a fix-b
x-ai/grok-4.5:max__json_schema PASS 100% 404 in → 493 out (477 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
x-ai/grok-4.5:medium PASS 100% 315 in → 380 out (373 reasoning)
fix-a fix-b
x-ai/grok-4.5:medium__json_schema PASS 100% 398 in → 468 out (452 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
x-ai/grok-4.5:xhigh PASS 100% 315 in → 359 out (352 reasoning)
fix-a fix-b
x-ai/grok-4.5:xhigh__json_schema PASS 100% 399 in → 398 out (382 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["fix-a","fix-b"]}
z-ai/glm-4.7-flash:high PASS 100% 112 in → 483 out (454 reasoning)
fix-a fix-b
z-ai/glm-4.7-flash:high__json_schema PASS 100% 113 in → 2,180 out (2,075 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
z-ai/glm-4.7-flash:low PASS 100% 115 in → 478 out (424 reasoning)
fix-a fix-b
z-ai/glm-4.7-flash:medium PASS 100% 111 in → 529 out (485 reasoning)
fix-a fix-b
z-ai/glm-4.7-flash:none PASS 100% 114 in → 5 out (0 reasoning)
fix-a fix-b
z-ai/glm-4.7-flash:xhigh PASS 100% 114 in → 1,096 out (1,110 reasoning)
fix-a fix-b
z-ai/glm-5.2:high PASS 100% 116 in → 212 out (206 reasoning)
fix-a fix-b
z-ai/glm-5.2:high__json_schema PASS 100% 122 in → 336 out (319 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
z-ai/glm-5.2:low PASS 100% 122 in → 205 out (197 reasoning)
fix-a fix-b
z-ai/glm-5.2:low__json_schema PASS 100% 119 in → 374 out (357 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["fix-a", "fix-b"] }
z-ai/glm-5.2:medium PASS 100% 120 in → 371 out (339 reasoning)
fix-a fix-b
z-ai/glm-5.2:none PASS 100% 111 in → 6 out (0 reasoning)
fix-a fix-b
z-ai/glm-5.2:none__json_schema PASS 100% 114 in → 24 out (0 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "fix-a", "fix-b" ] }
z-ai/glm-5.2:xhigh PASS 100% 117 in → 464 out (457 reasoning)
fix-a fix-b
z-ai/glm-5.2:xhigh__json_schema PASS 100% 302 in → 311 out (296 reasoning)
fix-a fix-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["fix-a", "fix-b"]}
anthropic/claude-opus-4.7:medium FAIL 50% 158 in → 8 out (0 reasoning)
fix-a
Failure: Missing: ['fix-b']
openai/gpt-oss-120b:medium FAIL 50% 179 in → 144 out (141 reasoning)
fix-a fix
Failure: Missing: ['fix-b'] Extra: ['fix']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 109 in → 773 out (826 reasoning)
none
Failure: Missing: ['fix-a', 'fix-b'] Extra: ['none']
ibm-granite/granite-4.1-8b FAIL 0% 125 in → 55 out
feature-new The branch "feature-new" should be deleted because it is fully merged into main. The branches "fix-a", "fix-b", and "main" are already merged (or are the main branch itself), so no further action is needed for them.
Failure: Missing: ['fix-a', 'fix-b'] Extra: ['The branch "feature-new" should be deleted because it is fully merged into main. The branches "fix-a", "fix-b", and "main" are already merged (or are the main branch itself), so no further action is needed for them.', 'feature-new']
liquid/lfm-2-24b-a2b FAIL 0% 118 in → 3 out
feature-new
Failure: Missing: ['fix-a', 'fix-b'] Extra: ['feature-new']
Invalid JSON. Output: feature-new
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:medium__json_schema FAIL 0% 268 in → 10 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['fix-a', 'fix-b']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
openai/gpt-oss-20b:low__json_schema FAIL 0% 173 in → 10 out (0 reasoning)
feature-new
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-new"] }
Failure: Missing: ['fix-a', 'fix-b'] Extra: ['feature-new']
poolside/laguna-xs.2:none FAIL 0% 173 in → 58 out (0 reasoning)
Looking at the branches and the merge information, I can see that `fix-a` and `fix-b` are both listed as merged into main. feature-new The branch `feature-new` is not listed as merged into main, so it should not be deleted.
Failure: Missing: ['fix-a', 'fix-b'] Extra: ['Looking at the branches and the merge information, I can see that `fix-a` and `fix-b` are both listed as merged into main.', 'The branch `feature-new` is not listed as merged into main, so it should not be deleted.', 'feature-new']
qwen/qwen3.6-27b:none FAIL 0% 130 in → 3 out (0 reasoning)
feature-new
Failure: Missing: ['fix-a', 'fix-b'] Extra: ['feature-new']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c31ffa0c-56d3-9379-a6f6-a658c6c4eb9e","request_id":"c31ffa0c-56d3-9379-a6f6-a658c6c4eb9e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4d3d810d-be09-9e62-a6c0-6093c18cee64","request_id":"4d3d810d-be09-9e62-a6c0-6093c18cee64"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-845fd2f7-de8f-9a00-8f33-d710cae67e83","request_id":"845fd2f7-de8f-9a00-8f33-d710cae67e83"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-df0be10d-7c44-91ed-b0ed-dd130bbecb57","request_id":"df0be10d-7c44-91ed-b0ed-dd130bbecb57"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a410d0d7-1dfb-9bab-91e4-ac59e82800ef","request_id":"a410d0d7-1dfb-9bab-91e4-ac59e82800ef"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-62a67518-eb94-9e88-9175-a6ffbbdff19e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-02fe8c25-f20c-9cb5-a4d1-597b1cb2040c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e418c5c8-19f5-96a2-b063-00334842afc6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d31c91eb-b6d6-92d1-99ce-53c96549b815"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-853c90da-fe5c-90f2-8144-2186e0bbb3c8"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-31a9246c-eaec-9f38-b06d-986bf2056e8d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f46dd611-aeb9-97eb-897c-70ce785069fd","request_id":"f46dd611-aeb9-97eb-897c-70ce785069fd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c413c68d-4737-9993-9fc7-9bd67b244b30","request_id":"c413c68d-4737-9993-9fc7-9bd67b244b30"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bb37a5fe-689e-9091-98fe-d7f0735ece08","request_id":"bb37a5fe-689e-9091-98fe-d7f0735ece08"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6ac49a67-1303-9b1e-ad8e-f50ad1f7f5df","request_id":"6ac49a67-1303-9b1e-ad8e-f50ad1f7f5df"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-406d1bce-638f-9b3a-b7b3-ba05dd0a8994","request_id":"406d1bce-638f-9b3a-b7b3-ba05dd0a8994"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6bfab9b0-c112-9cfc-8e58-a160c4868157","request_id":"6bfab9b0-c112-9cfc-8e58-a160c4868157"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3f9b3af8-b84e-932e-bafb-42c5ef423eb6","request_id":"3f9b3af8-b84e-932e-bafb-42c5ef423eb6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c9f524c1-91c0-9767-8765-ea10d7c00b18","request_id":"c9f524c1-91c0-9767-8765-ea10d7c00b18"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6c9b3564-9c25-9e4e-9dd5-5bf6c8e39aa7","request_id":"6c9b3564-9c25-9e4e-9dd5-5bf6c8e39aa7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8847a664-e6f1-9588-ab15-2cb14eaabe56","request_id":"8847a664-e6f1-9588-ab15-2cb14eaabe56"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: fix-a fix-b{ "branches_to_delete": ["fix-a", "fix-b"] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: fix-a fix-b
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: fix-a fix-b
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: fix-a fix-b{ "branches_to_delete": [ "fix-a", "fix-b" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: fix-a fix-b
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none FAIL 0% 288 in → 1 out (0 reasoning)
none
Failure: Missing: ['fix-a', 'fix-b'] Extra: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 114 in → 15 out (0 reasoning)
feature-new
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-new"] }
Failure: Missing: ['fix-a', 'fix-b'] Extra: ['feature-new']
Invalid JSON. Output: fix-a fix-b
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: fix-a fix-b
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)