All branches merged
Tests ability to identify that all branches are merged and can be deleted. Evaluates the full-cleanup scenario.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'start' > start.txt
  5. 05 git add start.txt
  6. 06 git commit -m 'Start'
  7. 07 git checkout -b fix-1
  8. 08 echo 'fix1' > fix1.txt
  9. 09 git add fix1.txt
  10. 10 git commit -m 'Fix 1'
  11. 11 git checkout main
  12. 12 git merge fix-1
  13. 13 git checkout -b fix-2
  14. 14 echo 'fix2' > fix2.txt
  15. 15 git add fix2.txt
  16. 16 git commit -m 'Fix 2'
  17. 17 git checkout main
  18. 18 git merge fix-2
  19. 19 git checkout -b cleanup
  20. 20 echo 'cleanup' > cleanup.txt
  21. 21 git add cleanup.txt
  22. 22 git commit -m 'Cleanup'
  23. 23 git checkout main
  24. 24 git merge cleanup
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
cleanup fix-1 fix-2

Scoped model quality, cost, API time, and token usage for branch_cleanup/f008.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high FAIL 100% 164 in → 208 out (51 reasoning)
cleanup fix-1 fix-2 All three branches appear in the `git branch --merged main` output, meaning their commits are fully contained in main's history: - **cleanup** (7daa55c) — points to the exact same commit as main - **fix-1** (7a02301) — merged into main's history - **fix-2** (81e3bc5) — merged into main's history They can be safely deleted with `git branch -d cleanup fix-1 fix-2`.
Failure: Extra: ['- **cleanup** (7daa55c) — points to the exact same commit as main', "- **fix-1** (7a02301) — merged into main's history", "- **fix-2** (81e3bc5) — merged into main's history", "All three branches appear in the `git branch --merged main` output, meaning their commits are fully contained in main's history:", 'They can be safely deleted with `git branch -d cleanup fix-1 fix-2`.']
anthropic/claude-fable-5:high__json_schema PASS 100% 407 in → 87 out (44 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
anthropic/claude-fable-5:low FAIL 100% 157 in → 118 out (12 reasoning)
``` cleanup fix-1 fix-2 ``` These three branches are fully merged into main (shown in the "Merged into main" list), so they can be safely deleted. Note that `cleanup` even points to the same commit as main (556852d). The `main` branch itself is excluded since it's the current branch and the merge target.
Failure: Extra: ['These three branches are fully merged into main (shown in the "Merged into main" list), so they can be safely deleted. Note that `cleanup` even points to the same commit as main (556852d). The `main` branch itself is excluded since it\'s the current branch and the merge target.', '```']
anthropic/claude-fable-5:low__json_schema PASS 100% 407 in → 33 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
anthropic/claude-fable-5:max PASS 100% 166 in → 285 out (72 reasoning)
cleanup fix-1 fix-2
anthropic/claude-fable-5:max__json_schema PASS 100% 404 in → 209 out (51 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
anthropic/claude-fable-5:medium PASS 100% 164 in → 60 out (37 reasoning)
cleanup fix-1 fix-2
anthropic/claude-fable-5:medium__json_schema PASS 100% 404 in → 43 out (18 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
anthropic/claude-fable-5:xhigh PASS 100% 167 in → 86 out (46 reasoning)
cleanup fix-1 fix-2
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 399 in → 64 out (15 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
anthropic/claude-haiku-4.5:high PASS 100% 155 in → 173 out (147 reasoning)
cleanup fix-1 fix-2
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 347 in → 221 out (183 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
anthropic/claude-haiku-4.5:low PASS 100% 152 in → 214 out (178 reasoning)
cleanup fix-1 fix-2
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 346 in → 325 out (286 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
anthropic/claude-haiku-4.5:medium PASS 100% 159 in → 148 out (126 reasoning)
cleanup fix-1 fix-2
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 343 in → 390 out (329 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
anthropic/claude-haiku-4.5:none PASS 100% 125 in → 12 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 312 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 150 in → 189 out (152 reasoning)
cleanup fix-1 fix-2
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 345 in → 215 out (190 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
anthropic/claude-opus-4.6:high PASS 100% 128 in → 148 out (129 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.6:high__json_schema PASS 100% 316 in → 72 out (36 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.6:low PASS 100% 125 in → 12 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.6:low__json_schema PASS 100% 311 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.6:max PASS 100% 123 in → 109 out (84 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.6:max__json_schema PASS 100% 317 in → 127 out (88 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.6:medium PASS 100% 126 in → 94 out (62 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 317 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.6:none PASS 100% 126 in → 12 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.6:none__json_schema PASS 100% 313 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 121 in → 103 out (79 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 314 in → 66 out (28 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.7:high PASS 100% 169 in → 18 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.7:high__json_schema PASS 100% 405 in → 32 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.7:low PASS 100% 169 in → 18 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.7:low__json_schema PASS 100% 409 in → 32 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.7:max PASS 100% 170 in → 270 out (71 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.7:max__json_schema PASS 100% 414 in → 166 out (67 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.7:medium PASS 100% 170 in → 18 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 409 in → 32 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.7:none PASS 100% 166 in → 18 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.7:none__json_schema PASS 100% 407 in → 32 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 170 in → 18 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 408 in → 32 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.8:high PASS 100% 168 in → 446 out (61 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.8:high__json_schema PASS 100% 409 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.8:low PASS 100% 161 in → 16 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.8:low__json_schema PASS 100% 402 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.8:max FAIL 100% 167 in → 2,053 out (225 reasoning)
Based on the `git branch --merged main` output, the following branches are fully merged into main and safe to delete (excluding `main` itself): cleanup fix-1 fix-2 **Reasoning:** - `cleanup` points to the exact same commit as `main` (`6ab8499`), so it's trivially merged. - `fix-1` and `fix-2` both appear in the "Merged into main" list, meaning their commits are already reachable from `main`—all their work is included. - `main` is excluded because it's the primary branch (and the currently checked-out branch), never a deletion candidate.
Failure: Extra: ['**Reasoning:**', "- `cleanup` points to the exact same commit as `main` (`6ab8499`), so it's trivially merged.", '- `fix-1` and `fix-2` both appear in the "Merged into main" list, meaning their commits are already reachable from `main`—all their work is included.', "- `main` is excluded because it's the primary branch (and the currently checked-out branch), never a deletion candidate.", 'Based on the `git branch --merged main` output, the following branches are fully merged into main and safe to delete (excluding `main` itself):']
anthropic/claude-opus-4.8:max__json_schema PASS 100% 408 in → 317 out (56 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.8:medium PASS 100% 166 in → 125 out (58 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 401 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.8:none PASS 100% 164 in → 16 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.8:none__json_schema PASS 100% 401 in → 29 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 159 in → 353 out (71 reasoning)
cleanup fix-1 fix-2
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 402 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-4.6:high PASS 100% 125 in → 99 out (72 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 311 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-4.6:low PASS 100% 127 in → 12 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 318 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-4.6:max PASS 100% 128 in → 113 out (86 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 311 in → 120 out (88 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 126 in → 12 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 316 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-4.6:none PASS 100% 125 in → 12 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 316 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 127 in → 89 out (66 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 313 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-5:high PASS 100% 165 in → 100 out (42 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-5:high__json_schema PASS 100% 405 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-5:low PASS 100% 164 in → 16 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-5:low__json_schema PASS 100% 408 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-5:medium PASS 100% 161 in → 16 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 403 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-5:none PASS 100% 158 in → 16 out (0 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-5:none__json_schema PASS 100% 401 in → 29 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 163 in → 110 out (49 reasoning)
cleanup fix-1 fix-2
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 405 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
arcee-ai/trinity-large-thinking:high PASS 100% 112 in → 393 out (387 reasoning)
cleanup fix-1 fix-2
arcee-ai/trinity-large-thinking:low PASS 100% 118 in → 682 out (681 reasoning)
cleanup fix-1 fix-2
arcee-ai/trinity-large-thinking:medium PASS 100% 114 in → 335 out (325 reasoning)
cleanup fix-1 fix-2
arcee-ai/trinity-large-thinking:xhigh PASS 100% 117 in → 544 out (543 reasoning)
cleanup fix-1 fix-2
arcee-ai/trinity-mini:high__json_schema PASS 100% 115 in → 414 out (416 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
arcee-ai/trinity-mini:low PASS 100% 117 in → 261 out (262 reasoning)
cleanup fix-1 fix-2
arcee-ai/trinity-mini:low__json_schema PASS 100% 119 in → 282 out (271 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete":[ "cleanup", "fix-1", "fix-2" ] }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 120 in → 278 out (266 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 116 in → 262 out (266 reasoning)
cleanup fix-1 fix-2
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 115 in → 281 out (270 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 196 in → 235 out (210 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 196 in → 180 out (155 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 118 in → 159 out (148 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 118 in → 60 out (49 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
deepseek/deepseek-v4-flash-0731:max PASS 100% 193 in → 350 out (327 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 197 in → 21 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 114 in → 41 out (37 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 117 in → 378 out (318 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 119 in → 11 out (0 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 115 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 119 in → 259 out (232 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 124 in → 177 out (152 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
deepseek/deepseek-v4-flash:high PASS 100% 119 in → 198 out (186 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 224 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
deepseek/deepseek-v4-flash:low PASS 100% 118 in → 83 out (65 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash:medium PASS 100% 117 in → 231 out (218 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 199 in → 219 out (200 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
deepseek/deepseek-v4-flash:none PASS 100% 116 in → 11 out (0 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 120 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
deepseek/deepseek-v4-flash:xhigh PASS 100% 198 in → 244 out (232 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 227 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
deepseek/deepseek-v4-pro:high PASS 100% 117 in → 420 out (409 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 121 in → 420 out (389 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
deepseek/deepseek-v4-pro:low PASS 100% 119 in → 326 out (314 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 318 in → 354 out (331 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
deepseek/deepseek-v4-pro:medium PASS 100% 117 in → 298 out (287 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 314 in → 108 out (85 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
deepseek/deepseek-v4-pro:none PASS 100% 118 in → 10 out (0 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 120 in → 29 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 202 in → 317 out (305 reasoning)
cleanup fix-1 fix-2
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 397 in → 593 out (570 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
google/gemini-3-flash-preview:high PASS 100% 119 in → 326 out (317 reasoning)
cleanup fix-1 fix-2
google/gemini-3-flash-preview:high__json_schema PASS 100% 200 in → 339 out (322 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3-flash-preview:low PASS 100% 121 in → 341 out (332 reasoning)
cleanup fix-1 fix-2
google/gemini-3-flash-preview:low__json_schema PASS 100% 120 in → 328 out (311 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3-flash-preview:medium PASS 100% 118 in → 414 out (405 reasoning)
cleanup fix-1 fix-2
google/gemini-3-flash-preview:medium__json_schema PASS 100% 197 in → 370 out (353 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3-flash-preview:xhigh PASS 100% 116 in → 254 out (245 reasoning)
cleanup fix-1 fix-2
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 200 in → 389 out (373 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 116 in → 464 out (455 reasoning)
cleanup fix-1 fix-2
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 119 in → 334 out (314 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 117 in → 129 out (120 reasoning)
cleanup fix-1 fix-2
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 119 in → 143 out (108 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 122 in → 214 out (205 reasoning)
cleanup fix-1 fix-2
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 201 in → 309 out (274 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 119 in → 635 out (626 reasoning)
cleanup fix-1 fix-2
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 119 in → 446 out (429 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.1-pro-preview:high PASS 100% 113 in → 304 out (295 reasoning)
cleanup fix-1 fix-2
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 196 in → 473 out (456 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.1-pro-preview:low PASS 100% 119 in → 204 out (195 reasoning)
cleanup fix-1 fix-2
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 196 in → 277 out (260 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.1-pro-preview:medium PASS 100% 115 in → 456 out (447 reasoning)
cleanup fix-1 fix-2
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 202 in → 454 out (437 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 121 in → 454 out (445 reasoning)
cleanup fix-1 fix-2
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 197 in → 317 out (300 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.5-flash-lite:high PASS 100% 120 in → 486 out (477 reasoning)
cleanup fix-1 fix-2
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 199 in → 431 out (414 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.5-flash-lite:low PASS 100% 117 in → 9 out (0 reasoning)
cleanup fix-1 fix-2
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 117 in → 35 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
google/gemini-3.5-flash-lite:max PASS 100% 117 in → 425 out (416 reasoning)
cleanup fix-1 fix-2
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 196 in → 538 out (521 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.5-flash-lite:medium PASS 100% 115 in → 379 out (370 reasoning)
cleanup fix-1 fix-2
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 201 in → 329 out (294 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
google/gemini-3.5-flash-lite:xhigh PASS 100% 120 in → 402 out (393 reasoning)
cleanup fix-1 fix-2
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 195 in → 323 out (303 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
google/gemini-3.5-flash:high PASS 100% 119 in → 480 out (471 reasoning)
cleanup fix-1 fix-2
google/gemini-3.5-flash:high__json_schema PASS 100% 199 in → 451 out (434 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.5-flash:low PASS 100% 120 in → 309 out (300 reasoning)
cleanup fix-1 fix-2
google/gemini-3.5-flash:low__json_schema PASS 100% 197 in → 208 out (191 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.5-flash:medium PASS 100% 116 in → 510 out (501 reasoning)
cleanup fix-1 fix-2
google/gemini-3.5-flash:medium__json_schema PASS 100% 199 in → 232 out (215 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.5-flash:xhigh PASS 100% 120 in → 495 out (486 reasoning)
cleanup fix-1 fix-2
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 197 in → 436 out (419 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.6-flash:high PASS 100% 116 in → 451 out (442 reasoning)
cleanup fix-1 fix-2
google/gemini-3.6-flash:high__json_schema PASS 100% 114 in → 315 out (298 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.6-flash:low PASS 100% 121 in → 147 out (138 reasoning)
cleanup fix-1 fix-2
google/gemini-3.6-flash:low__json_schema PASS 100% 119 in → 35 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
google/gemini-3.6-flash:max PASS 100% 118 in → 477 out (468 reasoning)
cleanup fix-1 fix-2
google/gemini-3.6-flash:max__json_schema PASS 100% 117 in → 367 out (350 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.6-flash:medium PASS 100% 120 in → 383 out (374 reasoning)
cleanup fix-1 fix-2
google/gemini-3.6-flash:medium__json_schema PASS 100% 199 in → 312 out (295 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemini-3.6-flash:xhigh PASS 100% 118 in → 403 out (394 reasoning)
cleanup fix-1 fix-2
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 199 in → 424 out (407 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemma-4-26b-a4b-it:high PASS 100% 134 in → 325 out (250 reasoning)
cleanup fix-1 fix-2
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 131 in → 369 out (271 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 135 in → 350 out (270 reasoning)
cleanup fix-1 fix-2
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 135 in → 381 out (294 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
google/gemma-4-26b-a4b-it:medium PASS 100% 132 in → 346 out (269 reasoning)
cleanup fix-1 fix-2
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 130 in → 322 out (221 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 134 in → 10 out (0 reasoning)
cleanup fix-1 fix-2
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 129 in → 36 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 135 in → 391 out (302 reasoning)
cleanup fix-1 fix-2
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 135 in → 395 out (304 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
google/gemma-4-31b-it:high PASS 100% 134 in → 214 out (1 reasoning)
cleanup fix-1 fix-2
google/gemma-4-31b-it:high__json_schema PASS 100% 129 in → 292 out (215 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
google/gemma-4-31b-it:low PASS 100% 133 in → 198 out (152 reasoning)
cleanup fix-1 fix-2
google/gemma-4-31b-it:medium PASS 100% 135 in → 229 out (183 reasoning)
cleanup fix-1 fix-2
google/gemma-4-31b-it:medium__json_schema PASS 100% 138 in → 259 out (240 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemma-4-31b-it:none PASS 100% 134 in → 10 out (0 reasoning)
cleanup fix-1 fix-2
google/gemma-4-31b-it:none__json_schema PASS 100% 134 in → 19 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
google/gemma-4-31b-it:xhigh PASS 100% 129 in → 211 out (1 reasoning)
cleanup fix-1 fix-2
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 133 in → 316 out (219 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
ibm-granite/granite-4.1-8b PASS 100% 123 in → 10 out
cleanup fix-1 fix-2
ibm-granite/granite-4.1-8b__json_schema PASS 100% 123 in → 30 out
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
inclusionai/ling-2.6-flash PASS 100% 147 in → 10 out
cleanup fix-1 fix-2
inclusionai/ling-2.6-flash__json_schema PASS 100% 148 in → 35 out
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
liquid/lfm-2-24b-a2b PASS 100% 130 in → 11 out
cleanup fix-1 fix-2
minimax/minimax-m2.5:high PASS 100% 149 in → 502 out (491 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m2.5:high__json_schema PASS 100% 147 in → 436 out (399 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
minimax/minimax-m2.5:low PASS 100% 159 in → 369 out (358 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m2.5:low__json_schema PASS 100% 145 in → 416 out (410 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
minimax/minimax-m2.5:medium PASS 100% 151 in → 582 out (571 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m2.5:medium__json_schema PASS 100% 147 in → 628 out (636 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
minimax/minimax-m2.5:xhigh PASS 100% 149 in → 666 out (692 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 154 in → 488 out (458 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
minimax/minimax-m2.7:high PASS 100% 148 in → 179 out (159 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m2.7:high__json_schema PASS 100% 152 in → 21 out (21 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
minimax/minimax-m2.7:low PASS 100% 150 in → 414 out (404 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m2.7:low__json_schema PASS 100% 299 in → 132 out (113 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
minimax/minimax-m2.7:medium PASS 100% 147 in → 287 out (278 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m2.7:medium__json_schema PASS 100% 301 in → 463 out (439 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
minimax/minimax-m2.7:xhigh PASS 100% 146 in → 263 out (252 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 301 in → 462 out (438 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
minimax/minimax-m3:high PASS 100% 290 in → 95 out (93 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m3:high__json_schema PASS 100% 238 in → 23 out (22 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
minimax/minimax-m3:low PASS 100% 283 in → 152 out (144 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m3:low__json_schema PASS 100% 277 in → 23 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
minimax/minimax-m3:medium PASS 100% 287 in → 168 out (155 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m3:medium__json_schema PASS 100% 275 in → 29 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
minimax/minimax-m3:xhigh PASS 100% 291 in → 154 out (128 reasoning)
cleanup fix-1 fix-2
minimax/minimax-m3:xhigh__json_schema PASS 100% 271 in → 29 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
mistralai/devstral-2512 PASS 100% 123 in → 11 out
cleanup fix-1 fix-2
mistralai/devstral-2512__json_schema PASS 100% 128 in → 22 out
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
mistralai/mistral-medium-3-5:high PASS 100% 141 in → 346 out (288 reasoning)
cleanup fix-1 fix-2
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 126 in → 230 out (183 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
mistralai/mistral-medium-3-5:low PASS 100% 135 in → 554 out (569 reasoning)
cleanup fix-1 fix-2
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 122 in → 315 out (272 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
mistralai/mistral-medium-3-5:medium PASS 100% 136 in → 338 out (318 reasoning)
cleanup fix-1 fix-2
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 128 in → 224 out (177 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
mistralai/mistral-medium-3-5:none PASS 100% 137 in → 11 out (0 reasoning)
cleanup fix-1 fix-2
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 128 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 136 in → 248 out (238 reasoning)
cleanup fix-1 fix-2
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 125 in → 198 out (180 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
moonshotai/kimi-k2.6:high PASS 100% 121 in → 292 out (272 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k2.6:high__json_schema PASS 100% 115 in → 598 out (577 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
moonshotai/kimi-k2.6:low PASS 100% 119 in → 472 out (461 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k2.6:low__json_schema PASS 100% 113 in → 590 out (279 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
moonshotai/kimi-k2.6:medium PASS 100% 120 in → 396 out (391 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 312 in → 426 out (404 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
moonshotai/kimi-k2.6:xhigh PASS 100% 117 in → 578 out (518 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 120 in → 851 out (821 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
moonshotai/kimi-k2.7-code:high PASS 100% 114 in → 121 out (110 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 115 in → 300 out (128 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
moonshotai/kimi-k2.7-code:low PASS 100% 117 in → 72 out (61 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 213 in → 122 out (103 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
moonshotai/kimi-k2.7-code:medium PASS 100% 117 in → 162 out (151 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 117 in → 117 out (91 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 120 in → 82 out (68 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 118 in → 178 out (148 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
moonshotai/kimi-k3:high PASS 100% 195 in → 177 out (153 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k3:high__json_schema PASS 100% 197 in → 104 out (80 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k3:low PASS 100% 197 in → 172 out (148 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k3:low__json_schema PASS 100% 196 in → 379 out (355 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k3:max PASS 100% 194 in → 172 out (148 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k3:max__json_schema PASS 100% 300 in → 157 out (125 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
moonshotai/kimi-k3:medium PASS 100% 193 in → 153 out (129 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k3:medium__json_schema PASS 100% 303 in → 115 out (83 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
moonshotai/kimi-k3:xhigh PASS 100% 198 in → 186 out (162 reasoning)
cleanup fix-1 fix-2
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 197 in → 168 out (144 reasoning)
cleanup fix-1 fix-2
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 137 in → 117 out (109 reasoning)
cleanup fix-1 fix-2
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 140 in → 143 out (115 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 136 in → 233 out (203 reasoning)
cleanup fix-1 fix-2
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 140 in → 131 out (104 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 138 in → 210 out (184 reasoning)
cleanup fix-1 fix-2
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 140 in → 125 out (89 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 141 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 137 in → 145 out (133 reasoning)
cleanup fix-1 fix-2
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 140 in → 135 out (102 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 139 in → 73 out (59 reasoning)
cleanup fix-1 fix-2
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 138 in → 89 out (57 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 142 in → 73 out (59 reasoning)
cleanup fix-1 fix-2
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 138 in → 133 out (100 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 140 in → 253 out (220 reasoning)
cleanup fix-1 fix-2
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 140 in → 124 out (92 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 136 in → 11 out (0 reasoning)
cleanup fix-1 fix-2
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 142 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 136 in → 84 out (63 reasoning)
cleanup fix-1 fix-2
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 138 in → 155 out (125 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
openai/gpt-5.4-mini:high PASS 100% 115 in → 95 out (80 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4-mini:high__json_schema PASS 100% 155 in → 120 out (94 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4-mini:low PASS 100% 117 in → 30 out (15 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4-mini:low__json_schema PASS 100% 164 in → 101 out (75 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4-mini:medium PASS 100% 118 in → 55 out (40 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4-mini:medium__json_schema PASS 100% 160 in → 161 out (135 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4-mini:none PASS 100% 113 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4-mini:none__json_schema PASS 100% 156 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4-mini:xhigh PASS 100% 116 in → 157 out (142 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 155 in → 296 out (270 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4-nano:high PASS 100% 118 in → 48 out (33 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4-nano:high__json_schema PASS 100% 156 in → 73 out (47 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4-nano:low PASS 100% 113 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4-nano:low__json_schema PASS 100% 154 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4-nano:medium PASS 100% 116 in → 45 out (30 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4-nano:medium__json_schema PASS 100% 159 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4-nano:none PASS 100% 118 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4-nano:none__json_schema PASS 100% 157 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4-nano:xhigh PASS 100% 116 in → 132 out (117 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 153 in → 242 out (216 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4:high PASS 100% 117 in → 96 out (81 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4:high__json_schema PASS 100% 157 in → 84 out (58 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4:low PASS 100% 115 in → 50 out (35 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4:low__json_schema PASS 100% 160 in → 35 out (9 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4:medium PASS 100% 117 in → 66 out (51 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4:medium__json_schema PASS 100% 161 in → 92 out (66 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4:none PASS 100% 116 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4:none__json_schema PASS 100% 159 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.4:xhigh PASS 100% 116 in → 78 out (63 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.4:xhigh__json_schema PASS 100% 162 in → 115 out (89 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.5:high PASS 100% 114 in → 128 out (113 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.5:high__json_schema PASS 100% 160 in → 172 out (146 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.5:low PASS 100% 115 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.5:low__json_schema PASS 100% 157 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.5:medium PASS 100% 116 in → 79 out (64 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.5:medium__json_schema PASS 100% 158 in → 84 out (58 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.5:none PASS 100% 116 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.5:none__json_schema PASS 100% 158 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.5:xhigh PASS 100% 118 in → 145 out (130 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.5:xhigh__json_schema PASS 100% 156 in → 158 out (132 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-luna:high PASS 100% 115 in → 59 out (44 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-luna:high__json_schema PASS 100% 162 in → 112 out (86 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-luna:low PASS 100% 115 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-luna:low__json_schema PASS 100% 160 in → 49 out (23 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-luna:max PASS 100% 117 in → 84 out (69 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-luna:max__json_schema PASS 100% 161 in → 129 out (103 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-luna:medium PASS 100% 117 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-luna:medium__json_schema PASS 100% 159 in → 57 out (31 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-luna:none PASS 100% 117 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-luna:none__json_schema PASS 100% 157 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-luna:xhigh PASS 100% 118 in → 57 out (42 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 158 in → 252 out (226 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-sol:high PASS 100% 119 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-sol:high__json_schema PASS 100% 156 in → 77 out (51 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-sol:low PASS 100% 114 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-sol:low__json_schema PASS 100% 160 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-sol:max PASS 100% 115 in → 61 out (46 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-sol:max__json_schema PASS 100% 158 in → 125 out (99 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-sol:medium PASS 100% 114 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-sol:medium__json_schema PASS 100% 159 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-sol:none PASS 100% 120 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-sol:none__json_schema PASS 100% 156 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-sol:xhigh PASS 100% 113 in → 50 out (35 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 157 in → 92 out (66 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-terra:high PASS 100% 122 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-terra:high__json_schema PASS 100% 165 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-terra:low PASS 100% 118 in → 36 out (21 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-terra:low__json_schema PASS 100% 159 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-terra:max PASS 100% 114 in → 68 out (53 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-terra:max__json_schema PASS 100% 160 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-terra:medium PASS 100% 115 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-terra:medium__json_schema PASS 100% 160 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-terra:none PASS 100% 120 in → 13 out (0 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-terra:none__json_schema PASS 100% 155 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-5.6-terra:xhigh PASS 100% 120 in → 39 out (24 reasoning)
cleanup fix-1 fix-2
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 157 in → 24 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-oss-120b:high PASS 100% 175 in → 220 out (201 reasoning)
cleanup fix-1 fix-2
openai/gpt-oss-120b:high__json_schema PASS 100% 176 in → 17 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
openai/gpt-oss-120b:low PASS 100% 181 in → 95 out (75 reasoning)
cleanup fix-1 fix-2
openai/gpt-oss-120b:low__json_schema PASS 100% 177 in → 28 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
openai/gpt-oss-120b:medium PASS 100% 174 in → 104 out (95 reasoning)
cleanup fix-1 fix-2
openai/gpt-oss-120b:medium__json_schema PASS 100% 177 in → 163 out (127 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
openai/gpt-oss-120b:xhigh PASS 100% 177 in → 223 out (195 reasoning)
cleanup fix-1 fix-2
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 177 in → 474 out (449 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
openai/gpt-oss-20b:high PASS 100% 175 in → 397 out (385 reasoning)
cleanup fix-1 fix-2
openai/gpt-oss-20b:high__json_schema PASS 100% 175 in → 308 out (272 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
openai/gpt-oss-20b:low PASS 100% 172 in → 86 out (70 reasoning)
cleanup fix-1 fix-2
openai/gpt-oss-20b:medium PASS 100% 177 in → 201 out (204 reasoning)
cleanup fix-1 fix-2
openai/gpt-oss-20b:medium__json_schema PASS 100% 182 in → 135 out (100 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
openai/gpt-oss-20b:xhigh PASS 100% 163 in → 253 out (249 reasoning)
cleanup fix-1 fix-2
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 172 in → 344 out (318 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
poolside/laguna-m.1:high PASS 100% 135 in → 249 out (237 reasoning)
cleanup fix-1 fix-2
poolside/laguna-m.1:high__json_schema PASS 100% 136 in → 358 out (338 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-m.1:low PASS 100% 136 in → 356 out (344 reasoning)
cleanup fix-1 fix-2
poolside/laguna-m.1:low__json_schema PASS 100% 130 in → 263 out (243 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-m.1:medium PASS 100% 134 in → 342 out (330 reasoning)
cleanup fix-1 fix-2
poolside/laguna-m.1:medium__json_schema PASS 100% 136 in → 378 out (358 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-m.1:none PASS 100% 129 in → 11 out (0 reasoning)
cleanup fix-1 fix-2
poolside/laguna-m.1:none__json_schema PASS 100% 130 in → 19 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-m.1:xhigh PASS 100% 134 in → 274 out (262 reasoning)
cleanup fix-1 fix-2
poolside/laguna-m.1:xhigh__json_schema PASS 100% 135 in → 358 out (325 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
poolside/laguna-xs-2.1:high PASS 100% 136 in → 206 out (196 reasoning)
cleanup fix-1 fix-2
poolside/laguna-xs-2.1:high__json_schema PASS 100% 131 in → 188 out (167 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-xs-2.1:low PASS 100% 134 in → 185 out (175 reasoning)
cleanup fix-1 fix-2
poolside/laguna-xs-2.1:low__json_schema PASS 100% 136 in → 308 out (273 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
poolside/laguna-xs-2.1:medium PASS 100% 134 in → 299 out (289 reasoning)
cleanup fix-1 fix-2
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 133 in → 210 out (190 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-xs-2.1:none PASS 100% 131 in → 9 out (0 reasoning)
cleanup fix-1 fix-2
poolside/laguna-xs-2.1:none__json_schema PASS 100% 132 in → 19 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-xs-2.1:xhigh PASS 100% 132 in → 243 out (233 reasoning)
cleanup fix-1 fix-2
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 134 in → 277 out (241 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
poolside/laguna-xs.2:high PASS 100% 173 in → 243 out (231 reasoning)
cleanup fix-1 fix-2
poolside/laguna-xs.2:high__json_schema PASS 100% 172 in → 251 out (231 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-xs.2:low PASS 100% 173 in → 229 out (217 reasoning)
cleanup fix-1 fix-2
poolside/laguna-xs.2:low__json_schema PASS 100% 171 in → 185 out (165 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-xs.2:medium PASS 100% 170 in → 276 out (264 reasoning)
cleanup fix-1 fix-2
poolside/laguna-xs.2:medium__json_schema PASS 100% 171 in → 172 out (152 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-xs.2:none PASS 100% 171 in → 11 out (0 reasoning)
cleanup fix-1 fix-2
poolside/laguna-xs.2:none__json_schema PASS 100% 171 in → 19 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
poolside/laguna-xs.2:xhigh PASS 100% 172 in → 196 out (184 reasoning)
cleanup fix-1 fix-2
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 171 in → 251 out (231 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["cleanup", "fix-1", "fix-2"]}
qwen/qwen3.6-27b:high PASS 100% 128 in → 1,002 out (1 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-27b:high__json_schema PASS 100% 129 in → 986 out (866 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
qwen/qwen3.6-27b:low PASS 100% 128 in → 445 out (384 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-27b:medium PASS 100% 126 in → 475 out (413 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-27b:medium__json_schema PASS 100% 130 in → 34 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
qwen/qwen3.6-27b:none PASS 100% 134 in → 10 out (0 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-27b:none__json_schema PASS 100% 130 in → 34 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 135 in → 1,031 out (960 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 135 in → 808 out (714 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
qwen/qwen3.6-35b-a3b:high PASS 100% 131 in → 770 out (726 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 134 in → 1,023 out (923 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 133 in → 747 out (664 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 132 in → 729 out (641 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 132 in → 918 out (904 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 131 in → 535 out (489 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 136 in → 10 out (0 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 132 in → 34 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 134 in → 725 out (711 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 134 in → 1,030 out (927 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
qwen/qwen3.6-flash:high PASS 100% 129 in → 389 out (375 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-flash:low PASS 100% 131 in → 364 out (350 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-flash:medium PASS 100% 131 in → 791 out (777 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-flash:none PASS 100% 134 in → 9 out (0 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.6-flash:xhigh PASS 100% 128 in → 946 out (932 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-flash:high PASS 100% 130 in → 677 out (663 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-flash:low PASS 100% 134 in → 824 out (810 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-flash:max PASS 100% 134 in → 920 out (906 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-flash:medium PASS 100% 128 in → 836 out (822 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-flash:none PASS 100% 127 in → 9 out (0 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-flash:xhigh PASS 100% 129 in → 365 out (351 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-max:high PASS 100% 134 in → 215 out (201 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-max:low PASS 100% 130 in → 251 out (237 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-max:medium PASS 100% 129 in → 218 out (204 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-max:none PASS 100% 135 in → 9 out (0 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-max:xhigh PASS 100% 133 in → 260 out (246 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-plus:high PASS 100% 133 in → 509 out (495 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-plus:low PASS 100% 136 in → 477 out (463 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-plus:medium PASS 100% 129 in → 189 out (175 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-plus:none PASS 100% 136 in → 9 out (0 reasoning)
cleanup fix-1 fix-2
qwen/qwen3.7-plus:xhigh PASS 100% 133 in → 509 out (495 reasoning)
cleanup fix-1 fix-2
tencent/hy3:high PASS 100% 126 in → 546 out (534 reasoning)
cleanup fix-1 fix-2
tencent/hy3:high__json_schema PASS 100% 124 in → 605 out (543 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
tencent/hy3:low PASS 100% 131 in → 491 out (398 reasoning)
cleanup fix-1 fix-2
tencent/hy3:low__json_schema PASS 100% 122 in → 387 out (334 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
tencent/hy3:medium PASS 100% 130 in → 442 out (373 reasoning)
cleanup fix-1 fix-2
tencent/hy3:medium__json_schema PASS 100% 128 in → 467 out (401 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
tencent/hy3:none PASS 100% 128 in → 11 out (0 reasoning)
cleanup fix-1 fix-2
tencent/hy3:none__json_schema PASS 100% 128 in → 23 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
tencent/hy3:xhigh PASS 100% 127 in → 382 out (370 reasoning)
cleanup fix-1 fix-2
tencent/hy3:xhigh__json_schema PASS 100% 124 in → 631 out (558 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
thinkingmachines/inkling-small:high PASS 100% 125 in → 120 out (105 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling-small:high__json_schema PASS 100% 122 in → 182 out (149 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
thinkingmachines/inkling-small:low PASS 100% 125 in → 38 out (22 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling-small:low__json_schema PASS 100% 120 in → 58 out (25 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
thinkingmachines/inkling-small:max PASS 100% 126 in → 286 out (244 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling-small:max__json_schema PASS 100% 126 in → 230 out (198 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
thinkingmachines/inkling-small:medium PASS 100% 124 in → 122 out (106 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling-small:medium__json_schema PASS 100% 124 in → 100 out (68 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 125 in → 213 out (183 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 122 in → 179 out (152 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
thinkingmachines/inkling:high PASS 100% 128 in → 230 out (197 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling:low PASS 100% 126 in → 82 out (71 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling:max PASS 100% 127 in → 168 out (152 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling:medium PASS 100% 123 in → 166 out (151 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling:medium__json_schema PASS 100% 124 in → 99 out (74 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
thinkingmachines/inkling:none PASS 100% 123 in → 12 out (0 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling:none__json_schema PASS 100% 121 in → 30 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
thinkingmachines/inkling:xhigh PASS 100% 121 in → 212 out (184 reasoning)
cleanup fix-1 fix-2
thinkingmachines/inkling:xhigh__json_schema PASS 100% 126 in → 273 out (241 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
x-ai/grok-4.3:high PASS 100% 291 in → 436 out (427 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.3:high__json_schema PASS 100% 359 in → 498 out (483 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
x-ai/grok-4.3:low PASS 100% 301 in → 500 out (491 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.3:low__json_schema PASS 100% 366 in → 322 out (307 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
x-ai/grok-4.3:max PASS 100% 291 in → 176 out (167 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.3:max__json_schema PASS 100% 357 in → 398 out (383 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
x-ai/grok-4.3:medium PASS 100% 300 in → 632 out (623 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.3:medium__json_schema PASS 100% 368 in → 361 out (346 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
x-ai/grok-4.3:none PASS 100% 290 in → 9 out (0 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.3:none__json_schema PASS 100% 359 in → 15 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
x-ai/grok-4.3:xhigh PASS 100% 291 in → 399 out (390 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.3:xhigh__json_schema PASS 100% 361 in → 366 out (351 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
x-ai/grok-4.5:high PASS 100% 318 in → 519 out (510 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.5:high__json_schema PASS 100% 400 in → 510 out (492 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
x-ai/grok-4.5:low PASS 100% 319 in → 403 out (394 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.5:low__json_schema PASS 100% 397 in → 343 out (325 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
x-ai/grok-4.5:max PASS 100% 320 in → 625 out (616 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.5:max__json_schema PASS 100% 399 in → 404 out (386 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
x-ai/grok-4.5:medium PASS 100% 314 in → 412 out (403 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.5:medium__json_schema PASS 100% 398 in → 395 out (377 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
x-ai/grok-4.5:xhigh PASS 100% 315 in → 513 out (504 reasoning)
cleanup fix-1 fix-2
x-ai/grok-4.5:xhigh__json_schema PASS 100% 397 in → 494 out (476 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) {"branches_to_delete":["cleanup","fix-1","fix-2"]}
z-ai/glm-4.7-flash:high PASS 100% 121 in → 696 out (658 reasoning)
cleanup fix-1 fix-2
z-ai/glm-4.7-flash:low PASS 100% 114 in → 593 out (544 reasoning)
cleanup fix-1 fix-2
z-ai/glm-4.7-flash:medium PASS 100% 119 in → 654 out (650 reasoning)
cleanup fix-1 fix-2
z-ai/glm-4.7-flash:none__json_schema PASS 100% 119 in → 28 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
z-ai/glm-4.7-flash:xhigh PASS 100% 118 in → 476 out (408 reasoning)
cleanup fix-1 fix-2
z-ai/glm-5.2:high PASS 100% 127 in → 282 out (271 reasoning)
cleanup fix-1 fix-2
z-ai/glm-5.2:low PASS 100% 125 in → 257 out (0 reasoning)
cleanup fix-1 fix-2
z-ai/glm-5.2:low__json_schema PASS 100% 118 in → 28 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
z-ai/glm-5.2:medium PASS 100% 127 in → 454 out (444 reasoning)
cleanup fix-1 fix-2
z-ai/glm-5.2:medium__json_schema PASS 100% 127 in → 314 out (285 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
z-ai/glm-5.2:none PASS 100% 122 in → 10 out (0 reasoning)
cleanup fix-1 fix-2
z-ai/glm-5.2:none__json_schema PASS 100% 121 in → 22 out (0 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup", "fix-1", "fix-2"] }
z-ai/glm-5.2:xhigh PASS 100% 121 in → 392 out (381 reasoning)
cleanup fix-1 fix-2
z-ai/glm-5.2:xhigh__json_schema PASS 100% 125 in → 29 out (27 reasoning)
cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
openai/gpt-oss-20b:low__json_schema FAIL 66.7% 160 in → 140 out (92 reasoning)
cleanup cc51144 Cleanup fix-1 fix-2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["cleanup cc51144 Cleanup", "fix-1", "fix-2"] }
Failure: Missing: ['cleanup'] Extra: ['cleanup cc51144 Cleanup']
thinkingmachines/inkling-small:none FAIL 66.7% 125 in → 12 out (0 reasoning)
cleanup fix-1 fix-2<|end_message|>
Failure: Missing: ['fix-2'] Extra: ['fix-2<|end_message|>']
arcee-ai/trinity-mini:high FAIL 33.3% 117 in → 410 out (442 reasoning)
cleanupfix-1 fix-2
Failure: Missing: ['cleanup', 'fix-1'] Extra: ['cleanupfix-1']
arcee-ai/trinity-mini:medium FAIL 33.3% 119 in → 350 out (367 reasoning)
cleanupfix-1 fix-2
Failure: Missing: ['cleanup', 'fix-1'] Extra: ['cleanupfix-1']
z-ai/glm-4.7-flash:none FAIL 33.3% 119 in → 4 out (0 reasoning)
fix-2
Failure: Missing: ['cleanup', 'fix-1']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: cleanup{ "branches_to_delete": [ "cleanup", "fix-1", "fix-2" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: cleanup fix-1 fix-2
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 142 in → 2 out (0 reasoning)
none
Failure: Missing: ['cleanup', 'fix-1', 'fix-2'] Extra: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b91ddf16-ab5d-93a2-9bd8-63673e1dd639","request_id":"b91ddf16-ab5d-93a2-9bd8-63673e1dd639"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3044256a-148e-9c0b-a2c7-eaf8429fffe9","request_id":"3044256a-148e-9c0b-a2c7-eaf8429fffe9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-276b3cc2-6907-9f20-b291-bf60c44a71dd","request_id":"276b3cc2-6907-9f20-b291-bf60c44a71dd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4e023341-f375-9d60-9cac-0dc27e25ecd2","request_id":"4e023341-f375-9d60-9cac-0dc27e25ecd2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5421d77b-af40-98f5-883d-2ec0b2727586","request_id":"5421d77b-af40-98f5-883d-2ec0b2727586"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-dd1ffcc6-6af7-933f-b8df-3db40f245ee0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-acd1a3cb-fe3d-962d-8cc0-d300a9889598"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4d7f130a-5f4f-90bc-a171-45a979dd0899"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-da499f2f-96f2-9896-b229-2b1f84fd8eaa"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a85f4f86-4039-933c-9b4d-d8855c345153"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-418bd8bd-3e6c-9af9-bd76-cd5076a5814c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1bba8102-d858-9485-a9ad-bfac4e2ee15a","request_id":"1bba8102-d858-9485-a9ad-bfac4e2ee15a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-14e18664-2d17-9c29-beaa-2aa475daa65e","request_id":"14e18664-2d17-9c29-beaa-2aa475daa65e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0bfdc13f-4622-93c2-92ad-82157528c550","request_id":"0bfdc13f-4622-93c2-92ad-82157528c550"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-517c0563-17c5-9e6f-9256-ab788b5967ce","request_id":"517c0563-17c5-9e6f-9256-ab788b5967ce"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3e94642d-22bc-9109-a6a7-191b71b40711","request_id":"3e94642d-22bc-9109-a6a7-191b71b40711"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e3b1db26-321a-9462-afbd-a9048b34c156","request_id":"e3b1db26-321a-9462-afbd-a9048b34c156"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1ad7114f-a578-9436-8400-9e446779b77d","request_id":"1ad7114f-a578-9436-8400-9e446779b77d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-90b80ebc-e355-9aee-aa48-7fa87ac875ce","request_id":"90b80ebc-e355-9aee-aa48-7fa87ac875ce"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ecdcbbf3-3cd0-9aa5-8f81-de2f4a448366","request_id":"ecdcbbf3-3cd0-9aa5-8f81-de2f4a448366"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-74c46f38-0c52-9b20-b06a-f32eff32a094","request_id":"74c46f38-0c52-9b20-b06a-f32eff32a094"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: cleanup fix-1 fix-2{"branches_to_delete":["cleanup","fix-1","fix-2"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: cleanup fix-1 fix-2
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: cleanup fix-1 fix-2
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: cleanup fix-1 fix-2
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: cleanup fix-1 fix-2
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)