Mixed naming conventions
Tests ability to handle mixed branch naming conventions (feature/, bugfix/, hotfix/) when identifying merged branches.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'init' > init.txt
  5. 05 git add init.txt
  6. 06 git commit -m 'Init'
  7. 07 git checkout -b bugfix-123
  8. 08 echo 'fix' > fix.txt
  9. 09 git add fix.txt
  10. 10 git commit -m 'Fix bug 123'
  11. 11 git checkout main
  12. 12 git merge bugfix-123
  13. 13 git checkout -b feature/search
  14. 14 echo 'search' > search.py
  15. 15 git add search.py
  16. 16 git commit -m 'Add search'
  17. 17 git checkout main
  18. 18 git merge feature/search
  19. 19 git checkout -b experiment-ml
  20. 20 echo 'ml' > ml.py
  21. 21 git add ml.py
  22. 22 git commit -m 'ML experiment'
  23. 23 git checkout main
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
bugfix-123 feature/search

Scoped model quality, cost, API time, and token usage for branch_cleanup/f006.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 164 in → 63 out (45 reasoning)
bugfix-123 feature/search
anthropic/claude-fable-5:high__json_schema PASS 100% 403 in → 57 out (28 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
anthropic/claude-fable-5:low FAIL 100% 161 in → 84 out (24 reasoning)
bugfix-123 feature/search Both branches are fully merged into main (feature/search even points to the same commit as main). `experiment-ml` is not merged, and `main` itself should never be deleted.
Failure: Extra: ['Both branches are fully merged into main (feature/search even points to the same commit as main). `experiment-ml` is not merged, and `main` itself should never be deleted.']
anthropic/claude-fable-5:low__json_schema PASS 100% 399 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-fable-5:max PASS 100% 160 in → 232 out (63 reasoning)
bugfix-123 feature/search
anthropic/claude-fable-5:max__json_schema PASS 100% 402 in → 126 out (55 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
anthropic/claude-fable-5:medium PASS 100% 165 in → 61 out (37 reasoning)
bugfix-123 feature/search
anthropic/claude-fable-5:medium__json_schema PASS 100% 399 in → 49 out (31 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
anthropic/claude-fable-5:xhigh PASS 100% 160 in → 76 out (33 reasoning)
bugfix-123 feature/search
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 401 in → 57 out (28 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
anthropic/claude-haiku-4.5:high PASS 100% 157 in → 268 out (227 reasoning)
bugfix-123 feature/search
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 345 in → 440 out (380 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
anthropic/claude-haiku-4.5:low PASS 100% 153 in → 177 out (148 reasoning)
bugfix-123 feature/search
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 346 in → 284 out (241 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
anthropic/claude-haiku-4.5:medium PASS 100% 157 in → 223 out (192 reasoning)
bugfix-123 feature/search
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 347 in → 355 out (310 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
anthropic/claude-haiku-4.5:none PASS 100% 125 in → 11 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 319 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 153 in → 203 out (181 reasoning)
bugfix-123 feature/search
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 346 in → 256 out (232 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
anthropic/claude-opus-4.6:high PASS 100% 124 in → 69 out (45 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.6:high__json_schema PASS 100% 314 in → 56 out (20 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.6:low PASS 100% 125 in → 11 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.6:low__json_schema PASS 100% 317 in → 21 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.6:max FAIL 100% 127 in → 206 out (127 reasoning)
bugfix-123 feature/search These two branches are listed as merged into `main` (and `feature/search` even points to the same commit `3b4f081` as `main`). The `experiment-ml` branch is **not** merged and should be kept. `main` itself is obviously not deleted.
Failure: Extra: ['These two branches are listed as merged into `main` (and `feature/search` even points to the same commit `3b4f081` as `main`). The `experiment-ml` branch is **not** merged and should be kept. `main` itself is obviously not deleted.']
anthropic/claude-opus-4.6:max__json_schema PASS 100% 317 in → 135 out (99 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.6:medium PASS 100% 128 in → 48 out (20 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 319 in → 21 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.6:none PASS 100% 126 in → 11 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.6:none__json_schema PASS 100% 317 in → 21 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 123 in → 75 out (50 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 317 in → 56 out (20 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.7:high PASS 100% 165 in → 15 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.7:high__json_schema PASS 100% 407 in → 29 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.7:low PASS 100% 162 in → 15 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.7:low__json_schema PASS 100% 402 in → 29 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.7:max PASS 100% 166 in → 178 out (56 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.7:max__json_schema PASS 100% 405 in → 190 out (81 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.7:medium PASS 100% 163 in → 15 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 406 in → 29 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.7:none PASS 100% 163 in → 15 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.7:none__json_schema PASS 100% 409 in → 29 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 163 in → 15 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 403 in → 29 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.8:high PASS 100% 159 in → 110 out (46 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.8:high__json_schema PASS 100% 399 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.8:low PASS 100% 162 in → 45 out (20 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.8:low__json_schema PASS 100% 403 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.8:max PASS 100% 164 in → 439 out (68 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.8:max__json_schema PASS 100% 401 in → 271 out (75 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.8:medium PASS 100% 161 in → 45 out (26 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 402 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.8:none PASS 100% 161 in → 13 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-opus-4.8:none__json_schema PASS 100% 399 in → 26 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-opus-4.8:xhigh FAIL 100% 160 in → 213 out (62 reasoning)
Based on the "Merged into main" list, the following branches are fully merged and can be safely deleted (excluding `main` itself): bugfix-123 feature/search
Failure: Extra: ['Based on the "Merged into main" list, the following branches are fully merged and can be safely deleted (excluding `main` itself):']
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 398 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-4.6:high FAIL 100% 123 in → 139 out (110 reasoning)
``` bugfix-123 feature/search ```
Failure: Extra: ['```']
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 311 in → 79 out (42 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-4.6:low PASS 100% 125 in → 11 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 317 in → 21 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-4.6:max PASS 100% 122 in → 144 out (123 reasoning)
bugfix-123 feature/search
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 317 in → 117 out (86 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 129 in → 11 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 316 in → 21 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-4.6:none PASS 100% 127 in → 11 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 317 in → 21 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 126 in → 132 out (102 reasoning)
bugfix-123 feature/search
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 316 in → 75 out (40 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-5:high FAIL 100% 158 in → 106 out (0 reasoning)
Based on the merge status provided, the following branches are fully merged into main and can be safely deleted: bugfix-123 feature/search **Note:** `experiment-ml` is not listed as merged into main, so it should be kept (deleting it could result in losing unmerged work). `main` is the current branch and should never be deleted.
Failure: Extra: ['**Note:** `experiment-ml` is not listed as merged into main, so it should be kept (deleting it could result in losing unmerged work). `main` is the current branch and should never be deleted.', 'Based on the merge status provided, the following branches are fully merged into main and can be safely deleted:']
anthropic/claude-sonnet-5:high__json_schema PASS 100% 401 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-5:low PASS 100% 156 in → 13 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-sonnet-5:low__json_schema PASS 100% 400 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-5:medium PASS 100% 159 in → 13 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 400 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-5:none PASS 100% 161 in → 13 out (0 reasoning)
bugfix-123 feature/search
anthropic/claude-sonnet-5:none__json_schema PASS 100% 402 in → 26 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 162 in → 134 out (60 reasoning)
bugfix-123 feature/search
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 401 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
arcee-ai/trinity-large-thinking:high PASS 100% 116 in → 539 out (541 reasoning)
bugfix-123 feature/search
arcee-ai/trinity-large-thinking:low PASS 100% 119 in → 355 out (351 reasoning)
bugfix-123 feature/search
arcee-ai/trinity-large-thinking:medium PASS 100% 117 in → 295 out (292 reasoning)
bugfix-123 feature/search
arcee-ai/trinity-large-thinking:xhigh PASS 100% 117 in → 386 out (382 reasoning)
bugfix-123 feature/search
arcee-ai/trinity-mini:high PASS 100% 119 in → 307 out (334 reasoning)
bugfix-123 feature/search
arcee-ai/trinity-mini:high__json_schema PASS 100% 118 in → 316 out (336 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
arcee-ai/trinity-mini:low PASS 100% 116 in → 431 out (483 reasoning)
bugfix-123 feature/search
arcee-ai/trinity-mini:low__json_schema PASS 100% 115 in → 268 out (276 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
arcee-ai/trinity-mini:medium PASS 100% 116 in → 233 out (255 reasoning)
bugfix-123 feature/search
arcee-ai/trinity-mini:medium__json_schema PASS 100% 116 in → 287 out (289 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 121 in → 413 out (455 reasoning)
bugfix-123 feature/search
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 114 in → 271 out (276 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 201 in → 93 out (91 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 118 in → 381 out (389 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
deepseek/deepseek-v4-flash-0731:low PASS 100% 117 in → 521 out (539 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 115 in → 99 out (103 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 198 in → 155 out (144 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 196 in → 158 out (143 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 118 in → 71 out (63 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 119 in → 123 out (106 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 120 in → 8 out (0 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 123 in → 23 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 198 in → 181 out (173 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 118 in → 63 out (51 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"] }
deepseek/deepseek-v4-flash:high PASS 100% 117 in → 103 out (96 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 116 in → 125 out (105 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
deepseek/deepseek-v4-flash:low PASS 100% 117 in → 90 out (82 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 225 in → 18 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
deepseek/deepseek-v4-flash:medium PASS 100% 114 in → 104 out (95 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 114 in → 123 out (103 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
deepseek/deepseek-v4-flash:none PASS 100% 115 in → 7 out (0 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 116 in → 20 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
deepseek/deepseek-v4-flash:xhigh PASS 100% 195 in → 240 out (231 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 200 in → 891 out (865 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
deepseek/deepseek-v4-pro:high PASS 100% 117 in → 184 out (175 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 314 in → 116 out (97 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
deepseek/deepseek-v4-pro:low PASS 100% 114 in → 235 out (226 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 121 in → 276 out (250 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
deepseek/deepseek-v4-pro:medium PASS 100% 117 in → 240 out (231 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 114 in → 25 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
deepseek/deepseek-v4-pro:none PASS 100% 118 in → 7 out (0 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 117 in → 24 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 120 in → 249 out (255 reasoning)
bugfix-123 feature/search
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 396 in → 511 out (492 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
google/gemini-3-flash-preview:high PASS 100% 130 in → 269 out (259 reasoning)
bugfix-123 feature/search
google/gemini-3-flash-preview:high__json_schema PASS 100% 208 in → 260 out (227 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemini-3-flash-preview:low PASS 100% 128 in → 384 out (374 reasoning)
bugfix-123 feature/search
google/gemini-3-flash-preview:low__json_schema PASS 100% 209 in → 390 out (372 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3-flash-preview:medium PASS 100% 130 in → 260 out (250 reasoning)
bugfix-123 feature/search
google/gemini-3-flash-preview:medium__json_schema PASS 100% 209 in → 299 out (281 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3-flash-preview:xhigh PASS 100% 129 in → 276 out (266 reasoning)
bugfix-123 feature/search
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 208 in → 318 out (300 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 126 in → 380 out (370 reasoning)
bugfix-123 feature/search
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 127 in → 926 out (908 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 124 in → 151 out (141 reasoning)
bugfix-123 feature/search
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 128 in → 139 out (110 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 129 in → 199 out (189 reasoning)
bugfix-123 feature/search
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 125 in → 274 out (254 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 127 in → 417 out (407 reasoning)
bugfix-123 feature/search
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 127 in → 442 out (424 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.1-pro-preview:high PASS 100% 130 in → 403 out (393 reasoning)
bugfix-123 feature/search
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 204 in → 298 out (280 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.1-pro-preview:low PASS 100% 128 in → 312 out (302 reasoning)
bugfix-123 feature/search
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 202 in → 123 out (90 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemini-3.1-pro-preview:medium PASS 100% 123 in → 369 out (359 reasoning)
bugfix-123 feature/search
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 203 in → 154 out (136 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 126 in → 463 out (453 reasoning)
bugfix-123 feature/search
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 207 in → 595 out (577 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.5-flash-lite:high PASS 100% 125 in → 416 out (406 reasoning)
bugfix-123 feature/search
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 208 in → 322 out (302 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
google/gemini-3.5-flash-lite:low PASS 100% 128 in → 10 out (0 reasoning)
bugfix-123 feature/search
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 205 in → 33 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemini-3.5-flash-lite:max PASS 100% 124 in → 433 out (423 reasoning)
bugfix-123 feature/search
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 210 in → 324 out (306 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.5-flash-lite:medium PASS 100% 129 in → 375 out (365 reasoning)
bugfix-123 feature/search
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 128 in → 357 out (324 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemini-3.5-flash-lite:xhigh PASS 100% 126 in → 381 out (371 reasoning)
bugfix-123 feature/search
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 208 in → 408 out (390 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.5-flash:high PASS 100% 129 in → 516 out (506 reasoning)
bugfix-123 feature/search
google/gemini-3.5-flash:high__json_schema PASS 100% 208 in → 381 out (363 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.5-flash:low PASS 100% 128 in → 149 out (139 reasoning)
bugfix-123 feature/search
google/gemini-3.5-flash:low__json_schema PASS 100% 206 in → 128 out (110 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.5-flash:medium PASS 100% 128 in → 379 out (369 reasoning)
bugfix-123 feature/search
google/gemini-3.5-flash:medium__json_schema PASS 100% 209 in → 296 out (278 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.5-flash:xhigh PASS 100% 123 in → 490 out (480 reasoning)
bugfix-123 feature/search
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 208 in → 339 out (321 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.6-flash:high PASS 100% 131 in → 479 out (469 reasoning)
bugfix-123 feature/search
google/gemini-3.6-flash:high__json_schema PASS 100% 128 in → 364 out (346 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.6-flash:low PASS 100% 131 in → 115 out (105 reasoning)
bugfix-123 feature/search
google/gemini-3.6-flash:low__json_schema PASS 100% 130 in → 33 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemini-3.6-flash:max PASS 100% 129 in → 507 out (497 reasoning)
bugfix-123 feature/search
google/gemini-3.6-flash:max__json_schema PASS 100% 128 in → 434 out (416 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.6-flash:medium PASS 100% 130 in → 395 out (385 reasoning)
bugfix-123 feature/search
google/gemini-3.6-flash:medium__json_schema PASS 100% 130 in → 247 out (229 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemini-3.6-flash:xhigh PASS 100% 126 in → 459 out (449 reasoning)
bugfix-123 feature/search
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 129 in → 480 out (462 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemma-4-26b-a4b-it:high PASS 100% 142 in → 337 out (284 reasoning)
bugfix-123 feature/search
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 141 in → 383 out (281 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 140 in → 344 out (263 reasoning)
bugfix-123 feature/search
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 143 in → 388 out (289 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 141 in → 356 out (285 reasoning)
bugfix-123 feature/search
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 144 in → 365 out (269 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 138 in → 11 out (0 reasoning)
bugfix-123 feature/search
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 138 in → 34 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 142 in → 349 out (270 reasoning)
bugfix-123 feature/search
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 140 in → 362 out (279 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemma-4-31b-it:high PASS 100% 144 in → 221 out (1 reasoning)
bugfix-123 feature/search
google/gemma-4-31b-it:high__json_schema PASS 100% 144 in → 251 out (179 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
google/gemma-4-31b-it:low PASS 100% 142 in → 286 out (224 reasoning)
bugfix-123 feature/search
google/gemma-4-31b-it:low__json_schema PASS 100% 144 in → 263 out (243 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemma-4-31b-it:medium PASS 100% 142 in → 296 out (217 reasoning)
bugfix-123 feature/search
google/gemma-4-31b-it:medium__json_schema PASS 100% 144 in → 215 out (195 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemma-4-31b-it:none PASS 100% 141 in → 11 out (0 reasoning)
bugfix-123 feature/search
google/gemma-4-31b-it:none__json_schema PASS 100% 140 in → 21 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
google/gemma-4-31b-it:xhigh PASS 100% 142 in → 260 out (1 reasoning)
bugfix-123 feature/search
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 147 in → 270 out (249 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
ibm-granite/granite-4.1-8b PASS 100% 126 in → 8 out
bugfix-123 feature/search
inclusionai/ling-2.6-flash PASS 100% 148 in → 10 out
bugfix-123 feature/search
inclusionai/ling-2.6-flash__json_schema PASS 100% 153 in → 32 out
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
liquid/lfm-2-24b-a2b PASS 100% 125 in → 9 out
bugfix-123 feature/search
minimax/minimax-m2.5:high PASS 100% 153 in → 227 out (262 reasoning)
bugfix-123 feature/search
minimax/minimax-m2.5:high__json_schema PASS 100% 150 in → 295 out (295 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
minimax/minimax-m2.5:low PASS 100% 157 in → 169 out (159 reasoning)
bugfix-123 feature/search
minimax/minimax-m2.5:low__json_schema PASS 100% 152 in → 202 out (175 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
minimax/minimax-m2.5:medium PASS 100% 148 in → 253 out (271 reasoning)
bugfix-123 feature/search
minimax/minimax-m2.5:medium__json_schema PASS 100% 154 in → 377 out (352 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
minimax/minimax-m2.5:xhigh PASS 100% 156 in → 132 out (143 reasoning)
bugfix-123 feature/search
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 149 in → 208 out (202 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
minimax/minimax-m2.7:high PASS 100% 152 in → 283 out (273 reasoning)
bugfix-123 feature/search
minimax/minimax-m2.7:high__json_schema PASS 100% 153 in → 21 out (20 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
minimax/minimax-m2.7:low PASS 100% 150 in → 295 out (287 reasoning)
bugfix-123 feature/search
minimax/minimax-m2.7:low__json_schema PASS 100% 300 in → 392 out (373 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
minimax/minimax-m2.7:medium PASS 100% 146 in → 255 out (279 reasoning)
bugfix-123 feature/search
minimax/minimax-m2.7:medium__json_schema PASS 100% 301 in → 473 out (456 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
minimax/minimax-m2.7:xhigh PASS 100% 153 in → 120 out (111 reasoning)
bugfix-123 feature/search
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 301 in → 376 out (357 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
minimax/minimax-m3:high PASS 100% 287 in → 110 out (124 reasoning)
bugfix-123 feature/search
minimax/minimax-m3:high__json_schema PASS 100% 277 in → 20 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
minimax/minimax-m3:low PASS 100% 462 in → 148 out (144 reasoning)
bugfix-123 feature/search
minimax/minimax-m3:low__json_schema PASS 100% 275 in → 25 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
minimax/minimax-m3:medium PASS 100% 287 in → 103 out (106 reasoning)
bugfix-123 feature/search
minimax/minimax-m3:medium__json_schema PASS 100% 274 in → 25 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
minimax/minimax-m3:xhigh PASS 100% 290 in → 98 out (101 reasoning)
bugfix-123 feature/search
minimax/minimax-m3:xhigh__json_schema PASS 100% 276 in → 25 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
mistralai/devstral-2512 PASS 100% 125 in → 10 out
bugfix-123 feature/search
mistralai/devstral-2512__json_schema PASS 100% 132 in → 20 out
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
mistralai/mistral-medium-3-5:high PASS 100% 140 in → 494 out (453 reasoning)
bugfix-123 feature/search
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 133 in → 338 out (306 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
mistralai/mistral-medium-3-5:low PASS 100% 143 in → 360 out (338 reasoning)
bugfix-123 feature/search
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 130 in → 233 out (189 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
mistralai/mistral-medium-3-5:medium PASS 100% 144 in → 276 out (247 reasoning)
bugfix-123 feature/search
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 129 in → 214 out (176 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
mistralai/mistral-medium-3-5:none PASS 100% 147 in → 10 out (0 reasoning)
bugfix-123 feature/search
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 129 in → 20 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 145 in → 407 out (365 reasoning)
bugfix-123 feature/search
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 132 in → 150 out (139 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
moonshotai/kimi-k2.6:high PASS 100% 121 in → 368 out (359 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k2.6:high__json_schema PASS 100% 115 in → 1,020 out (1,002 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
moonshotai/kimi-k2.6:low PASS 100% 120 in → 281 out (274 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k2.6:low__json_schema PASS 100% 115 in → 411 out (385 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
moonshotai/kimi-k2.6:medium PASS 100% 118 in → 558 out (550 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 306 in → 507 out (490 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
moonshotai/kimi-k2.6:xhigh PASS 100% 118 in → 190 out (182 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 119 in → 305 out (288 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
moonshotai/kimi-k2.7-code:high PASS 100% 120 in → 168 out (157 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 121 in → 155 out (148 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
moonshotai/kimi-k2.7-code:low PASS 100% 120 in → 122 out (124 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 120 in → 117 out (105 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
moonshotai/kimi-k2.7-code:medium PASS 100% 114 in → 144 out (135 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 120 in → 140 out (121 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 117 in → 164 out (155 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 119 in → 178 out (156 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
moonshotai/kimi-k3:high PASS 100% 197 in → 215 out (193 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k3:high__json_schema PASS 100% 199 in → 120 out (98 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k3:low PASS 100% 198 in → 208 out (186 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k3:low__json_schema PASS 100% 194 in → 192 out (170 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k3:max PASS 100% 193 in → 192 out (170 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k3:max__json_schema PASS 100% 192 in → 137 out (115 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k3:medium PASS 100% 194 in → 219 out (197 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k3:medium__json_schema PASS 100% 198 in → 148 out (126 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k3:xhigh PASS 100% 190 in → 136 out (114 reasoning)
bugfix-123 feature/search
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 195 in → 208 out (186 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 136 in → 149 out (154 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 146 in → 85 out (63 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 143 in → 87 out (86 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 145 in → 107 out (84 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 138 in → 90 out (84 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 142 in → 1,605 out (1,689 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 143 in → 10 out (0 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 148 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete" : ["bugfix-123", "feature/search"] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 145 in → 87 out (88 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 143 in → 235 out (196 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 146 in → 70 out (61 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 146 in → 123 out (85 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 143 in → 109 out (93 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 144 in → 82 out (50 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 144 in → 92 out (73 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 144 in → 179 out (134 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 143 in → 10 out (0 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 142 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 143 in → 98 out (88 reasoning)
bugfix-123 feature/search
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 145 in → 159 out (122 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
openai/gpt-5.4-mini:high PASS 100% 120 in → 97 out (84 reasoning)
bugfix-123 feature/search
openai/gpt-5.4-mini:high__json_schema PASS 100% 160 in → 214 out (190 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4-mini:low PASS 100% 115 in → 54 out (41 reasoning)
bugfix-123 feature/search
openai/gpt-5.4-mini:low__json_schema PASS 100% 162 in → 63 out (39 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4-mini:medium PASS 100% 120 in → 73 out (60 reasoning)
bugfix-123 feature/search
openai/gpt-5.4-mini:medium__json_schema PASS 100% 161 in → 119 out (95 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4-mini:none PASS 100% 120 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.4-mini:none__json_schema PASS 100% 156 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4-mini:xhigh PASS 100% 115 in → 138 out (125 reasoning)
bugfix-123 feature/search
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 162 in → 132 out (108 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4-nano:high PASS 100% 117 in → 49 out (36 reasoning)
bugfix-123 feature/search
openai/gpt-5.4-nano:high__json_schema PASS 100% 161 in → 61 out (37 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4-nano:low PASS 100% 123 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.4-nano:low__json_schema PASS 100% 157 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4-nano:medium PASS 100% 113 in → 51 out (38 reasoning)
bugfix-123 feature/search
openai/gpt-5.4-nano:medium__json_schema PASS 100% 162 in → 105 out (81 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4-nano:none PASS 100% 119 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.4-nano:none__json_schema PASS 100% 159 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4-nano:xhigh PASS 100% 112 in → 140 out (127 reasoning)
bugfix-123 feature/search
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 160 in → 412 out (388 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4:high PASS 100% 116 in → 72 out (59 reasoning)
bugfix-123 feature/search
openai/gpt-5.4:high__json_schema PASS 100% 157 in → 89 out (65 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4:low PASS 100% 116 in → 73 out (60 reasoning)
bugfix-123 feature/search
openai/gpt-5.4:low__json_schema PASS 100% 153 in → 59 out (35 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4:medium PASS 100% 117 in → 68 out (55 reasoning)
bugfix-123 feature/search
openai/gpt-5.4:medium__json_schema PASS 100% 159 in → 79 out (55 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4:none PASS 100% 117 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.4:none__json_schema PASS 100% 159 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.4:xhigh PASS 100% 113 in → 138 out (125 reasoning)
bugfix-123 feature/search
openai/gpt-5.4:xhigh__json_schema PASS 100% 159 in → 355 out (331 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.5:high PASS 100% 120 in → 104 out (91 reasoning)
bugfix-123 feature/search
openai/gpt-5.5:high__json_schema PASS 100% 155 in → 91 out (67 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.5:low PASS 100% 114 in → 67 out (54 reasoning)
bugfix-123 feature/search
openai/gpt-5.5:low__json_schema PASS 100% 158 in → 65 out (41 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.5:medium PASS 100% 116 in → 71 out (58 reasoning)
bugfix-123 feature/search
openai/gpt-5.5:medium__json_schema PASS 100% 159 in → 92 out (68 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.5:none PASS 100% 117 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.5:none__json_schema PASS 100% 158 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.5:xhigh PASS 100% 118 in → 108 out (95 reasoning)
bugfix-123 feature/search
openai/gpt-5.5:xhigh__json_schema PASS 100% 162 in → 540 out (516 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-luna:high PASS 100% 114 in → 60 out (47 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-luna:high__json_schema PASS 100% 157 in → 80 out (56 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-luna:low__json_schema PASS 100% 163 in → 44 out (20 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-luna:max PASS 100% 119 in → 107 out (94 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-luna:max__json_schema PASS 100% 155 in → 104 out (80 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-luna:medium PASS 100% 121 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-luna:medium__json_schema PASS 100% 160 in → 45 out (21 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-luna:none PASS 100% 119 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-luna:none__json_schema PASS 100% 160 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-luna:xhigh PASS 100% 116 in → 75 out (62 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 158 in → 63 out (39 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-sol:high PASS 100% 117 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-sol:high__json_schema PASS 100% 158 in → 63 out (39 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-sol:low PASS 100% 115 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-sol:low__json_schema PASS 100% 156 in → 63 out (39 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-sol:max PASS 100% 121 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-sol:max__json_schema PASS 100% 158 in → 109 out (85 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-sol:medium PASS 100% 118 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-sol:medium__json_schema PASS 100% 160 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-sol:none PASS 100% 115 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-sol:none__json_schema PASS 100% 156 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-sol:xhigh PASS 100% 114 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 158 in → 61 out (37 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-terra:high PASS 100% 111 in → 36 out (23 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-terra:high__json_schema PASS 100% 162 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-terra:low PASS 100% 114 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-terra:low__json_schema PASS 100% 163 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-terra:max PASS 100% 115 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-terra:max__json_schema PASS 100% 160 in → 139 out (115 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-terra:medium PASS 100% 115 in → 30 out (17 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-terra:medium__json_schema PASS 100% 158 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-terra:none PASS 100% 120 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-terra:none__json_schema PASS 100% 158 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-5.6-terra:xhigh PASS 100% 118 in → 11 out (0 reasoning)
bugfix-123 feature/search
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 160 in → 22 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-oss-120b:high PASS 100% 177 in → 320 out (335 reasoning)
bugfix-123 feature/search
openai/gpt-oss-120b:high__json_schema PASS 100% 174 in → 270 out (273 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
openai/gpt-oss-120b:low PASS 100% 174 in → 69 out (55 reasoning)
bugfix-123 feature/search
openai/gpt-oss-120b:low__json_schema PASS 100% 165 in → 106 out (86 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
openai/gpt-oss-120b:medium PASS 100% 175 in → 213 out (207 reasoning)
bugfix-123 feature/search
openai/gpt-oss-120b:medium__json_schema PASS 100% 178 in → 192 out (179 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-oss-120b:xhigh PASS 100% 175 in → 246 out (256 reasoning)
bugfix-123 feature/search
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 178 in → 490 out (523 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
openai/gpt-oss-20b:high PASS 100% 164 in → 230 out (230 reasoning)
bugfix-123 feature/search
openai/gpt-oss-20b:high__json_schema PASS 100% 175 in → 15 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-oss-20b:low PASS 100% 166 in → 50 out (37 reasoning)
bugfix-123 feature/search
openai/gpt-oss-20b:medium PASS 100% 184 in → 165 out (173 reasoning)
bugfix-123 feature/search
openai/gpt-oss-20b:medium__json_schema PASS 100% 176 in → 160 out (149 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
openai/gpt-oss-20b:xhigh PASS 100% 178 in → 170 out (163 reasoning)
bugfix-123 feature/search
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 176 in → 262 out (231 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
poolside/laguna-m.1:high PASS 100% 143 in → 342 out (330 reasoning)
bugfix-123 feature/search
poolside/laguna-m.1:high__json_schema PASS 100% 140 in → 308 out (289 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
poolside/laguna-m.1:low PASS 100% 142 in → 190 out (178 reasoning)
bugfix-123 feature/search
poolside/laguna-m.1:low__json_schema PASS 100% 145 in → 528 out (496 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
poolside/laguna-m.1:medium PASS 100% 142 in → 307 out (295 reasoning)
bugfix-123 feature/search
poolside/laguna-m.1:medium__json_schema PASS 100% 139 in → 330 out (300 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
poolside/laguna-m.1:none PASS 100% 145 in → 11 out (0 reasoning)
bugfix-123 feature/search
poolside/laguna-m.1:none__json_schema PASS 100% 144 in → 18 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
poolside/laguna-m.1:xhigh PASS 100% 143 in → 199 out (186 reasoning)
bugfix-123 feature/search
poolside/laguna-m.1:xhigh__json_schema PASS 100% 143 in → 214 out (190 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
poolside/laguna-xs-2.1:high PASS 100% 141 in → 173 out (163 reasoning)
bugfix-123 feature/search
poolside/laguna-xs-2.1:high__json_schema PASS 100% 144 in → 256 out (237 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
poolside/laguna-xs-2.1:low PASS 100% 140 in → 223 out (213 reasoning)
bugfix-123 feature/search
poolside/laguna-xs-2.1:low__json_schema PASS 100% 140 in → 363 out (344 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
poolside/laguna-xs-2.1:medium PASS 100% 146 in → 226 out (216 reasoning)
bugfix-123 feature/search
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 144 in → 348 out (329 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
poolside/laguna-xs-2.1:none PASS 100% 142 in → 9 out (0 reasoning)
bugfix-123 feature/search
poolside/laguna-xs-2.1:none__json_schema PASS 100% 147 in → 18 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
poolside/laguna-xs-2.1:xhigh PASS 100% 140 in → 168 out (158 reasoning)
bugfix-123 feature/search
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 138 in → 381 out (362 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
poolside/laguna-xs.2:high PASS 100% 178 in → 252 out (240 reasoning)
bugfix-123 feature/search
poolside/laguna-xs.2:high__json_schema PASS 100% 177 in → 257 out (233 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
poolside/laguna-xs.2:low PASS 100% 176 in → 279 out (267 reasoning)
bugfix-123 feature/search
poolside/laguna-xs.2:low__json_schema PASS 100% 184 in → 163 out (144 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
poolside/laguna-xs.2:medium PASS 100% 183 in → 212 out (200 reasoning)
bugfix-123 feature/search
poolside/laguna-xs.2:medium__json_schema PASS 100% 178 in → 240 out (221 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
poolside/laguna-xs.2:none PASS 100% 179 in → 11 out (0 reasoning)
bugfix-123 feature/search
poolside/laguna-xs.2:none__json_schema PASS 100% 178 in → 18 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
poolside/laguna-xs.2:xhigh PASS 100% 182 in → 123 out (111 reasoning)
bugfix-123 feature/search
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 180 in → 256 out (235 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123", "feature/search"]}
qwen/qwen3.6-27b:high PASS 100% 143 in → 839 out (825 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-27b:high__json_schema PASS 100% 141 in → 624 out (532 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
qwen/qwen3.6-27b:low PASS 100% 142 in → 1,074 out (1,004 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-27b:low__json_schema PASS 100% 138 in → 31 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
qwen/qwen3.6-27b:medium PASS 100% 137 in → 784 out (1 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-27b:medium__json_schema PASS 100% 139 in → 778 out (703 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
qwen/qwen3.6-27b:none PASS 100% 141 in → 10 out (0 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-27b:none__json_schema PASS 100% 136 in → 31 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 138 in → 1,018 out (928 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 140 in → 31 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 139 in → 833 out (784 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 133 in → 611 out (543 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 141 in → 796 out (704 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 138 in → 816 out (703 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 138 in → 859 out (845 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 137 in → 888 out (813 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 144 in → 9 out (0 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 140 in → 27 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 141 in → 663 out (640 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 140 in → 1,096 out (1,017 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
qwen/qwen3.6-flash:high PASS 100% 137 in → 317 out (303 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-flash:low PASS 100% 136 in → 735 out (721 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-flash:medium PASS 100% 138 in → 421 out (407 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-flash:none PASS 100% 141 in → 9 out (0 reasoning)
bugfix-123 feature/search
qwen/qwen3.6-flash:xhigh PASS 100% 135 in → 826 out (812 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-flash:high PASS 100% 140 in → 1,018 out (1,004 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-flash:low PASS 100% 138 in → 687 out (673 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-flash:max PASS 100% 141 in → 677 out (663 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-flash:medium PASS 100% 139 in → 956 out (942 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-flash:none PASS 100% 143 in → 9 out (0 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-flash:xhigh PASS 100% 137 in → 971 out (957 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-max:high PASS 100% 137 in → 213 out (199 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-max:low PASS 100% 137 in → 171 out (157 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-max:medium PASS 100% 136 in → 188 out (174 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-max:none PASS 100% 138 in → 9 out (0 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-max:xhigh PASS 100% 140 in → 168 out (154 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-plus:high PASS 100% 141 in → 516 out (502 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-plus:low PASS 100% 139 in → 530 out (516 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-plus:medium PASS 100% 142 in → 621 out (607 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-plus:none PASS 100% 139 in → 9 out (0 reasoning)
bugfix-123 feature/search
qwen/qwen3.7-plus:xhigh PASS 100% 141 in → 505 out (491 reasoning)
bugfix-123 feature/search
tencent/hy3:high PASS 100% 124 in → 311 out (302 reasoning)
bugfix-123 feature/search
tencent/hy3:high__json_schema PASS 100% 123 in → 549 out (568 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
tencent/hy3:low PASS 100% 124 in → 318 out (313 reasoning)
bugfix-123 feature/search
tencent/hy3:low__json_schema PASS 100% 126 in → 290 out (269 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
tencent/hy3:medium PASS 100% 127 in → 256 out (247 reasoning)
bugfix-123 feature/search
tencent/hy3:medium__json_schema PASS 100% 123 in → 374 out (353 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
tencent/hy3:none PASS 100% 129 in → 8 out (0 reasoning)
bugfix-123 feature/search
tencent/hy3:none__json_schema PASS 100% 126 in → 25 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
tencent/hy3:xhigh PASS 100% 123 in → 428 out (419 reasoning)
bugfix-123 feature/search
tencent/hy3:xhigh__json_schema PASS 100% 126 in → 426 out (424 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
thinkingmachines/inkling-small:high PASS 100% 127 in → 147 out (130 reasoning)
bugfix-123 feature/search
thinkingmachines/inkling-small:high__json_schema PASS 100% 125 in → 222 out (194 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
thinkingmachines/inkling-small:low PASS 100% 128 in → 52 out (45 reasoning)
bugfix-123 feature/search
thinkingmachines/inkling-small:low__json_schema PASS 100% 123 in → 56 out (27 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
thinkingmachines/inkling-small:max PASS 100% 130 in → 187 out (171 reasoning)
bugfix-123 feature/search
thinkingmachines/inkling-small:max__json_schema PASS 100% 128 in → 242 out (213 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
thinkingmachines/inkling-small:medium PASS 100% 127 in → 84 out (71 reasoning)
bugfix-123 feature/search
thinkingmachines/inkling-small:medium__json_schema PASS 100% 128 in → 71 out (42 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 124 in → 102 out (93 reasoning)
bugfix-123 feature/search
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 123 in → 161 out (132 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
thinkingmachines/inkling:high PASS 100% 125 in → 138 out (124 reasoning)
bugfix-123 feature/search
thinkingmachines/inkling:low PASS 100% 125 in → 42 out (34 reasoning)
bugfix-123 feature/search
thinkingmachines/inkling:low__json_schema PASS 100% 127 in → 47 out (19 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
thinkingmachines/inkling:max PASS 100% 125 in → 172 out (159 reasoning)
bugfix-123 feature/search
thinkingmachines/inkling:medium PASS 100% 123 in → 72 out (65 reasoning)
bugfix-123 feature/search
thinkingmachines/inkling:xhigh PASS 100% 128 in → 184 out (171 reasoning)
bugfix-123 feature/search
thinkingmachines/inkling:xhigh__json_schema PASS 100% 124 in → 152 out (124 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
x-ai/grok-4.3:high PASS 100% 292 in → 160 out (153 reasoning)
bugfix-123 feature/search
x-ai/grok-4.3:high__json_schema PASS 100% 356 in → 393 out (380 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
x-ai/grok-4.3:low PASS 100% 297 in → 343 out (336 reasoning)
bugfix-123 feature/search
x-ai/grok-4.3:low__json_schema PASS 100% 368 in → 357 out (344 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
x-ai/grok-4.3:max PASS 100% 297 in → 314 out (307 reasoning)
bugfix-123 feature/search
x-ai/grok-4.3:max__json_schema PASS 100% 363 in → 495 out (477 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
x-ai/grok-4.3:medium PASS 100% 307 in → 506 out (499 reasoning)
bugfix-123 feature/search
x-ai/grok-4.3:medium__json_schema PASS 100% 363 in → 429 out (416 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
x-ai/grok-4.3:none PASS 100% 289 in → 7 out (0 reasoning)
bugfix-123 feature/search
x-ai/grok-4.3:none__json_schema PASS 100% 362 in → 13 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
x-ai/grok-4.3:xhigh PASS 100% 293 in → 263 out (256 reasoning)
bugfix-123 feature/search
x-ai/grok-4.3:xhigh__json_schema PASS 100% 359 in → 467 out (449 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
x-ai/grok-4.5:high PASS 100% 318 in → 310 out (302 reasoning)
bugfix-123 feature/search
x-ai/grok-4.5:high__json_schema PASS 100% 396 in → 243 out (226 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
x-ai/grok-4.5:low PASS 100% 321 in → 287 out (279 reasoning)
bugfix-123 feature/search
x-ai/grok-4.5:low__json_schema PASS 100% 401 in → 350 out (333 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
x-ai/grok-4.5:max PASS 100% 321 in → 362 out (354 reasoning)
bugfix-123 feature/search
x-ai/grok-4.5:max__json_schema PASS 100% 397 in → 415 out (398 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
x-ai/grok-4.5:medium PASS 100% 321 in → 369 out (361 reasoning)
bugfix-123 feature/search
x-ai/grok-4.5:medium__json_schema PASS 100% 407 in → 478 out (461 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
x-ai/grok-4.5:xhigh PASS 100% 322 in → 369 out (361 reasoning)
bugfix-123 feature/search
x-ai/grok-4.5:xhigh__json_schema PASS 100% 404 in → 381 out (364 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) {"branches_to_delete":["bugfix-123","feature/search"]}
z-ai/glm-4.7-flash:high PASS 100% 122 in → 530 out (490 reasoning)
bugfix-123 feature/search
z-ai/glm-4.7-flash:high__json_schema PASS 100% 118 in → 440 out (387 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
z-ai/glm-4.7-flash:low PASS 100% 120 in → 600 out (568 reasoning)
bugfix-123 feature/search
z-ai/glm-4.7-flash:medium PASS 100% 119 in → 491 out (458 reasoning)
bugfix-123 feature/search
z-ai/glm-4.7-flash:xhigh PASS 100% 119 in → 551 out (508 reasoning)
bugfix-123 feature/search
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 121 in → 517 out (453 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
z-ai/glm-5.2:high PASS 100% 124 in → 335 out (326 reasoning)
bugfix-123 feature/search
z-ai/glm-5.2:low PASS 100% 126 in → 165 out (157 reasoning)
bugfix-123 feature/search
z-ai/glm-5.2:low__json_schema PASS 100% 128 in → 24 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
z-ai/glm-5.2:medium PASS 100% 127 in → 138 out (130 reasoning)
bugfix-123 feature/search
z-ai/glm-5.2:medium__json_schema PASS 100% 125 in → 290 out (263 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
z-ai/glm-5.2:none PASS 100% 118 in → 8 out (0 reasoning)
bugfix-123 feature/search
z-ai/glm-5.2:none__json_schema PASS 100% 124 in → 26 out (0 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "bugfix-123", "feature/search" ] }
z-ai/glm-5.2:xhigh PASS 100% 130 in → 430 out (423 reasoning)
bugfix-123 feature/search
z-ai/glm-5.2:xhigh__json_schema PASS 100% 127 in → 401 out (382 reasoning)
bugfix-123 feature/search
JSON Schema Structured Output
(raw) { "branches_to_delete": ["bugfix-123", "feature/search"] }
openai/gpt-5.6-luna:low FAIL 50% 112 in → 8 out (0 reasoning)
bugfix-123
Failure: Missing: ['feature/search']
openai/gpt-oss-20b:low__json_schema FAIL 50% 178 in → 14 out (0 reasoning)
bugfix-123
JSON Schema Structured Output
(raw) {"branches_to_delete": ["bugfix-123"]}
Failure: Missing: ['feature/search']
thinkingmachines/inkling-small:none FAIL 50% 126 in → 10 out (0 reasoning)
bugfix-123 feature/search<|end_message|>
Failure: Missing: ['feature/search'] Extra: ['feature/search<|end_message|>']
thinkingmachines/inkling:none FAIL 50% 126 in → 10 out (0 reasoning)
bugfix-123 feature/search<|end_message|>
Failure: Missing: ['feature/search'] Extra: ['feature/search<|end_message|>']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
experiment-ml
JSON Schema Structured Output
(raw) { "branches_to_delete": ["experiment-ml"] }
Failure: Missing: ['bugfix-123', 'feature/search'] Extra: ['experiment-ml']
Invalid JSON. Output: bugfix-123 feature/search
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e20676de-8a42-9c78-8c66-280aa57312dd","request_id":"e20676de-8a42-9c78-8c66-280aa57312dd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c67ff1f3-acc2-9ef9-822d-639d1dc91690","request_id":"c67ff1f3-acc2-9ef9-822d-639d1dc91690"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-64d549fe-0e3e-9efd-925b-aa42663409a7","request_id":"64d549fe-0e3e-9efd-925b-aa42663409a7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-da1453cc-5b82-9ffa-a849-b8f67ba70ade","request_id":"da1453cc-5b82-9ffa-a849-b8f67ba70ade"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-143eb21f-17ce-9729-8043-236ec0974684","request_id":"143eb21f-17ce-9729-8043-236ec0974684"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c616923f-3519-95c5-a28a-6d224e6b5654"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7bdee6b5-abda-9082-a9d5-2a5992748f0f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4df6d6d0-7041-9ac1-8e3b-a3583158efc9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-21d05a2a-6125-9d22-823b-e6e40460f230"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2ecb21e7-7bd0-9b08-89e3-210f480bab6c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2e506caa-2dfa-9763-9bb1-10ed1a965436"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ade322fb-9fce-935d-b389-3fc123013e1e","request_id":"ade322fb-9fce-935d-b389-3fc123013e1e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0d49b6de-4cda-9695-951d-8e2b097105f7","request_id":"0d49b6de-4cda-9695-951d-8e2b097105f7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4c701e8a-b4e9-9777-8ef8-a5d22cdc66d5","request_id":"4c701e8a-b4e9-9777-8ef8-a5d22cdc66d5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0ef2b7dc-23d1-9764-82c5-0501a1d5648c","request_id":"0ef2b7dc-23d1-9764-82c5-0501a1d5648c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-799c648c-6c76-907a-9d1c-3d84c88a30c1","request_id":"799c648c-6c76-907a-9d1c-3d84c88a30c1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0a77239a-3dac-9201-ab1b-6871ef708a92","request_id":"0a77239a-3dac-9201-ab1b-6871ef708a92"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b6893ed1-0cc4-97ea-a47c-1a9712cabb28","request_id":"b6893ed1-0cc4-97ea-a47c-1a9712cabb28"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3ad9d377-1a65-9e63-8800-7e67645c158c","request_id":"3ad9d377-1a65-9e63-8800-7e67645c158c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4a6f100d-5a87-97b4-b0e9-9ed6c2213fec","request_id":"4a6f100d-5a87-97b4-b0e9-9ed6c2213fec"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-95b4b37c-5960-9834-a668-8b51af4d564f","request_id":"95b4b37c-5960-9834-a668-8b51af4d564f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: bugfix-123 feature/search{"branches_to_delete": ["bugfix-123", "feature/search"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: bugfix-123 feature/search
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: bugfix-123 feature/search
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: bugfix-123 feature/search
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: bugfix-123 feature/search{ "branches_to_delete": [ "bugfix-123", "feature/search" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: bugfix-123 feature/search
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: bugfix-123 feature/search
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 117 in → 4 out (0 reasoning)
experiment-ml
Failure: Missing: ['bugfix-123', 'feature/search'] Extra: ['experiment-ml']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 125 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['bugfix-123', 'feature/search']
z-ai/glm-5.2:high__json_schema FAIL 0% 131 in → 321 out (306 reasoning)
experiment-ml
JSON Schema Structured Output
(raw) { "branches_to_delete": ["experiment-ml"] }
Failure: Missing: ['bugfix-123', 'feature/search'] Extra: ['experiment-ml']