Release branches
Tests ability to identify merged release branches for cleanup. Evaluates release-branch lifecycle management.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'v1' > version.txt
  5. 05 git add version.txt
  6. 06 git commit -m 'v1.0'
  7. 07 git checkout -b release-1.0
  8. 08 echo 'v1.0' > release.txt
  9. 09 git add release.txt
  10. 10 git commit -m 'Release 1.0'
  11. 11 git checkout main
  12. 12 git merge release-1.0
  13. 13 git checkout -b release-2.0
  14. 14 echo 'v2.0' > release.txt
  15. 15 git add release.txt
  16. 16 git commit -m 'Release 2.0'
  17. 17 git checkout main
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
release-1.0

Scoped model quality, cost, API time, and token usage for branch_cleanup/f010.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high FAIL 100% 136 in → 156 out (47 reasoning)
release-1.0 **Reasoning:** The `release-1.0` branch points to the same commit as `main` (131591e) and appears in the "merged into main" list, so it's fully merged and safe to delete. The `release-2.0` branch (3a112c0) is not merged into main, so deleting it would lose work. `main` itself is never deleted.
Failure: Extra: ['**Reasoning:** The `release-1.0` branch points to the same commit as `main` (131591e) and appears in the "merged into main" list, so it\'s fully merged and safe to delete. The `release-2.0` branch (3a112c0) is not merged into main, so deleting it would lose work. `main` itself is never deleted.']
anthropic/claude-fable-5:high__json_schema PASS 100% 379 in → 49 out (29 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
anthropic/claude-fable-5:low FAIL 100% 139 in → 99 out (14 reasoning)
release-1.0 **Reasoning:** `release-1.0` points to the same commit as `main` (9be69ff) and appears in the "merged into main" list, so it's safe to delete. `release-2.0` (0da1274) is not merged into main, so deleting it would lose work.
Failure: Extra: ['**Reasoning:** `release-1.0` points to the same commit as `main` (9be69ff) and appears in the "merged into main" list, so it\'s safe to delete. `release-2.0` (0da1274) is not merged into main, so deleting it would lose work.']
anthropic/claude-fable-5:low__json_schema PASS 100% 380 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-fable-5:max PASS 100% 138 in → 941 out (149 reasoning)
release-1.0
anthropic/claude-fable-5:max__json_schema PASS 100% 378 in → 204 out (49 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
anthropic/claude-fable-5:medium FAIL 100% 141 in → 129 out (36 reasoning)
release-1.0 **Reasoning:** The `git branch --merged` output shows only `release-1.0` is fully merged into main (it points to the same commit, 9853a88). `release-2.0` contains commit de6c6e1 which is not in main, so it should not be deleted.
Failure: Extra: ['**Reasoning:** The `git branch --merged` output shows only `release-1.0` is fully merged into main (it points to the same commit, 9853a88). `release-2.0` contains commit de6c6e1 which is not in main, so it should not be deleted.']
anthropic/claude-fable-5:medium__json_schema PASS 100% 376 in → 34 out (10 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
anthropic/claude-fable-5:xhigh PASS 100% 141 in → 70 out (41 reasoning)
release-1.0
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 379 in → 54 out (52 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
anthropic/claude-haiku-4.5:high PASS 100% 152 in → 187 out (134 reasoning)
release-1.0
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 339 in → 400 out (339 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
anthropic/claude-haiku-4.5:low PASS 100% 150 in → 157 out (114 reasoning)
release-1.0
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 337 in → 295 out (240 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
anthropic/claude-haiku-4.5:medium PASS 100% 151 in → 191 out (156 reasoning)
release-1.0
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 340 in → 430 out (363 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
anthropic/claude-haiku-4.5:none PASS 100% 123 in → 8 out (0 reasoning)
release-1.0
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 306 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 146 in → 191 out (153 reasoning)
release-1.0
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 337 in → 394 out (334 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
anthropic/claude-opus-4.6:high PASS 100% 125 in → 66 out (34 reasoning)
release-1.0
anthropic/claude-opus-4.6:high__json_schema PASS 100% 313 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.6:low PASS 100% 118 in → 8 out (0 reasoning)
release-1.0
anthropic/claude-opus-4.6:low__json_schema PASS 100% 311 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.6:max PASS 100% 116 in → 143 out (120 reasoning)
release-1.0
anthropic/claude-opus-4.6:max__json_schema PASS 100% 309 in → 125 out (87 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.6:medium PASS 100% 120 in → 8 out (0 reasoning)
release-1.0
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 308 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.6:none PASS 100% 122 in → 8 out (0 reasoning)
release-1.0
anthropic/claude-opus-4.6:none__json_schema PASS 100% 307 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 119 in → 110 out (88 reasoning)
release-1.0
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 312 in → 73 out (32 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.7:high PASS 100% 145 in → 9 out (0 reasoning)
release-1.0
anthropic/claude-opus-4.7:high__json_schema PASS 100% 382 in → 23 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.7:low PASS 100% 145 in → 9 out (0 reasoning)
release-1.0
anthropic/claude-opus-4.7:low__json_schema PASS 100% 383 in → 23 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.7:max PASS 100% 147 in → 485 out (79 reasoning)
release-1.0
anthropic/claude-opus-4.7:max__json_schema PASS 100% 389 in → 23 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.7:medium PASS 100% 146 in → 9 out (0 reasoning)
release-1.0
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 387 in → 23 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.7:none PASS 100% 149 in → 9 out (0 reasoning)
release-1.0
anthropic/claude-opus-4.7:none__json_schema PASS 100% 385 in → 23 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 146 in → 9 out (0 reasoning)
release-1.0
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 385 in → 23 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.8:high PASS 100% 137 in → 84 out (61 reasoning)
release-1.0
anthropic/claude-opus-4.8:high__json_schema PASS 100% 378 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.8:low PASS 100% 135 in → 39 out (34 reasoning)
release-1.0
anthropic/claude-opus-4.8:low__json_schema PASS 100% 379 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.8:max PASS 100% 142 in → 471 out (48 reasoning)
release-1.0
anthropic/claude-opus-4.8:max__json_schema PASS 100% 383 in → 257 out (74 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.8:medium PASS 100% 137 in → 90 out (68 reasoning)
release-1.0
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 380 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.8:none PASS 100% 139 in → 7 out (0 reasoning)
release-1.0
anthropic/claude-opus-4.8:none__json_schema PASS 100% 384 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 136 in → 177 out (47 reasoning)
release-1.0
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 377 in → 22 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
anthropic/claude-sonnet-4.6:high PASS 100% 122 in → 112 out (87 reasoning)
release-1.0
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 310 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-sonnet-4.6:low PASS 100% 123 in → 8 out (0 reasoning)
release-1.0
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 306 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-sonnet-4.6:max PASS 100% 115 in → 112 out (91 reasoning)
release-1.0
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 311 in → 118 out (82 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 121 in → 8 out (0 reasoning)
release-1.0
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 313 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-sonnet-4.6:none PASS 100% 123 in → 8 out (0 reasoning)
release-1.0
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 309 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 124 in → 105 out (78 reasoning)
release-1.0
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 308 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-sonnet-5:high PASS 100% 138 in → 34 out (21 reasoning)
release-1.0
anthropic/claude-sonnet-5:high__json_schema PASS 100% 383 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-sonnet-5:low PASS 100% 143 in → 7 out (0 reasoning)
release-1.0
anthropic/claude-sonnet-5:low__json_schema PASS 100% 380 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-sonnet-5:medium PASS 100% 142 in → 7 out (0 reasoning)
release-1.0
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 378 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-sonnet-5:none FAIL 100% 145 in → 11 out (0 reasoning)
``` release-1.0 ```
Failure: Extra: ['```']
anthropic/claude-sonnet-5:none__json_schema PASS 100% 379 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
anthropic/claude-sonnet-5:xhigh FAIL 100% 141 in → 18 out (0 reasoning)
Based on the merge status provided: release-1.0
Failure: Extra: ['Based on the merge status provided:']
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 378 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
arcee-ai/trinity-large-thinking:high PASS 100% 115 in → 736 out (736 reasoning)
release-1.0
arcee-ai/trinity-large-thinking:low PASS 100% 114 in → 755 out (752 reasoning)
release-1.0
arcee-ai/trinity-large-thinking:medium PASS 100% 114 in → 591 out (587 reasoning)
release-1.0
arcee-ai/trinity-large-thinking:xhigh PASS 100% 113 in → 505 out (500 reasoning)
release-1.0
arcee-ai/trinity-mini:high PASS 100% 115 in → 232 out (233 reasoning)
release-1.0
arcee-ai/trinity-mini:high__json_schema PASS 100% 112 in → 267 out (264 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
arcee-ai/trinity-mini:low PASS 100% 112 in → 228 out (232 reasoning)
release-1.0
arcee-ai/trinity-mini:low__json_schema PASS 100% 115 in → 377 out (364 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
arcee-ai/trinity-mini:medium PASS 100% 112 in → 376 out (393 reasoning)
release-1.0
arcee-ai/trinity-mini:medium__json_schema PASS 100% 114 in → 567 out (562 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 117 in → 279 out (274 reasoning)
release-1.0
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 118 in → 397 out (384 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 192 in → 175 out (157 reasoning)
release-1.0
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 111 in → 165 out (134 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 119 in → 384 out (371 reasoning)
release-1.0
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 114 in → 317 out (265 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 188 in → 113 out (115 reasoning)
release-1.0
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 205 in → 254 out (218 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 112 in → 497 out (490 reasoning)
release-1.0
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 115 in → 80 out (70 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 111 in → 6 out (0 reasoning)
release-1.0
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 113 in → 17 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"]}
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 190 in → 132 out (122 reasoning)
release-1.0
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 269 in → 522 out (518 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
deepseek/deepseek-v4-flash:high PASS 100% 111 in → 108 out (101 reasoning)
release-1.0
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 110 in → 114 out (88 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
deepseek/deepseek-v4-flash:low PASS 100% 112 in → 129 out (117 reasoning)
release-1.0
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 113 in → 318 out (272 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
deepseek/deepseek-v4-flash:medium PASS 100% 115 in → 91 out (84 reasoning)
release-1.0
deepseek/deepseek-v4-flash:none PASS 100% 109 in → 6 out (0 reasoning)
release-1.0
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 109 in → 17 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
deepseek/deepseek-v4-flash:xhigh PASS 100% 116 in → 100 out (95 reasoning)
release-1.0
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 217 in → 15 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
deepseek/deepseek-v4-pro:high PASS 100% 108 in → 261 out (230 reasoning)
release-1.0
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 112 in → 336 out (318 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
deepseek/deepseek-v4-pro:low PASS 100% 112 in → 185 out (178 reasoning)
release-1.0
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 113 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
deepseek/deepseek-v4-pro:medium PASS 100% 111 in → 134 out (121 reasoning)
release-1.0
deepseek/deepseek-v4-pro:none PASS 100% 110 in → 6 out (0 reasoning)
release-1.0
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 115 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 116 in → 336 out (329 reasoning)
release-1.0
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 390 in → 554 out (538 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemini-3-flash-preview:high PASS 100% 115 in → 239 out (234 reasoning)
release-1.0
google/gemini-3-flash-preview:high__json_schema PASS 100% 113 in → 437 out (424 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3-flash-preview:low PASS 100% 106 in → 295 out (290 reasoning)
release-1.0
google/gemini-3-flash-preview:low__json_schema PASS 100% 111 in → 273 out (260 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3-flash-preview:medium PASS 100% 111 in → 236 out (231 reasoning)
release-1.0
google/gemini-3-flash-preview:medium__json_schema PASS 100% 195 in → 271 out (258 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3-flash-preview:xhigh PASS 100% 113 in → 258 out (253 reasoning)
release-1.0
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 190 in → 280 out (267 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 113 in → 334 out (329 reasoning)
release-1.0
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 194 in → 534 out (520 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 110 in → 135 out (130 reasoning)
release-1.0
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 112 in → 148 out (123 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 113 in → 233 out (228 reasoning)
release-1.0
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 109 in → 254 out (240 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 115 in → 437 out (432 reasoning)
release-1.0
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 195 in → 411 out (397 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemini-3.1-pro-preview:high PASS 100% 111 in → 541 out (536 reasoning)
release-1.0
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 195 in → 429 out (416 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.1-pro-preview:low PASS 100% 114 in → 224 out (219 reasoning)
release-1.0
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 192 in → 323 out (310 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.1-pro-preview:medium PASS 100% 115 in → 290 out (285 reasoning)
release-1.0
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 196 in → 126 out (112 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 114 in → 220 out (215 reasoning)
release-1.0
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 191 in → 352 out (339 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.5-flash-lite:high PASS 100% 113 in → 322 out (317 reasoning)
release-1.0
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 196 in → 368 out (355 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.5-flash-lite:low PASS 100% 111 in → 5 out (0 reasoning)
release-1.0
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 189 in → 25 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
google/gemini-3.5-flash-lite:max PASS 100% 112 in → 324 out (319 reasoning)
release-1.0
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 109 in → 318 out (304 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemini-3.5-flash-lite:medium PASS 100% 114 in → 285 out (280 reasoning)
release-1.0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 112 in → 313 out (299 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 110 in → 337 out (332 reasoning)
release-1.0
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 113 in → 356 out (342 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemini-3.5-flash:high PASS 100% 107 in → 291 out (286 reasoning)
release-1.0
google/gemini-3.5-flash:high__json_schema PASS 100% 196 in → 313 out (300 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.5-flash:low PASS 100% 112 in → 224 out (219 reasoning)
release-1.0
google/gemini-3.5-flash:low__json_schema PASS 100% 194 in → 158 out (145 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.5-flash:medium PASS 100% 111 in → 413 out (408 reasoning)
release-1.0
google/gemini-3.5-flash:medium__json_schema PASS 100% 195 in → 276 out (251 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
google/gemini-3.5-flash:xhigh PASS 100% 114 in → 354 out (349 reasoning)
release-1.0
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 189 in → 533 out (520 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.6-flash:high PASS 100% 112 in → 282 out (277 reasoning)
release-1.0
google/gemini-3.6-flash:high__json_schema PASS 100% 194 in → 503 out (490 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.6-flash:low PASS 100% 113 in → 101 out (96 reasoning)
release-1.0
google/gemini-3.6-flash:low__json_schema PASS 100% 110 in → 25 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
google/gemini-3.6-flash:max PASS 100% 114 in → 389 out (384 reasoning)
release-1.0
google/gemini-3.6-flash:max__json_schema PASS 100% 195 in → 453 out (440 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.6-flash:medium PASS 100% 116 in → 291 out (286 reasoning)
release-1.0
google/gemini-3.6-flash:medium__json_schema PASS 100% 113 in → 236 out (223 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemini-3.6-flash:xhigh PASS 100% 114 in → 256 out (251 reasoning)
release-1.0
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 111 in → 434 out (421 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
google/gemma-4-26b-a4b-it:high PASS 100% 127 in → 242 out (173 reasoning)
release-1.0
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 127 in → 317 out (228 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 127 in → 315 out (262 reasoning)
release-1.0
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 126 in → 293 out (198 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 128 in → 416 out (340 reasoning)
release-1.0
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 128 in → 277 out (195 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 125 in → 6 out (0 reasoning)
release-1.0
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 128 in → 15 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemma-4-26b-a4b-it:xhigh PASS 100% 128 in → 315 out (256 reasoning)
release-1.0
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 129 in → 324 out (232 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
google/gemma-4-31b-it:high PASS 100% 128 in → 222 out (171 reasoning)
release-1.0
google/gemma-4-31b-it:high__json_schema PASS 100% 124 in → 180 out (139 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemma-4-31b-it:low PASS 100% 129 in → 244 out (1 reasoning)
release-1.0
google/gemma-4-31b-it:low__json_schema PASS 100% 130 in → 261 out (190 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
google/gemma-4-31b-it:medium PASS 100% 129 in → 202 out (171 reasoning)
release-1.0
google/gemma-4-31b-it:medium__json_schema PASS 100% 131 in → 210 out (1 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
google/gemma-4-31b-it:none PASS 100% 127 in → 6 out (0 reasoning)
release-1.0
google/gemma-4-31b-it:none__json_schema PASS 100% 127 in → 15 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
google/gemma-4-31b-it:xhigh PASS 100% 130 in → 179 out (1 reasoning)
release-1.0
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 128 in → 239 out (178 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
inclusionai/ling-2.6-flash PASS 100% 138 in → 6 out
release-1.0
inclusionai/ling-2.6-flash__json_schema PASS 100% 136 in → 25 out
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
minimax/minimax-m2.5:high PASS 100% 153 in → 204 out (197 reasoning)
release-1.0
minimax/minimax-m2.5:high__json_schema PASS 100% 150 in → 237 out (210 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
minimax/minimax-m2.5:low PASS 100% 145 in → 388 out (381 reasoning)
release-1.0
minimax/minimax-m2.5:low__json_schema PASS 100% 146 in → 483 out (464 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
minimax/minimax-m2.5:medium PASS 100% 144 in → 446 out (453 reasoning)
release-1.0
minimax/minimax-m2.5:medium__json_schema PASS 100% 145 in → 678 out (681 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
minimax/minimax-m2.5:xhigh PASS 100% 148 in → 440 out (449 reasoning)
release-1.0
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 147 in → 636 out (623 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
minimax/minimax-m2.7:high PASS 100% 144 in → 244 out (237 reasoning)
release-1.0
minimax/minimax-m2.7:high__json_schema PASS 100% 296 in → 461 out (446 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
minimax/minimax-m2.7:low PASS 100% 151 in → 399 out (393 reasoning)
release-1.0
minimax/minimax-m2.7:low__json_schema PASS 100% 146 in → 17 out (17 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
minimax/minimax-m2.7:medium PASS 100% 148 in → 124 out (118 reasoning)
release-1.0
minimax/minimax-m2.7:medium__json_schema PASS 100% 297 in → 352 out (337 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
minimax/minimax-m2.7:xhigh PASS 100% 145 in → 230 out (222 reasoning)
release-1.0
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 297 in → 446 out (431 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
minimax/minimax-m3:high PASS 100% 282 in → 197 out (184 reasoning)
release-1.0
minimax/minimax-m3:high__json_schema PASS 100% 229 in → 17 out (16 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
minimax/minimax-m3:low PASS 100% 284 in → 155 out (140 reasoning)
release-1.0
minimax/minimax-m3:low__json_schema PASS 100% 269 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
minimax/minimax-m3:medium PASS 100% 280 in → 162 out (157 reasoning)
release-1.0
minimax/minimax-m3:xhigh PASS 100% 284 in → 137 out (117 reasoning)
release-1.0
minimax/minimax-m3:xhigh__json_schema PASS 100% 230 in → 17 out (16 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
mistralai/devstral-2512 PASS 100% 118 in → 6 out
release-1.0
mistralai/devstral-2512__json_schema PASS 100% 118 in → 15 out
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
mistralai/mistral-medium-3-5:high PASS 100% 125 in → 235 out (198 reasoning)
release-1.0
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 116 in → 228 out (186 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
mistralai/mistral-medium-3-5:low PASS 100% 125 in → 629 out (589 reasoning)
release-1.0
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 118 in → 179 out (143 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
mistralai/mistral-medium-3-5:medium PASS 100% 130 in → 311 out (271 reasoning)
release-1.0
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 115 in → 365 out (340 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
mistralai/mistral-medium-3-5:none PASS 100% 129 in → 6 out (0 reasoning)
release-1.0
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 115 in → 15 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 129 in → 465 out (419 reasoning)
release-1.0
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 115 in → 219 out (170 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
moonshotai/kimi-k2.6:high PASS 100% 112 in → 210 out (203 reasoning)
release-1.0
moonshotai/kimi-k2.6:high__json_schema PASS 100% 302 in → 656 out (641 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
moonshotai/kimi-k2.6:low PASS 100% 116 in → 244 out (238 reasoning)
release-1.0
moonshotai/kimi-k2.6:low__json_schema PASS 100% 111 in → 418 out (213 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
moonshotai/kimi-k2.6:medium PASS 100% 108 in → 262 out (258 reasoning)
release-1.0
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 111 in → 525 out (238 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
moonshotai/kimi-k2.6:xhigh PASS 100% 112 in → 542 out (535 reasoning)
release-1.0
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 115 in → 17 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
moonshotai/kimi-k2.7-code:high PASS 100% 114 in → 161 out (154 reasoning)
release-1.0
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 205 in → 232 out (216 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
moonshotai/kimi-k2.7-code:low PASS 100% 112 in → 180 out (173 reasoning)
release-1.0
moonshotai/kimi-k2.7-code:medium PASS 100% 116 in → 204 out (197 reasoning)
release-1.0
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 111 in → 177 out (167 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 114 in → 119 out (112 reasoning)
release-1.0
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 113 in → 209 out (191 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
moonshotai/kimi-k3:high PASS 100% 188 in → 159 out (139 reasoning)
release-1.0
moonshotai/kimi-k3:high__json_schema PASS 100% 188 in → 149 out (129 reasoning)
release-1.0
moonshotai/kimi-k3:low PASS 100% 187 in → 180 out (160 reasoning)
release-1.0
moonshotai/kimi-k3:low__json_schema PASS 100% 191 in → 163 out (143 reasoning)
release-1.0
moonshotai/kimi-k3:max PASS 100% 189 in → 145 out (125 reasoning)
release-1.0
moonshotai/kimi-k3:max__json_schema PASS 100% 190 in → 196 out (176 reasoning)
release-1.0
moonshotai/kimi-k3:medium PASS 100% 191 in → 165 out (145 reasoning)
release-1.0
moonshotai/kimi-k3:medium__json_schema PASS 100% 191 in → 164 out (144 reasoning)
release-1.0
moonshotai/kimi-k3:xhigh PASS 100% 191 in → 152 out (132 reasoning)
release-1.0
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 188 in → 190 out (170 reasoning)
release-1.0
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 127 in → 241 out (216 reasoning)
release-1.0
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 129 in → 314 out (275 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 131 in → 285 out (253 reasoning)
release-1.0
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 128 in → 118 out (90 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 125 in → 188 out (163 reasoning)
release-1.0
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 128 in → 190 out (166 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 128 in → 6 out (0 reasoning)
release-1.0
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 128 in → 15 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 131 in → 312 out (261 reasoning)
release-1.0
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 133 in → 168 out (138 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 127 in → 151 out (119 reasoning)
release-1.0
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 129 in → 127 out (110 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 130 in → 119 out (110 reasoning)
release-1.0
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 133 in → 71 out (53 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 127 in → 139 out (131 reasoning)
release-1.0
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 130 in → 121 out (102 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 131 in → 6 out (0 reasoning)
release-1.0
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 132 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 130 in → 137 out (128 reasoning)
release-1.0
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 131 in → 133 out (110 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
openai/gpt-5.4-mini:high PASS 100% 112 in → 76 out (65 reasoning)
release-1.0
openai/gpt-5.4-mini:high__json_schema PASS 100% 151 in → 321 out (299 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4-mini:low PASS 100% 111 in → 49 out (38 reasoning)
release-1.0
openai/gpt-5.4-mini:low__json_schema PASS 100% 151 in → 63 out (41 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4-mini:medium PASS 100% 106 in → 75 out (64 reasoning)
release-1.0
openai/gpt-5.4-mini:medium__json_schema PASS 100% 155 in → 145 out (123 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4-mini:none PASS 100% 110 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.4-mini:none__json_schema PASS 100% 152 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4-mini:xhigh PASS 100% 107 in → 151 out (140 reasoning)
release-1.0
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 152 in → 365 out (343 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4-nano:high PASS 100% 107 in → 63 out (52 reasoning)
release-1.0
openai/gpt-5.4-nano:high__json_schema PASS 100% 151 in → 124 out (102 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4-nano:low PASS 100% 111 in → 44 out (33 reasoning)
release-1.0
openai/gpt-5.4-nano:low__json_schema PASS 100% 151 in → 55 out (33 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4-nano:medium PASS 100% 112 in → 62 out (51 reasoning)
release-1.0
openai/gpt-5.4-nano:medium__json_schema PASS 100% 152 in → 57 out (35 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4-nano:none PASS 100% 113 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.4-nano:none__json_schema PASS 100% 155 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4-nano:xhigh PASS 100% 111 in → 159 out (148 reasoning)
release-1.0
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 154 in → 482 out (460 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4:high PASS 100% 107 in → 92 out (81 reasoning)
release-1.0
openai/gpt-5.4:high__json_schema PASS 100% 156 in → 127 out (105 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4:low PASS 100% 112 in → 38 out (27 reasoning)
release-1.0
openai/gpt-5.4:low__json_schema PASS 100% 157 in → 50 out (28 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4:medium PASS 100% 109 in → 92 out (81 reasoning)
release-1.0
openai/gpt-5.4:medium__json_schema PASS 100% 156 in → 99 out (77 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4:none PASS 100% 110 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.4:none__json_schema PASS 100% 157 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.4:xhigh PASS 100% 112 in → 116 out (105 reasoning)
release-1.0
openai/gpt-5.4:xhigh__json_schema PASS 100% 154 in → 115 out (93 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.5:high PASS 100% 116 in → 82 out (71 reasoning)
release-1.0
openai/gpt-5.5:high__json_schema PASS 100% 154 in → 428 out (406 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.5:low PASS 100% 109 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.5:low__json_schema PASS 100% 152 in → 70 out (48 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.5:medium PASS 100% 109 in → 77 out (66 reasoning)
release-1.0
openai/gpt-5.5:medium__json_schema PASS 100% 151 in → 73 out (51 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.5:none PASS 100% 107 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.5:none__json_schema PASS 100% 153 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.5:xhigh PASS 100% 115 in → 96 out (85 reasoning)
release-1.0
openai/gpt-5.5:xhigh__json_schema PASS 100% 157 in → 126 out (104 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-luna:high PASS 100% 110 in → 45 out (34 reasoning)
release-1.0
openai/gpt-5.6-luna:high__json_schema PASS 100% 153 in → 174 out (152 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-luna:low PASS 100% 111 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.6-luna:low__json_schema PASS 100% 154 in → 45 out (23 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-luna:max PASS 100% 112 in → 90 out (79 reasoning)
release-1.0
openai/gpt-5.6-luna:max__json_schema PASS 100% 161 in → 315 out (293 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-luna:medium PASS 100% 110 in → 40 out (29 reasoning)
release-1.0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 155 in → 45 out (23 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-luna:none PASS 100% 110 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.6-luna:none__json_schema PASS 100% 154 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-luna:xhigh PASS 100% 109 in → 46 out (35 reasoning)
release-1.0
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 155 in → 47 out (25 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-sol:high PASS 100% 110 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.6-sol:high__json_schema PASS 100% 151 in → 82 out (60 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-sol:low PASS 100% 110 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.6-sol:low__json_schema PASS 100% 154 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-sol:max PASS 100% 111 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.6-sol:max__json_schema PASS 100% 150 in → 127 out (105 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-sol:medium PASS 100% 110 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.6-sol:medium__json_schema PASS 100% 153 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-sol:none PASS 100% 110 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.6-sol:none__json_schema PASS 100% 157 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-sol:xhigh PASS 100% 110 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 155 in → 130 out (108 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-terra:high PASS 100% 112 in → 28 out (17 reasoning)
release-1.0
openai/gpt-5.6-terra:high__json_schema PASS 100% 157 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-terra:low PASS 100% 113 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.6-terra:low__json_schema PASS 100% 154 in → 36 out (14 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-terra:max PASS 100% 107 in → 72 out (61 reasoning)
release-1.0
openai/gpt-5.6-terra:max__json_schema PASS 100% 153 in → 88 out (66 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-terra:medium PASS 100% 111 in → 25 out (14 reasoning)
release-1.0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 154 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-terra:none PASS 100% 115 in → 9 out (0 reasoning)
release-1.0
openai/gpt-5.6-terra:none__json_schema PASS 100% 153 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-5.6-terra:xhigh PASS 100% 112 in → 32 out (21 reasoning)
release-1.0
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 156 in → 36 out (14 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-oss-120b:high PASS 100% 160 in → 292 out (260 reasoning)
release-1.0
openai/gpt-oss-120b:high__json_schema PASS 100% 169 in → 290 out (261 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
openai/gpt-oss-120b:low__json_schema PASS 100% 174 in → 108 out (66 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
openai/gpt-oss-120b:medium PASS 100% 170 in → 141 out (105 reasoning)
release-1.0
openai/gpt-oss-120b:medium__json_schema PASS 100% 170 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
openai/gpt-oss-120b:xhigh PASS 100% 177 in → 127 out (122 reasoning)
release-1.0
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 168 in → 410 out (387 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
openai/gpt-oss-20b:high PASS 100% 168 in → 357 out (360 reasoning)
release-1.0
openai/gpt-oss-20b:high__json_schema PASS 100% 169 in → 734 out (690 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
openai/gpt-oss-20b:low PASS 100% 170 in → 30 out (15 reasoning)
release-1.0
openai/gpt-oss-20b:low__json_schema PASS 100% 172 in → 58 out (33 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
openai/gpt-oss-20b:medium PASS 100% 158 in → 194 out (173 reasoning)
release-1.0
openai/gpt-oss-20b:medium__json_schema PASS 100% 170 in → 15 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
openai/gpt-oss-20b:xhigh PASS 100% 171 in → 318 out (317 reasoning)
release-1.0
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 170 in → 427 out (412 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
poolside/laguna-m.1:high PASS 100% 130 in → 356 out (348 reasoning)
release-1.0
poolside/laguna-m.1:high__json_schema PASS 100% 128 in → 326 out (307 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
poolside/laguna-m.1:low PASS 100% 126 in → 175 out (167 reasoning)
release-1.0
poolside/laguna-m.1:low__json_schema PASS 100% 129 in → 193 out (179 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
poolside/laguna-m.1:medium PASS 100% 128 in → 136 out (128 reasoning)
release-1.0
poolside/laguna-m.1:medium__json_schema PASS 100% 128 in → 244 out (230 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
poolside/laguna-m.1:none PASS 100% 129 in → 7 out (0 reasoning)
release-1.0
poolside/laguna-m.1:none__json_schema PASS 100% 129 in → 13 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
poolside/laguna-m.1:xhigh PASS 100% 126 in → 147 out (139 reasoning)
release-1.0
poolside/laguna-m.1:xhigh__json_schema PASS 100% 129 in → 432 out (414 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
poolside/laguna-xs-2.1:high PASS 100% 122 in → 270 out (264 reasoning)
release-1.0
poolside/laguna-xs-2.1:high__json_schema PASS 100% 129 in → 181 out (156 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
poolside/laguna-xs-2.1:low PASS 100% 130 in → 184 out (178 reasoning)
release-1.0
poolside/laguna-xs-2.1:low__json_schema PASS 100% 128 in → 221 out (195 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
poolside/laguna-xs-2.1:medium PASS 100% 127 in → 214 out (208 reasoning)
release-1.0
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 126 in → 567 out (541 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
poolside/laguna-xs-2.1:none PASS 100% 126 in → 5 out (0 reasoning)
release-1.0
poolside/laguna-xs-2.1:none__json_schema PASS 100% 127 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
poolside/laguna-xs-2.1:xhigh PASS 100% 125 in → 298 out (292 reasoning)
release-1.0
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 127 in → 304 out (279 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
poolside/laguna-xs.2:high PASS 100% 161 in → 131 out (123 reasoning)
release-1.0
poolside/laguna-xs.2:high__json_schema PASS 100% 167 in → 145 out (131 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
poolside/laguna-xs.2:low PASS 100% 163 in → 129 out (121 reasoning)
release-1.0
poolside/laguna-xs.2:low__json_schema PASS 100% 159 in → 205 out (191 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
poolside/laguna-xs.2:medium PASS 100% 166 in → 130 out (122 reasoning)
release-1.0
poolside/laguna-xs.2:medium__json_schema PASS 100% 165 in → 135 out (121 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
poolside/laguna-xs.2:none PASS 100% 163 in → 7 out (0 reasoning)
release-1.0
poolside/laguna-xs.2:none__json_schema PASS 100% 160 in → 13 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
poolside/laguna-xs.2:xhigh PASS 100% 161 in → 208 out (200 reasoning)
release-1.0
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 166 in → 209 out (195 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
qwen/qwen3.6-27b:high PASS 100% 123 in → 419 out (356 reasoning)
release-1.0
qwen/qwen3.6-27b:high__json_schema PASS 100% 122 in → 1,011 out (888 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
qwen/qwen3.6-27b:low PASS 100% 125 in → 389 out (329 reasoning)
release-1.0
qwen/qwen3.6-27b:low__json_schema PASS 100% 126 in → 854 out (732 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
qwen/qwen3.6-27b:medium PASS 100% 125 in → 917 out (802 reasoning)
release-1.0
qwen/qwen3.6-27b:medium__json_schema PASS 100% 127 in → 868 out (1 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
qwen/qwen3.6-27b:none PASS 100% 130 in → 6 out (0 reasoning)
release-1.0
qwen/qwen3.6-27b:none__json_schema PASS 100% 126 in → 15 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
qwen/qwen3.6-27b:xhigh PASS 100% 122 in → 870 out (790 reasoning)
release-1.0
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 126 in → 18 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
qwen/qwen3.6-35b-a3b:high PASS 100% 126 in → 684 out (598 reasoning)
release-1.0
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 123 in → 1,049 out (906 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 122 in → 417 out (334 reasoning)
release-1.0
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 118 in → 1,029 out (903 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 127 in → 923 out (824 reasoning)
release-1.0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 128 in → 832 out (717 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 129 in → 6 out (0 reasoning)
release-1.0
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 128 in → 25 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 127 in → 652 out (642 reasoning)
release-1.0
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 118 in → 743 out (621 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
qwen/qwen3.6-flash:high PASS 100% 123 in → 982 out (972 reasoning)
release-1.0
qwen/qwen3.6-flash:low PASS 100% 127 in → 958 out (948 reasoning)
release-1.0
qwen/qwen3.6-flash:medium PASS 100% 119 in → 442 out (432 reasoning)
release-1.0
qwen/qwen3.6-flash:none PASS 100% 129 in → 5 out (0 reasoning)
release-1.0
qwen/qwen3.6-flash:xhigh PASS 100% 125 in → 782 out (772 reasoning)
release-1.0
qwen/qwen3.7-flash:high PASS 100% 128 in → 412 out (402 reasoning)
release-1.0
qwen/qwen3.7-flash:low PASS 100% 123 in → 446 out (436 reasoning)
release-1.0
qwen/qwen3.7-flash:max PASS 100% 126 in → 357 out (347 reasoning)
release-1.0
qwen/qwen3.7-flash:medium PASS 100% 124 in → 726 out (716 reasoning)
release-1.0
qwen/qwen3.7-flash:none PASS 100% 129 in → 5 out (0 reasoning)
release-1.0
qwen/qwen3.7-flash:xhigh PASS 100% 124 in → 499 out (489 reasoning)
release-1.0
qwen/qwen3.7-max:high PASS 100% 122 in → 195 out (185 reasoning)
release-1.0
qwen/qwen3.7-max:low PASS 100% 126 in → 243 out (233 reasoning)
release-1.0
qwen/qwen3.7-max:medium PASS 100% 120 in → 213 out (203 reasoning)
release-1.0
qwen/qwen3.7-max:none PASS 100% 123 in → 5 out (0 reasoning)
release-1.0
qwen/qwen3.7-max:xhigh PASS 100% 124 in → 290 out (280 reasoning)
release-1.0
qwen/qwen3.7-plus:high PASS 100% 125 in → 514 out (504 reasoning)
release-1.0
qwen/qwen3.7-plus:low PASS 100% 122 in → 531 out (521 reasoning)
release-1.0
qwen/qwen3.7-plus:medium PASS 100% 125 in → 563 out (553 reasoning)
release-1.0
qwen/qwen3.7-plus:none PASS 100% 127 in → 5 out (0 reasoning)
release-1.0
qwen/qwen3.7-plus:xhigh PASS 100% 123 in → 321 out (311 reasoning)
release-1.0
tencent/hy3:high PASS 100% 122 in → 877 out (870 reasoning)
release-1.0
tencent/hy3:high__json_schema PASS 100% 118 in → 585 out (519 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
tencent/hy3:low PASS 100% 122 in → 478 out (427 reasoning)
release-1.0
tencent/hy3:low__json_schema PASS 100% 118 in → 364 out (309 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
tencent/hy3:medium PASS 100% 121 in → 468 out (461 reasoning)
release-1.0
tencent/hy3:medium__json_schema PASS 100% 123 in → 327 out (262 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
tencent/hy3:none__json_schema PASS 100% 122 in → 21 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
tencent/hy3:xhigh PASS 100% 119 in → 456 out (449 reasoning)
release-1.0
tencent/hy3:xhigh__json_schema PASS 100% 122 in → 585 out (533 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
thinkingmachines/inkling-small:high PASS 100% 117 in → 172 out (146 reasoning)
release-1.0
thinkingmachines/inkling-small:high__json_schema PASS 100% 120 in → 152 out (127 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
thinkingmachines/inkling-small:low PASS 100% 117 in → 39 out (27 reasoning)
release-1.0
thinkingmachines/inkling-small:low__json_schema PASS 100% 123 in → 35 out (11 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
thinkingmachines/inkling-small:max PASS 100% 119 in → 260 out (247 reasoning)
release-1.0
thinkingmachines/inkling-small:max__json_schema PASS 100% 120 in → 216 out (195 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
thinkingmachines/inkling-small:medium PASS 100% 120 in → 90 out (54 reasoning)
release-1.0
thinkingmachines/inkling-small:medium__json_schema PASS 100% 120 in → 112 out (91 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
thinkingmachines/inkling-small:none PASS 100% 120 in → 8 out (0 reasoning)
release-1.0
thinkingmachines/inkling-small:xhigh PASS 100% 120 in → 89 out (53 reasoning)
release-1.0
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 123 in → 108 out (84 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
thinkingmachines/inkling:high PASS 100% 120 in → 221 out (210 reasoning)
release-1.0
thinkingmachines/inkling:low PASS 100% 119 in → 44 out (32 reasoning)
release-1.0
thinkingmachines/inkling:max PASS 100% 123 in → 177 out (166 reasoning)
release-1.0
thinkingmachines/inkling:medium PASS 100% 120 in → 159 out (129 reasoning)
release-1.0
thinkingmachines/inkling:medium__json_schema PASS 100% 120 in → 125 out (101 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
thinkingmachines/inkling:none PASS 100% 121 in → 9 out (0 reasoning)
release-1.0
thinkingmachines/inkling:xhigh PASS 100% 121 in → 306 out (266 reasoning)
release-1.0
thinkingmachines/inkling:xhigh__json_schema PASS 100% 124 in → 200 out (175 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
x-ai/grok-4.3:high PASS 100% 289 in → 338 out (333 reasoning)
release-1.0
x-ai/grok-4.3:high__json_schema PASS 100% 353 in → 477 out (466 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
x-ai/grok-4.3:low PASS 100% 293 in → 262 out (257 reasoning)
release-1.0
x-ai/grok-4.3:low__json_schema PASS 100% 363 in → 430 out (419 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
x-ai/grok-4.3:max PASS 100% 286 in → 326 out (321 reasoning)
release-1.0
x-ai/grok-4.3:max__json_schema PASS 100% 354 in → 501 out (490 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
x-ai/grok-4.3:medium FAIL 100% 297 in → 797 out (786 reasoning)
release-1.0 \confidence{80}
Failure: Extra: ['\\confidence{80}']
x-ai/grok-4.3:medium__json_schema PASS 100% 361 in → 491 out (472 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
x-ai/grok-4.3:none PASS 100% 285 in → 5 out (0 reasoning)
release-1.0
x-ai/grok-4.3:none__json_schema PASS 100% 356 in → 19 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
x-ai/grok-4.3:xhigh PASS 100% 291 in → 389 out (384 reasoning)
release-1.0
x-ai/grok-4.3:xhigh__json_schema PASS 100% 354 in → 457 out (446 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
x-ai/grok-4.5:high PASS 100% 315 in → 482 out (477 reasoning)
release-1.0
x-ai/grok-4.5:high__json_schema PASS 100% 393 in → 466 out (452 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
x-ai/grok-4.5:low PASS 100% 310 in → 256 out (251 reasoning)
release-1.0
x-ai/grok-4.5:low__json_schema PASS 100% 392 in → 244 out (230 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
x-ai/grok-4.5:max PASS 100% 311 in → 307 out (302 reasoning)
release-1.0
x-ai/grok-4.5:max__json_schema PASS 100% 398 in → 422 out (408 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
x-ai/grok-4.5:medium PASS 100% 313 in → 294 out (289 reasoning)
release-1.0
x-ai/grok-4.5:medium__json_schema PASS 100% 397 in → 413 out (399 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
x-ai/grok-4.5:xhigh PASS 100% 313 in → 276 out (271 reasoning)
release-1.0
x-ai/grok-4.5:xhigh__json_schema PASS 100% 392 in → 447 out (433 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
z-ai/glm-4.7-flash:high PASS 100% 112 in → 455 out (404 reasoning)
release-1.0
z-ai/glm-4.7-flash:low PASS 100% 113 in → 532 out (470 reasoning)
release-1.0
z-ai/glm-4.7-flash:medium PASS 100% 110 in → 526 out (505 reasoning)
release-1.0
z-ai/glm-4.7-flash:xhigh PASS 100% 115 in → 388 out (381 reasoning)
release-1.0
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 114 in → 575 out (514 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-1.0"] }
z-ai/glm-5.2:high PASS 100% 116 in → 263 out (224 reasoning)
release-1.0
z-ai/glm-5.2:high__json_schema PASS 100% 122 in → 301 out (283 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
z-ai/glm-5.2:low PASS 100% 125 in → 382 out (375 reasoning)
release-1.0
z-ai/glm-5.2:medium PASS 100% 118 in → 322 out (275 reasoning)
release-1.0
z-ai/glm-5.2:medium__json_schema PASS 100% 303 in → 197 out (183 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete": ["release-1.0"]}
z-ai/glm-5.2:none PASS 100% 114 in → 6 out (0 reasoning)
release-1.0
z-ai/glm-5.2:none__json_schema PASS 100% 115 in → 20 out (0 reasoning)
release-1.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-1.0" ] }
z-ai/glm-5.2:xhigh PASS 100% 121 in → 350 out (339 reasoning)
release-1.0
z-ai/glm-5.2:xhigh__json_schema PASS 100% 305 in → 191 out (178 reasoning)
release-1.0
JSON Schema Structured Output
(raw) {"branches_to_delete":["release-1.0"]}
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 115 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['release-1.0']
ibm-granite/granite-4.1-8b FAIL 0% 118 in → 6 out
release-2.0
Failure: Missing: ['release-1.0'] Extra: ['release-2.0']
release-2.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-2.0" ] }
Failure: Missing: ['release-1.0'] Extra: ['release-2.0']
liquid/lfm-2-24b-a2b FAIL 0% 117 in → 7 out
release-2.0
Failure: Missing: ['release-1.0'] Extra: ['release-2.0']
Invalid JSON. Output: release-2.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:medium__json_schema FAIL 0% 276 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['release-1.0']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
openai/gpt-oss-120b:low FAIL 0% 177 in → 60 out (41 reasoning)
release
Failure: Missing: ['release-1.0'] Extra: ['release']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-38aba5f7-5754-99ae-9f9d-5c65cd970806","request_id":"38aba5f7-5754-99ae-9f9d-5c65cd970806"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-42c24c1c-87ea-9ec2-b385-a5948059cc30","request_id":"42c24c1c-87ea-9ec2-b385-a5948059cc30"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-022f8784-936f-913e-93f9-e5d9b3b92c7c","request_id":"022f8784-936f-913e-93f9-e5d9b3b92c7c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2720dde0-2254-9591-9c32-072244ca05b6","request_id":"2720dde0-2254-9591-9c32-072244ca05b6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-48fa235c-9d1f-9044-aa42-74f713f47883","request_id":"48fa235c-9d1f-9044-aa42-74f713f47883"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1fc6eb83-c7e2-92f7-a834-ad2786eeed5f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-327dd61e-962f-9708-81f4-a13d8cca4a54"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-096dba99-8ea4-9e8e-819d-36dd0b37280e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-08a3415f-15c4-985a-801c-d0711da888fe"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2dba636b-8bed-943b-98a6-95cd021ab278"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e278ce28-fcb6-9db2-8fa1-f256ac504d94"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-348c1a2d-9e34-90eb-9cd0-30920b177d1a","request_id":"348c1a2d-9e34-90eb-9cd0-30920b177d1a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-38e072ba-6e5d-9614-9526-fd02e6eb61ab","request_id":"38e072ba-6e5d-9614-9526-fd02e6eb61ab"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1b839510-84d8-9a91-8663-a87f4f848b26","request_id":"1b839510-84d8-9a91-8663-a87f4f848b26"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-653e248a-fce6-9711-8c0d-9d4d7bb95e08","request_id":"653e248a-fce6-9711-8c0d-9d4d7bb95e08"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bf2789d0-3043-9ead-8124-b868c0daed1e","request_id":"bf2789d0-3043-9ead-8124-b868c0daed1e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a082feee-0c97-98fd-84fb-c8397a436e65","request_id":"a082feee-0c97-98fd-84fb-c8397a436e65"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-792936a2-82a2-997a-be04-673caea6d2b7","request_id":"792936a2-82a2-997a-be04-673caea6d2b7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e7ff0c91-ba28-9c15-9f28-bb96576ea015","request_id":"e7ff0c91-ba28-9c15-9f28-bb96576ea015"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dda63b20-7863-9b08-8711-c7f2b5e62dc6","request_id":"dda63b20-7863-9b08-8711-c7f2b5e62dc6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7b0fd67a-522f-9d02-9854-27e6766e08f8","request_id":"7b0fd67a-522f-9d02-9854-27e6766e08f8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
Failure: Model 'tencent/hy3:none' violated the reasoning_level=none invariant: provider reported 495 reasoning token(s)
Invalid JSON. Output: release-1.0{ "branches_to_delete": ["release-1.0"] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: release-1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: release-1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: release-1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: release-1.0{ "branches_to_delete": [ "release-1.0" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 114 in → 6 out (0 reasoning)
release-2.0
Failure: Missing: ['release-1.0'] Extra: ['release-2.0']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 116 in → 20 out (0 reasoning)
release-2.0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "release-2.0" ] }
Failure: Missing: ['release-1.0'] Extra: ['release-2.0']
Invalid JSON. Output: release-1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)