Hotfix branch merged
Tests ability to identify a merged hotfix branch. Evaluates recognition of hotfix naming conventions in branch cleanup.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'v1.0' > version.txt
  5. 05 git add version.txt
  6. 06 git commit -m 'Release v1.0'
  7. 07 git checkout -b hotfix-security
  8. 08 echo 'v1.0.1-patched' > version.txt
  9. 09 git add version.txt
  10. 10 git commit -m 'Security patch'
  11. 11 git checkout main
  12. 12 git merge hotfix-security
  13. 13 git checkout -b release-v2
  14. 14 echo 'v2.0-dev' > version.txt
  15. 15 git add version.txt
  16. 16 git commit -m 'Start v2.0'
  17. 17 git checkout main
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
hotfix-security

Scoped model quality, cost, API time, and token usage for branch_cleanup/f005.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 137 in → 47 out (40 reasoning)
hotfix-security
anthropic/claude-fable-5:high__json_schema PASS 100% 372 in → 45 out (24 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
anthropic/claude-fable-5:low PASS 100% 134 in → 16 out (13 reasoning)
hotfix-security
anthropic/claude-fable-5:low__json_schema PASS 100% 377 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-fable-5:max PASS 100% 137 in → 298 out (71 reasoning)
hotfix-security
anthropic/claude-fable-5:max__json_schema PASS 100% 376 in → 211 out (69 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
anthropic/claude-fable-5:medium FAIL 100% 139 in → 127 out (25 reasoning)
hotfix-security **Explanation:** The `hotfix-security` branch points to the same commit as `main` (7dcb517) and appears in the merged list, so it's safe to delete. `main` itself is never deleted, and `release-v2` (c5e6aad) is not merged into main, so it should be kept.
Failure: Extra: ["**Explanation:** The `hotfix-security` branch points to the same commit as `main` (7dcb517) and appears in the merged list, so it's safe to delete. `main` itself is never deleted, and `release-v2` (c5e6aad) is not merged into main, so it should be kept."]
anthropic/claude-fable-5:medium__json_schema PASS 100% 376 in → 38 out (20 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
anthropic/claude-fable-5:xhigh PASS 100% 135 in → 52 out (33 reasoning)
hotfix-security
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 377 in → 61 out (22 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
anthropic/claude-haiku-4.5:high PASS 100% 142 in → 204 out (170 reasoning)
hotfix-security
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 327 in → 348 out (295 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
anthropic/claude-haiku-4.5:low PASS 100% 136 in → 197 out (172 reasoning)
hotfix-security
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 327 in → 226 out (194 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
anthropic/claude-haiku-4.5:medium PASS 100% 142 in → 174 out (148 reasoning)
hotfix-security
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 327 in → 287 out (237 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
anthropic/claude-haiku-4.5:none PASS 100% 109 in → 7 out (0 reasoning)
hotfix-security
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 296 in → 17 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 139 in → 191 out (170 reasoning)
hotfix-security
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 330 in → 497 out (489 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
anthropic/claude-opus-4.6:high PASS 100% 107 in → 110 out (83 reasoning)
hotfix-security
anthropic/claude-opus-4.6:high__json_schema PASS 100% 297 in → 53 out (20 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.6:low PASS 100% 109 in → 7 out (0 reasoning)
hotfix-security
anthropic/claude-opus-4.6:low__json_schema PASS 100% 299 in → 17 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.6:max PASS 100% 111 in → 150 out (124 reasoning)
hotfix-security
anthropic/claude-opus-4.6:max__json_schema PASS 100% 303 in → 100 out (76 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.6:medium PASS 100% 112 in → 7 out (0 reasoning)
hotfix-security
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 297 in → 17 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.6:none PASS 100% 109 in → 7 out (0 reasoning)
hotfix-security
anthropic/claude-opus-4.6:none__json_schema PASS 100% 297 in → 17 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 108 in → 93 out (72 reasoning)
hotfix-security
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 297 in → 79 out (45 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.7:high PASS 100% 140 in → 10 out (0 reasoning)
hotfix-security
anthropic/claude-opus-4.7:high__json_schema PASS 100% 381 in → 24 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.7:low PASS 100% 141 in → 10 out (0 reasoning)
hotfix-security
anthropic/claude-opus-4.7:low__json_schema PASS 100% 377 in → 24 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.7:max PASS 100% 141 in → 196 out (49 reasoning)
hotfix-security
anthropic/claude-opus-4.7:max__json_schema PASS 100% 383 in → 24 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.7:medium PASS 100% 141 in → 10 out (0 reasoning)
hotfix-security
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 378 in → 24 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.7:none PASS 100% 140 in → 10 out (0 reasoning)
hotfix-security
anthropic/claude-opus-4.7:none__json_schema PASS 100% 386 in → 24 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 142 in → 10 out (0 reasoning)
hotfix-security
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 382 in → 24 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.8:high PASS 100% 133 in → 80 out (44 reasoning)
hotfix-security
anthropic/claude-opus-4.8:high__json_schema PASS 100% 376 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.8:low PASS 100% 138 in → 8 out (0 reasoning)
hotfix-security
anthropic/claude-opus-4.8:low__json_schema PASS 100% 377 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.8:max PASS 100% 135 in → 369 out (67 reasoning)
hotfix-security
anthropic/claude-opus-4.8:max__json_schema PASS 100% 372 in → 218 out (66 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.8:medium PASS 100% 141 in → 64 out (52 reasoning)
hotfix-security
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 377 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.8:none PASS 100% 137 in → 8 out (0 reasoning)
hotfix-security
anthropic/claude-opus-4.8:none__json_schema PASS 100% 374 in → 21 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 136 in → 154 out (41 reasoning)
hotfix-security
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 376 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-4.6:high PASS 100% 111 in → 58 out (34 reasoning)
hotfix-security
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 301 in → 117 out (93 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-4.6:low PASS 100% 111 in → 7 out (0 reasoning)
hotfix-security
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 302 in → 17 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-4.6:max PASS 100% 109 in → 119 out (103 reasoning)
hotfix-security
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 304 in → 127 out (99 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 113 in → 7 out (0 reasoning)
hotfix-security
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 304 in → 17 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-4.6:none PASS 100% 109 in → 7 out (0 reasoning)
hotfix-security
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 298 in → 17 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 109 in → 96 out (79 reasoning)
hotfix-security
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 297 in → 17 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-5:high PASS 100% 138 in → 71 out (55 reasoning)
hotfix-security
anthropic/claude-sonnet-5:high__json_schema PASS 100% 377 in → 58 out (49 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-5:low PASS 100% 140 in → 8 out (0 reasoning)
hotfix-security
anthropic/claude-sonnet-5:low__json_schema PASS 100% 377 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-5:medium PASS 100% 135 in → 8 out (0 reasoning)
hotfix-security
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 372 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-5:none PASS 100% 141 in → 8 out (0 reasoning)
hotfix-security
anthropic/claude-sonnet-5:none__json_schema PASS 100% 377 in → 21 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 139 in → 96 out (51 reasoning)
hotfix-security
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 373 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
arcee-ai/trinity-large-thinking:high PASS 100% 104 in → 675 out (682 reasoning)
hotfix-security
arcee-ai/trinity-large-thinking:low PASS 100% 104 in → 486 out (487 reasoning)
hotfix-security
arcee-ai/trinity-large-thinking:medium PASS 100% 101 in → 547 out (551 reasoning)
hotfix-security
arcee-ai/trinity-large-thinking:xhigh PASS 100% 100 in → 441 out (441 reasoning)
hotfix-security
arcee-ai/trinity-mini:high PASS 100% 99 in → 316 out (355 reasoning)
hotfix-security
arcee-ai/trinity-mini:high__json_schema PASS 100% 101 in → 444 out (501 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
arcee-ai/trinity-mini:low PASS 100% 98 in → 353 out (408 reasoning)
hotfix-security
arcee-ai/trinity-mini:low__json_schema PASS 100% 105 in → 478 out (533 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
arcee-ai/trinity-mini:medium PASS 100% 104 in → 332 out (376 reasoning)
hotfix-security
arcee-ai/trinity-mini:medium__json_schema PASS 100% 100 in → 305 out (335 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 100 in → 288 out (324 reasoning)
hotfix-security
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 103 in → 276 out (292 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 100 in → 507 out (510 reasoning)
hotfix-security
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 106 in → 74 out (67 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 98 in → 258 out (241 reasoning)
hotfix-security
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 100 in → 89 out (92 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 182 in → 164 out (160 reasoning)
hotfix-security
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 178 in → 193 out (190 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 101 in → 189 out (184 reasoning)
hotfix-security
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 103 in → 142 out (178 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 101 in → 3 out (0 reasoning)
hotfix-security
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 102 in → 19 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 98 in → 70 out (89 reasoning)
hotfix-security
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 187 in → 117 out (98 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
deepseek/deepseek-v4-flash:high PASS 100% 118 in → 118 out (113 reasoning)
hotfix-security
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 101 in → 113 out (106 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
deepseek/deepseek-v4-flash:low PASS 100% 97 in → 169 out (164 reasoning)
hotfix-security
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 101 in → 177 out (161 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
deepseek/deepseek-v4-flash:medium PASS 100% 99 in → 219 out (214 reasoning)
hotfix-security
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 98 in → 185 out (171 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
deepseek/deepseek-v4-flash:none PASS 100% 99 in → 4 out (0 reasoning)
hotfix-security
deepseek/deepseek-v4-flash:xhigh PASS 100% 193 in → 288 out (283 reasoning)
hotfix-security
deepseek/deepseek-v4-pro:high PASS 100% 98 in → 255 out (250 reasoning)
hotfix-security
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 102 in → 323 out (303 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
deepseek/deepseek-v4-pro:low PASS 100% 105 in → 302 out (298 reasoning)
hotfix-security
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 301 in → 224 out (210 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
deepseek/deepseek-v4-pro:medium PASS 100% 100 in → 246 out (241 reasoning)
hotfix-security
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 101 in → 272 out (252 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
deepseek/deepseek-v4-pro:none PASS 100% 99 in → 4 out (0 reasoning)
hotfix-security
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 301 in → 13 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
deepseek/deepseek-v4-pro:xhigh PASS 100% 99 in → 504 out (552 reasoning)
hotfix-security
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 378 in → 959 out (945 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
google/gemini-3-flash-preview:high PASS 100% 103 in → 173 out (169 reasoning)
hotfix-security
google/gemini-3-flash-preview:high__json_schema PASS 100% 185 in → 248 out (236 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3-flash-preview:low PASS 100% 104 in → 280 out (276 reasoning)
hotfix-security
google/gemini-3-flash-preview:low__json_schema PASS 100% 184 in → 291 out (279 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3-flash-preview:medium PASS 100% 103 in → 290 out (286 reasoning)
hotfix-security
google/gemini-3-flash-preview:medium__json_schema PASS 100% 184 in → 346 out (334 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3-flash-preview:xhigh PASS 100% 104 in → 309 out (305 reasoning)
hotfix-security
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 186 in → 303 out (291 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 105 in → 365 out (361 reasoning)
hotfix-security
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 104 in → 327 out (315 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 103 in → 147 out (143 reasoning)
hotfix-security
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 105 in → 146 out (122 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 103 in → 226 out (222 reasoning)
hotfix-security
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 106 in → 203 out (191 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 103 in → 403 out (399 reasoning)
hotfix-security
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 185 in → 284 out (260 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemini-3.1-pro-preview:high PASS 100% 104 in → 369 out (365 reasoning)
hotfix-security
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 185 in → 322 out (298 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemini-3.1-pro-preview:low PASS 100% 103 in → 279 out (275 reasoning)
hotfix-security
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 186 in → 119 out (95 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemini-3.1-pro-preview:medium PASS 100% 105 in → 312 out (308 reasoning)
hotfix-security
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 184 in → 200 out (188 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 104 in → 356 out (352 reasoning)
hotfix-security
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 184 in → 234 out (210 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemini-3.5-flash-lite:high PASS 100% 101 in → 338 out (334 reasoning)
hotfix-security
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 184 in → 427 out (415 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.5-flash-lite:low PASS 100% 102 in → 4 out (0 reasoning)
hotfix-security
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 182 in → 24 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemini-3.5-flash-lite:max PASS 100% 102 in → 371 out (367 reasoning)
hotfix-security
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 182 in → 337 out (325 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.5-flash-lite:medium PASS 100% 102 in → 299 out (295 reasoning)
hotfix-security
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 180 in → 390 out (378 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 107 in → 462 out (458 reasoning)
hotfix-security
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 105 in → 367 out (343 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemini-3.5-flash:high PASS 100% 102 in → 395 out (391 reasoning)
hotfix-security
google/gemini-3.5-flash:high__json_schema PASS 100% 184 in → 556 out (544 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.5-flash:low PASS 100% 106 in → 308 out (304 reasoning)
hotfix-security
google/gemini-3.5-flash:low__json_schema PASS 100% 185 in → 166 out (154 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.5-flash:medium PASS 100% 106 in → 397 out (393 reasoning)
hotfix-security
google/gemini-3.5-flash:medium__json_schema PASS 100% 187 in → 233 out (221 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.5-flash:xhigh PASS 100% 104 in → 342 out (338 reasoning)
hotfix-security
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 186 in → 269 out (257 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.6-flash:high PASS 100% 106 in → 311 out (307 reasoning)
hotfix-security
google/gemini-3.6-flash:high__json_schema PASS 100% 107 in → 332 out (320 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.6-flash:low PASS 100% 107 in → 188 out (184 reasoning)
hotfix-security
google/gemini-3.6-flash:low__json_schema PASS 100% 185 in → 105 out (93 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.6-flash:max PASS 100% 100 in → 378 out (374 reasoning)
hotfix-security
google/gemini-3.6-flash:max__json_schema PASS 100% 102 in → 276 out (264 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.6-flash:medium PASS 100% 104 in → 261 out (257 reasoning)
hotfix-security
google/gemini-3.6-flash:medium__json_schema PASS 100% 102 in → 438 out (426 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemini-3.6-flash:xhigh PASS 100% 102 in → 374 out (370 reasoning)
hotfix-security
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 185 in → 340 out (328 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
google/gemma-4-26b-a4b-it:high PASS 100% 118 in → 279 out (231 reasoning)
hotfix-security
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 120 in → 338 out (270 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 119 in → 249 out (244 reasoning)
hotfix-security
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 119 in → 276 out (218 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 119 in → 283 out (222 reasoning)
hotfix-security
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 122 in → 272 out (190 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 118 in → 5 out (0 reasoning)
hotfix-security
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 119 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 118 in → 255 out (250 reasoning)
hotfix-security
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 122 in → 391 out (315 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemma-4-31b-it:high PASS 100% 121 in → 178 out (157 reasoning)
hotfix-security
google/gemma-4-31b-it:high__json_schema PASS 100% 118 in → 203 out (159 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemma-4-31b-it:low PASS 100% 122 in → 219 out (191 reasoning)
hotfix-security
google/gemma-4-31b-it:low__json_schema PASS 100% 122 in → 180 out (143 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemma-4-31b-it:medium PASS 100% 120 in → 174 out (155 reasoning)
hotfix-security
google/gemma-4-31b-it:medium__json_schema PASS 100% 116 in → 288 out (218 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
google/gemma-4-31b-it:none PASS 100% 117 in → 5 out (0 reasoning)
hotfix-security
google/gemma-4-31b-it:none__json_schema PASS 100% 116 in → 25 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
google/gemma-4-31b-it:xhigh PASS 100% 120 in → 245 out (1 reasoning)
hotfix-security
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 121 in → 232 out (218 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
ibm-granite/granite-4.1-8b PASS 100% 109 in → 4 out
hotfix-security
ibm-granite/granite-4.1-8b__json_schema PASS 100% 112 in → 20 out
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
inclusionai/ling-2.6-flash PASS 100% 128 in → 4 out
hotfix-security
inclusionai/ling-2.6-flash__json_schema PASS 100% 128 in → 17 out
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
liquid/lfm-2-24b-a2b PASS 100% 109 in → 5 out
hotfix-security
minimax/minimax-m2.5:high PASS 100% 132 in → 388 out (435 reasoning)
hotfix-security
minimax/minimax-m2.5:high__json_schema PASS 100% 133 in → 519 out (503 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
minimax/minimax-m2.5:low PASS 100% 144 in → 134 out (129 reasoning)
hotfix-security
minimax/minimax-m2.5:low__json_schema PASS 100% 132 in → 363 out (389 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
minimax/minimax-m2.5:medium PASS 100% 130 in → 313 out (309 reasoning)
hotfix-security
minimax/minimax-m2.5:medium__json_schema PASS 100% 133 in → 435 out (465 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
minimax/minimax-m2.5:xhigh PASS 100% 133 in → 389 out (385 reasoning)
hotfix-security
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 136 in → 635 out (615 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
minimax/minimax-m2.7:high PASS 100% 131 in → 226 out (221 reasoning)
hotfix-security
minimax/minimax-m2.7:low PASS 100% 136 in → 154 out (148 reasoning)
hotfix-security
minimax/minimax-m2.7:low__json_schema PASS 100% 284 in → 421 out (408 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
minimax/minimax-m2.7:medium PASS 100% 140 in → 275 out (305 reasoning)
hotfix-security
minimax/minimax-m2.7:medium__json_schema PASS 100% 284 in → 368 out (355 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
minimax/minimax-m2.7:xhigh PASS 100% 131 in → 169 out (188 reasoning)
hotfix-security
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 283 in → 449 out (433 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
minimax/minimax-m3:high PASS 100% 272 in → 152 out (162 reasoning)
hotfix-security
minimax/minimax-m3:high__json_schema PASS 100% 260 in → 14 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
minimax/minimax-m3:low PASS 100% 272 in → 184 out (186 reasoning)
hotfix-security
minimax/minimax-m3:low__json_schema PASS 100% 258 in → 15 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
minimax/minimax-m3:medium PASS 100% 270 in → 189 out (188 reasoning)
hotfix-security
minimax/minimax-m3:medium__json_schema PASS 100% 260 in → 19 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
minimax/minimax-m3:xhigh PASS 100% 273 in → 184 out (195 reasoning)
hotfix-security
minimax/minimax-m3:xhigh__json_schema PASS 100% 262 in → 13 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
mistralai/devstral-2512 PASS 100% 105 in → 4 out
hotfix-security
mistralai/devstral-2512__json_schema PASS 100% 107 in → 13 out
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
mistralai/mistral-medium-3-5:high PASS 100% 112 in → 454 out (496 reasoning)
hotfix-security
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 104 in → 266 out (281 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
mistralai/mistral-medium-3-5:low PASS 100% 119 in → 228 out (229 reasoning)
hotfix-security
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 108 in → 241 out (237 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
mistralai/mistral-medium-3-5:medium PASS 100% 118 in → 240 out (251 reasoning)
hotfix-security
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 109 in → 214 out (204 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
mistralai/mistral-medium-3-5:none PASS 100% 118 in → 4 out (0 reasoning)
hotfix-security
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 107 in → 13 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 114 in → 277 out (304 reasoning)
hotfix-security
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 109 in → 238 out (227 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
moonshotai/kimi-k2.6:high PASS 100% 104 in → 205 out (200 reasoning)
hotfix-security
moonshotai/kimi-k2.6:high__json_schema PASS 100% 103 in → 247 out (225 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
moonshotai/kimi-k2.6:low PASS 100% 99 in → 132 out (156 reasoning)
hotfix-security
moonshotai/kimi-k2.6:low__json_schema PASS 100% 103 in → 539 out (139 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
moonshotai/kimi-k2.6:medium PASS 100% 101 in → 318 out (313 reasoning)
hotfix-security
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 102 in → 373 out (185 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
moonshotai/kimi-k2.6:xhigh PASS 100% 102 in → 327 out (328 reasoning)
hotfix-security
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 295 in → 770 out (757 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
moonshotai/kimi-k2.7-code:high PASS 100% 103 in → 203 out (198 reasoning)
hotfix-security
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 103 in → 204 out (188 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
moonshotai/kimi-k2.7-code:low PASS 100% 99 in → 123 out (118 reasoning)
hotfix-security
moonshotai/kimi-k2.7-code:medium PASS 100% 100 in → 159 out (154 reasoning)
hotfix-security
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 196 in → 165 out (151 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 104 in → 172 out (167 reasoning)
hotfix-security
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 196 in → 165 out (152 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
moonshotai/kimi-k3:high PASS 100% 179 in → 254 out (236 reasoning)
hotfix-security
moonshotai/kimi-k3:high__json_schema PASS 100% 176 in → 152 out (134 reasoning)
hotfix-security
moonshotai/kimi-k3:low PASS 100% 185 in → 160 out (142 reasoning)
hotfix-security
moonshotai/kimi-k3:low__json_schema PASS 100% 182 in → 341 out (323 reasoning)
hotfix-security
moonshotai/kimi-k3:max PASS 100% 178 in → 170 out (152 reasoning)
hotfix-security
moonshotai/kimi-k3:max__json_schema PASS 100% 177 in → 156 out (138 reasoning)
hotfix-security
moonshotai/kimi-k3:medium PASS 100% 176 in → 103 out (85 reasoning)
hotfix-security
moonshotai/kimi-k3:medium__json_schema PASS 100% 176 in → 162 out (144 reasoning)
hotfix-security
moonshotai/kimi-k3:xhigh PASS 100% 180 in → 196 out (178 reasoning)
hotfix-security
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 177 in → 212 out (194 reasoning)
hotfix-security
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 114 in → 231 out (244 reasoning)
hotfix-security
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 100% 121 in → 198 out (181 reasoning)
hotfix-security none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security", "none" ] }
Failure: Extra: ['none']
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 121 in → 193 out (166 reasoning)
hotfix-security
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 120 in → 177 out (163 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 120 in → 194 out (205 reasoning)
hotfix-security
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 119 in → 189 out (176 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 122 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 119 in → 142 out (138 reasoning)
hotfix-security
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 120 in → 129 out (115 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 118 in → 121 out (103 reasoning)
hotfix-security
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 117 in → 76 out (56 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 120 in → 122 out (115 reasoning)
hotfix-security
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 121 in → 161 out (141 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 122 in → 53 out (56 reasoning)
hotfix-security
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 116 in → 185 out (161 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 122 in → 4 out (0 reasoning)
hotfix-security
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 118 in → 13 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 114 in → 166 out (173 reasoning)
hotfix-security
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 115 in → 133 out (112 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
openai/gpt-5.4-mini:high PASS 100% 101 in → 97 out (88 reasoning)
hotfix-security
openai/gpt-5.4-mini:high__json_schema PASS 100% 143 in → 196 out (176 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4-mini:low PASS 100% 101 in → 40 out (31 reasoning)
hotfix-security
openai/gpt-5.4-mini:low__json_schema PASS 100% 146 in → 53 out (33 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4-mini:medium PASS 100% 102 in → 61 out (52 reasoning)
hotfix-security
openai/gpt-5.4-mini:medium__json_schema PASS 100% 143 in → 131 out (111 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4-mini:none PASS 100% 98 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.4-mini:none__json_schema PASS 100% 140 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4-mini:xhigh PASS 100% 102 in → 250 out (241 reasoning)
hotfix-security
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 141 in → 1,037 out (1,017 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4-nano:high PASS 100% 99 in → 44 out (35 reasoning)
hotfix-security
openai/gpt-5.4-nano:high__json_schema PASS 100% 142 in → 69 out (49 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4-nano:low PASS 100% 103 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.4-nano:low__json_schema PASS 100% 143 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4-nano:medium PASS 100% 103 in → 29 out (20 reasoning)
hotfix-security
openai/gpt-5.4-nano:medium__json_schema PASS 100% 146 in → 46 out (26 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4-nano:none PASS 100% 104 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.4-nano:none__json_schema PASS 100% 145 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4-nano:xhigh PASS 100% 99 in → 119 out (110 reasoning)
hotfix-security
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 146 in → 414 out (394 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4:high PASS 100% 105 in → 71 out (62 reasoning)
hotfix-security
openai/gpt-5.4:high__json_schema PASS 100% 142 in → 77 out (57 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4:low PASS 100% 96 in → 48 out (39 reasoning)
hotfix-security
openai/gpt-5.4:low__json_schema PASS 100% 144 in → 57 out (37 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4:medium PASS 100% 100 in → 62 out (53 reasoning)
hotfix-security
openai/gpt-5.4:medium__json_schema PASS 100% 145 in → 74 out (54 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4:none PASS 100% 101 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.4:none__json_schema PASS 100% 145 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.4:xhigh PASS 100% 97 in → 132 out (123 reasoning)
hotfix-security
openai/gpt-5.4:xhigh__json_schema PASS 100% 141 in → 134 out (114 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.5:high PASS 100% 101 in → 56 out (47 reasoning)
hotfix-security
openai/gpt-5.5:high__json_schema PASS 100% 143 in → 117 out (97 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.5:low PASS 100% 103 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.5:low__json_schema PASS 100% 143 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.5:medium PASS 100% 103 in → 62 out (53 reasoning)
hotfix-security
openai/gpt-5.5:medium__json_schema PASS 100% 145 in → 60 out (40 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.5:none PASS 100% 96 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.5:none__json_schema PASS 100% 141 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.5:xhigh PASS 100% 104 in → 72 out (63 reasoning)
hotfix-security
openai/gpt-5.5:xhigh__json_schema PASS 100% 144 in → 174 out (154 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-luna:high PASS 100% 100 in → 40 out (31 reasoning)
hotfix-security
openai/gpt-5.6-luna:high__json_schema PASS 100% 139 in → 54 out (34 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-luna:low PASS 100% 99 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-luna:low__json_schema PASS 100% 145 in → 41 out (21 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-luna:max PASS 100% 99 in → 87 out (78 reasoning)
hotfix-security
openai/gpt-5.6-luna:max__json_schema PASS 100% 144 in → 90 out (70 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-luna:medium PASS 100% 99 in → 29 out (20 reasoning)
hotfix-security
openai/gpt-5.6-luna:medium__json_schema PASS 100% 144 in → 42 out (22 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-luna:none PASS 100% 99 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-luna:none__json_schema PASS 100% 142 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-luna:xhigh PASS 100% 101 in → 84 out (75 reasoning)
hotfix-security
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 141 in → 65 out (45 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-sol:high PASS 100% 103 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-sol:high__json_schema PASS 100% 143 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-sol:low PASS 100% 96 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-sol:low__json_schema PASS 100% 142 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-sol:max PASS 100% 95 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-sol:max__json_schema PASS 100% 145 in → 100 out (80 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-sol:medium PASS 100% 102 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-sol:medium__json_schema PASS 100% 142 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-sol:none PASS 100% 101 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-sol:none__json_schema PASS 100% 142 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-sol:xhigh PASS 100% 98 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 144 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-terra:high PASS 100% 103 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-terra:high__json_schema PASS 100% 142 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-terra:low PASS 100% 101 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-terra:low__json_schema PASS 100% 143 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-terra:max PASS 100% 101 in → 54 out (45 reasoning)
hotfix-security
openai/gpt-5.6-terra:max__json_schema PASS 100% 144 in → 68 out (48 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-terra:medium PASS 100% 100 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-terra:medium__json_schema PASS 100% 150 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-terra:none PASS 100% 99 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-terra:none__json_schema PASS 100% 145 in → 18 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-5.6-terra:xhigh PASS 100% 99 in → 7 out (0 reasoning)
hotfix-security
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 144 in → 36 out (16 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-oss-120b:high PASS 100% 159 in → 212 out (228 reasoning)
hotfix-security
openai/gpt-oss-120b:high__json_schema PASS 100% 150 in → 209 out (204 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
openai/gpt-oss-120b:low PASS 100% 161 in → 82 out (76 reasoning)
hotfix-security
openai/gpt-oss-120b:low__json_schema PASS 100% 160 in → 53 out (38 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-oss-120b:medium PASS 100% 165 in → 87 out (81 reasoning)
hotfix-security
openai/gpt-oss-120b:medium__json_schema PASS 100% 164 in → 176 out (152 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
openai/gpt-oss-120b:xhigh PASS 100% 161 in → 130 out (127 reasoning)
hotfix-security
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 146 in → 260 out (263 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
openai/gpt-oss-20b:high PASS 100% 163 in → 100 out (97 reasoning)
hotfix-security
openai/gpt-oss-20b:high__json_schema PASS 100% 158 in → 374 out (379 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
openai/gpt-oss-20b:low PASS 100% 148 in → 61 out (53 reasoning)
hotfix-security
openai/gpt-oss-20b:low__json_schema PASS 100% 148 in → 68 out (48 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-oss-20b:medium PASS 100% 160 in → 130 out (117 reasoning)
hotfix-security
openai/gpt-oss-20b:medium__json_schema PASS 100% 146 in → 211 out (211 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
openai/gpt-oss-20b:xhigh PASS 100% 163 in → 320 out (351 reasoning)
hotfix-security
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 163 in → 12 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
poolside/laguna-m.1:high PASS 100% 114 in → 172 out (166 reasoning)
hotfix-security
poolside/laguna-m.1:high__json_schema PASS 100% 115 in → 160 out (137 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
poolside/laguna-m.1:low PASS 100% 113 in → 229 out (223 reasoning)
hotfix-security
poolside/laguna-m.1:low__json_schema PASS 100% 117 in → 514 out (489 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
poolside/laguna-m.1:medium PASS 100% 114 in → 317 out (311 reasoning)
hotfix-security
poolside/laguna-m.1:medium__json_schema PASS 100% 115 in → 237 out (220 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
poolside/laguna-m.1:none FAIL 100% 111 in → 95 out (0 reasoning)
Based on the information provided, the branch `hotfix-security` is listed as merged into `main`. Since both `main` and `hotfix-security` point to the same commit (`f0af7eb`), this confirms that `hotfix-security` has been fully merged and can be safely deleted. The `release-v2` branch does not appear in the merged list and is therefore not recommended for deletion. Branches to delete: hotfix-security
Failure: Extra: ['Based on the information provided, the branch `hotfix-security` is listed as merged into `main`. Since both `main` and `hotfix-security` point to the same commit (`f0af7eb`), this confirms that `hotfix-security` has been fully merged and can be safely deleted. The `release-v2` branch does not appear in the merged list and is therefore not recommended for deletion.', 'Branches to delete:']
poolside/laguna-m.1:none__json_schema PASS 100% 111 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
poolside/laguna-m.1:xhigh PASS 100% 115 in → 135 out (129 reasoning)
hotfix-security
poolside/laguna-m.1:xhigh__json_schema PASS 100% 116 in → 237 out (223 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
poolside/laguna-xs-2.1:high PASS 100% 117 in → 196 out (192 reasoning)
hotfix-security
poolside/laguna-xs-2.1:high__json_schema PASS 100% 113 in → 238 out (226 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
poolside/laguna-xs-2.1:low PASS 100% 117 in → 162 out (158 reasoning)
hotfix-security
poolside/laguna-xs-2.1:low__json_schema PASS 100% 111 in → 134 out (122 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
poolside/laguna-xs-2.1:medium PASS 100% 115 in → 216 out (212 reasoning)
hotfix-security
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 111 in → 213 out (201 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
poolside/laguna-xs-2.1:none PASS 100% 117 in → 3 out (0 reasoning)
hotfix-security
poolside/laguna-xs-2.1:none__json_schema PASS 100% 115 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
poolside/laguna-xs-2.1:xhigh PASS 100% 112 in → 187 out (183 reasoning)
hotfix-security
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 117 in → 247 out (235 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
poolside/laguna-xs.2:high PASS 100% 151 in → 118 out (112 reasoning)
hotfix-security
poolside/laguna-xs.2:high__json_schema PASS 100% 152 in → 221 out (209 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
poolside/laguna-xs.2:low PASS 100% 153 in → 124 out (118 reasoning)
hotfix-security
poolside/laguna-xs.2:low__json_schema PASS 100% 150 in → 214 out (202 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
poolside/laguna-xs.2:medium PASS 100% 148 in → 188 out (182 reasoning)
hotfix-security
poolside/laguna-xs.2:medium__json_schema PASS 100% 152 in → 141 out (129 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
poolside/laguna-xs.2:none PASS 100% 148 in → 5 out (0 reasoning)
hotfix-security
poolside/laguna-xs.2:none__json_schema FAIL 100% 150 in → 16 out (0 reasoning)
hotfix-security release-v2
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security", "release-v2"]}
Failure: Extra: ['release-v2']
poolside/laguna-xs.2:xhigh PASS 100% 150 in → 214 out (208 reasoning)
hotfix-security
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 146 in → 117 out (105 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"]}
qwen/qwen3.6-27b:high PASS 100% 114 in → 357 out (321 reasoning)
hotfix-security
qwen/qwen3.6-27b:high__json_schema PASS 100% 111 in → 873 out (834 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
qwen/qwen3.6-27b:low PASS 100% 112 in → 1,084 out (991 reasoning)
hotfix-security
qwen/qwen3.6-27b:low__json_schema PASS 100% 112 in → 1,062 out (968 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
qwen/qwen3.6-27b:medium PASS 100% 113 in → 313 out (300 reasoning)
hotfix-security
qwen/qwen3.6-27b:medium__json_schema PASS 100% 115 in → 690 out (645 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
qwen/qwen3.6-27b:none PASS 100% 116 in → 4 out (0 reasoning)
hotfix-security
qwen/qwen3.6-27b:none__json_schema PASS 100% 119 in → 16 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
qwen/qwen3.6-27b:xhigh PASS 100% 113 in → 719 out (697 reasoning)
hotfix-security
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 114 in → 404 out (372 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 115 in → 877 out (817 reasoning)
hotfix-security
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 117 in → 674 out (634 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 114 in → 942 out (934 reasoning)
hotfix-security
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 115 in → 948 out (858 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 116 in → 805 out (797 reasoning)
hotfix-security
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 109 in → 621 out (586 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 119 in → 4 out (0 reasoning)
hotfix-security
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 117 in → 22 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 114 in → 764 out (730 reasoning)
hotfix-security
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 114 in → 621 out (547 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
qwen/qwen3.6-flash:high PASS 100% 110 in → 326 out (318 reasoning)
hotfix-security
qwen/qwen3.6-flash:low PASS 100% 117 in → 928 out (920 reasoning)
hotfix-security
qwen/qwen3.6-flash:medium PASS 100% 112 in → 902 out (894 reasoning)
hotfix-security
qwen/qwen3.6-flash:none PASS 100% 119 in → 3 out (0 reasoning)
hotfix-security
qwen/qwen3.6-flash:xhigh PASS 100% 116 in → 755 out (747 reasoning)
hotfix-security
qwen/qwen3.7-flash:high PASS 100% 109 in → 395 out (387 reasoning)
hotfix-security
qwen/qwen3.7-flash:low PASS 100% 115 in → 416 out (408 reasoning)
hotfix-security
qwen/qwen3.7-flash:max PASS 100% 113 in → 388 out (380 reasoning)
hotfix-security
qwen/qwen3.7-flash:medium PASS 100% 116 in → 851 out (843 reasoning)
hotfix-security
qwen/qwen3.7-flash:none PASS 100% 116 in → 3 out (0 reasoning)
hotfix-security
qwen/qwen3.7-flash:xhigh PASS 100% 113 in → 342 out (334 reasoning)
hotfix-security
qwen/qwen3.7-max:high PASS 100% 115 in → 200 out (192 reasoning)
hotfix-security
qwen/qwen3.7-max:low PASS 100% 114 in → 188 out (180 reasoning)
hotfix-security
qwen/qwen3.7-max:medium PASS 100% 117 in → 211 out (203 reasoning)
hotfix-security
qwen/qwen3.7-max:none PASS 100% 117 in → 3 out (0 reasoning)
hotfix-security
qwen/qwen3.7-max:xhigh PASS 100% 115 in → 136 out (128 reasoning)
hotfix-security
qwen/qwen3.7-plus:high PASS 100% 113 in → 383 out (375 reasoning)
hotfix-security
qwen/qwen3.7-plus:low PASS 100% 115 in → 559 out (551 reasoning)
hotfix-security
qwen/qwen3.7-plus:medium PASS 100% 115 in → 363 out (355 reasoning)
hotfix-security
qwen/qwen3.7-plus:none PASS 100% 115 in → 3 out (0 reasoning)
hotfix-security
qwen/qwen3.7-plus:xhigh PASS 100% 115 in → 283 out (275 reasoning)
hotfix-security
tencent/hy3:high PASS 100% 108 in → 392 out (387 reasoning)
hotfix-security
tencent/hy3:high__json_schema PASS 100% 112 in → 579 out (590 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
tencent/hy3:low PASS 100% 111 in → 350 out (365 reasoning)
hotfix-security
tencent/hy3:low__json_schema PASS 100% 106 in → 370 out (377 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
tencent/hy3:medium PASS 100% 108 in → 388 out (383 reasoning)
hotfix-security
tencent/hy3:medium__json_schema PASS 100% 107 in → 323 out (313 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
tencent/hy3:none PASS 100% 112 in → 4 out (0 reasoning)
hotfix-security
tencent/hy3:none__json_schema PASS 100% 110 in → 14 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
tencent/hy3:xhigh PASS 100% 108 in → 642 out (637 reasoning)
hotfix-security
tencent/hy3:xhigh__json_schema PASS 100% 108 in → 456 out (454 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
thinkingmachines/inkling-small:high PASS 100% 110 in → 132 out (123 reasoning)
hotfix-security
thinkingmachines/inkling-small:high__json_schema PASS 100% 114 in → 161 out (144 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
thinkingmachines/inkling-small:low PASS 100% 107 in → 18 out (9 reasoning)
hotfix-security
thinkingmachines/inkling-small:low__json_schema PASS 100% 113 in → 45 out (28 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
thinkingmachines/inkling-small:max PASS 100% 110 in → 113 out (97 reasoning)
hotfix-security
thinkingmachines/inkling-small:max__json_schema PASS 100% 108 in → 221 out (204 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
thinkingmachines/inkling-small:medium PASS 100% 104 in → 89 out (79 reasoning)
hotfix-security
thinkingmachines/inkling-small:medium__json_schema PASS 100% 109 in → 117 out (94 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
thinkingmachines/inkling-small:none PASS 100% 106 in → 6 out (0 reasoning)
hotfix-security
thinkingmachines/inkling-small:xhigh PASS 100% 108 in → 94 out (85 reasoning)
hotfix-security
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 110 in → 111 out (89 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
thinkingmachines/inkling:high PASS 100% 110 in → 243 out (233 reasoning)
hotfix-security
thinkingmachines/inkling:high__json_schema PASS 100% 109 in → 153 out (131 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
thinkingmachines/inkling:low PASS 100% 109 in → 37 out (34 reasoning)
hotfix-security
thinkingmachines/inkling:low__json_schema PASS 100% 105 in → 55 out (33 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
thinkingmachines/inkling:max PASS 100% 107 in → 143 out (133 reasoning)
hotfix-security
thinkingmachines/inkling:max__json_schema PASS 100% 107 in → 227 out (205 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
thinkingmachines/inkling:medium PASS 100% 108 in → 125 out (116 reasoning)
hotfix-security
thinkingmachines/inkling:xhigh PASS 100% 105 in → 168 out (159 reasoning)
hotfix-security
x-ai/grok-4.3:high PASS 100% 275 in → 248 out (245 reasoning)
hotfix-security
x-ai/grok-4.3:high__json_schema PASS 100% 343 in → 439 out (430 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
x-ai/grok-4.3:low FAIL 100% 287 in → 358 out (298 reasoning)
hotfix-security The output includes only the single branch name meeting deletion criteria (fully merged into main via identical tip commit), excluding main (current branch) and release-v2 (unmerged commit). This follows the standard git branch -d semantics where only fully merged refs are candidates for safe removal.
Failure: Extra: ['The output includes only the single branch name meeting deletion criteria (fully merged into main via identical tip commit), excluding main (current branch) and release-v2 (unmerged commit). This follows the standard git branch -d semantics where only fully merged refs are candidates for safe removal.']
x-ai/grok-4.3:low__json_schema PASS 100% 348 in → 403 out (394 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
x-ai/grok-4.3:max PASS 100% 281 in → 446 out (443 reasoning)
hotfix-security
x-ai/grok-4.3:max__json_schema PASS 100% 346 in → 407 out (398 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
x-ai/grok-4.3:medium FAIL 100% 288 in → 569 out (430 reasoning)
hotfix-security The repository shows three branches, with `main` as the current branch (marked by `*`). The commit history indicates `hotfix-security` and `main` point to the identical commit `08be3e1`, confirming `hotfix-security` is fully reachable (i.e., merged) from `main`. `release-v2` points to an unrelated earlier commit `4f215e9` and is not listed among branches merged into `main`, so it is not a candidate for deletion. Per standard Git branch hygiene, `main` itself is never deleted even if shown in a `--merged` listing. Thus only `hotfix-security` qualifies for removal.
Failure: Extra: ['The repository shows three branches, with `main` as the current branch (marked by `*`). The commit history indicates `hotfix-security` and `main` point to the identical commit `08be3e1`, confirming `hotfix-security` is fully reachable (i.e., merged) from `main`. `release-v2` points to an unrelated earlier commit `4f215e9` and is not listed among branches merged into `main`, so it is not a candidate for deletion. Per standard Git branch hygiene, `main` itself is never deleted even if shown in a `--merged` listing. Thus only `hotfix-security` qualifies for removal.']
x-ai/grok-4.3:medium__json_schema PASS 100% 350 in → 360 out (351 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
x-ai/grok-4.3:none__json_schema PASS 100% 341 in → 9 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
x-ai/grok-4.3:xhigh PASS 100% 276 in → 485 out (482 reasoning)
hotfix-security
x-ai/grok-4.3:xhigh__json_schema PASS 100% 347 in → 545 out (536 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
x-ai/grok-4.5:high PASS 100% 299 in → 256 out (253 reasoning)
hotfix-security
x-ai/grok-4.5:high__json_schema PASS 100% 387 in → 372 out (360 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
x-ai/grok-4.5:low PASS 100% 303 in → 184 out (181 reasoning)
hotfix-security
x-ai/grok-4.5:low__json_schema PASS 100% 380 in → 308 out (296 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
x-ai/grok-4.5:max PASS 100% 299 in → 412 out (409 reasoning)
hotfix-security
x-ai/grok-4.5:max__json_schema PASS 100% 383 in → 462 out (450 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
x-ai/grok-4.5:medium PASS 100% 304 in → 232 out (229 reasoning)
hotfix-security
x-ai/grok-4.5:medium__json_schema PASS 100% 385 in → 406 out (394 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
x-ai/grok-4.5:xhigh PASS 100% 299 in → 240 out (237 reasoning)
hotfix-security
x-ai/grok-4.5:xhigh__json_schema PASS 100% 390 in → 345 out (333 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
z-ai/glm-4.7-flash:high PASS 100% 104 in → 560 out (540 reasoning)
hotfix-security
z-ai/glm-4.7-flash:low PASS 100% 103 in → 393 out (372 reasoning)
hotfix-security
z-ai/glm-4.7-flash:low__json_schema PASS 100% 104 in → 454 out (419 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
z-ai/glm-4.7-flash:medium PASS 100% 101 in → 575 out (570 reasoning)
hotfix-security
z-ai/glm-4.7-flash:xhigh PASS 100% 101 in → 470 out (448 reasoning)
hotfix-security
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 104 in → 439 out (406 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": ["hotfix-security"] }
z-ai/glm-5.2:high PASS 100% 108 in → 162 out (0 reasoning)
hotfix-security
z-ai/glm-5.2:low PASS 100% 107 in → 271 out (246 reasoning)
hotfix-security
z-ai/glm-5.2:low__json_schema PASS 100% 102 in → 12 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete": ["hotfix-security"] }
z-ai/glm-5.2:medium PASS 100% 110 in → 300 out (274 reasoning)
hotfix-security
z-ai/glm-5.2:medium__json_schema PASS 100% 109 in → 403 out (362 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
z-ai/glm-5.2:none PASS 100% 101 in → 4 out (0 reasoning)
hotfix-security
z-ai/glm-5.2:none__json_schema PASS 100% 104 in → 20 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "hotfix-security" ] }
z-ai/glm-5.2:xhigh PASS 100% 101 in → 146 out (140 reasoning)
hotfix-security
z-ai/glm-5.2:xhigh__json_schema PASS 100% 111 in → 10 out (0 reasoning)
hotfix-security
JSON Schema Structured Output
(raw) {"branches_to_delete":["hotfix-security"]}
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 102 in → 14 out (0 reasoning)
release-v2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-v2"] }
Failure: Missing: ['hotfix-security'] Extra: ['release-v2']
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0% 184 in → 208 out (192 reasoning)
release-v2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-v2"] }
Failure: Missing: ['hotfix-security'] Extra: ['release-v2']
Invalid JSON. Output: hotfix-security
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:high__json_schema FAIL 0% 134 in → 12 out (11 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['hotfix-security']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 117 in → 2 out (0 reasoning)
none
Failure: Missing: ['hotfix-security'] Extra: ['none']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-48a0bd8a-22e3-9e94-9489-9cc7c799c550","request_id":"48a0bd8a-22e3-9e94-9489-9cc7c799c550"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a73b0936-e571-95db-85cc-f56614072f9f","request_id":"a73b0936-e571-95db-85cc-f56614072f9f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-73d62a1c-ac53-9718-a9a0-9a51ca236de5","request_id":"73d62a1c-ac53-9718-a9a0-9a51ca236de5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-76e78079-5aad-9141-97c2-089b81371959","request_id":"76e78079-5aad-9141-97c2-089b81371959"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-337f2f32-6840-9c72-8bab-be6cf82ec699","request_id":"337f2f32-6840-9c72-8bab-be6cf82ec699"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-039e852e-341b-9015-b542-b59652101d65"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ce9834f7-9151-9a70-992d-151acfe650cf"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e5bae432-0ff8-9a00-855a-48c54f0ad57f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-26190aa5-8653-9b42-a3bf-a796b37023f4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2c2a3892-4c60-9081-af71-0e2cc54f0544"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3aab2113-6709-9a5f-bc23-e1bf11208511"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6b832b8d-ecee-94cb-a6b5-3a7e506a5171","request_id":"6b832b8d-ecee-94cb-a6b5-3a7e506a5171"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fcb6fb76-c514-907f-b293-e4e501b2ee27","request_id":"fcb6fb76-c514-907f-b293-e4e501b2ee27"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c5823a9b-4510-9a2d-b8c2-67f3a0696c7a","request_id":"c5823a9b-4510-9a2d-b8c2-67f3a0696c7a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ca09549f-ec3f-957d-bb8e-02c47aa073dc","request_id":"ca09549f-ec3f-957d-bb8e-02c47aa073dc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3b4d41c1-eef4-9029-a81d-f2a871a5bfad","request_id":"3b4d41c1-eef4-9029-a81d-f2a871a5bfad"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-efad278f-f5b9-90b6-aafc-3c5813cf3188","request_id":"efad278f-f5b9-90b6-aafc-3c5813cf3188"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-551a994a-f088-9fd7-a41d-3531da20a5d4","request_id":"551a994a-f088-9fd7-a41d-3531da20a5d4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6ee073d5-827a-945d-a346-d8f5c69ddaa6","request_id":"6ee073d5-827a-945d-a346-d8f5c69ddaa6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-68cef022-9897-9ef6-a3d8-534388b6ef4d","request_id":"68cef022-9897-9ef6-a3d8-534388b6ef4d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-beddf9bd-eb2f-9db2-86e6-8ca30dc3adf8","request_id":"beddf9bd-eb2f-9db2-86e6-8ca30dc3adf8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: hotfix-security{"branches_to_delete": ["hotfix-security"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: hotfix-security
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 107 in → 6 out (0 reasoning)
hotfix-security<|end_message|>
Failure: Missing: ['hotfix-security'] Extra: ['hotfix-security<|end_message|>']
Invalid JSON. Output: hotfix-security{ "branches_to_delete": ["hotfix-security"] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: hotfix-security
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none FAIL 0% 278 in → 1 out (0 reasoning)
none
Failure: Missing: ['hotfix-security'] Extra: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 105 in → 3 out (0 reasoning)
release-v2
Failure: Missing: ['hotfix-security'] Extra: ['release-v2']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 105 in → 13 out (0 reasoning)
release-v2
JSON Schema Structured Output
(raw) { "branches_to_delete": ["release-v2"] }
Failure: Missing: ['hotfix-security'] Extra: ['release-v2']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)