Single unmerged branch
Tests ability to identify that zero branches should be deleted when only unmerged branches exist. Evaluates the all-unmerged edge case.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.py
  5. 05 git add app.py
  6. 06 git commit -m 'Initial'
  7. 07 git checkout -b feature-experimental
  8. 08 echo 'experimental' > exp.py
  9. 09 git add exp.py
  10. 10 git commit -m 'Experiment'
  11. 11 git checkout main
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
none

Scoped model quality, cost, API time, and token usage for branch_cleanup/f009.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high FAIL 100% 109 in → 116 out (40 reasoning)
none The "Merged into main" list only contains `main` itself. The `feature-experimental` branch (commit d1c2161) is not in the merged list, meaning it contains work not yet merged into main — deleting it would lose that commit. No branches are safe to delete.
Failure: Extra: ['The "Merged into main" list only contains `main` itself. The `feature-experimental` branch (commit d1c2161) is not in the merged list, meaning it contains work not yet merged into main — deleting it would lose that commit. No branches are safe to delete.']
anthropic/claude-fable-5:low FAIL 100% 107 in → 76 out (12 reasoning)
none No branches should be deleted. The only branch fully merged into main is main itself, and `feature-experimental` (commit 572baec) is not in the merged list, meaning it contains unmerged work.
Failure: Extra: ['No branches should be deleted. The only branch fully merged into main is main itself, and `feature-experimental` (commit 572baec) is not in the merged list, meaning it contains unmerged work.']
anthropic/claude-fable-5:max FAIL 100% 110 in → 325 out (84 reasoning)
none The `git branch --merged main` output shows only `main` itself. The `feature-experimental` branch (157db6e) does not appear in the merged list, meaning its commit has not been merged into main — deleting it would lose work.
Failure: Extra: ['The `git branch --merged main` output shows only `main` itself. The `feature-experimental` branch (157db6e) does not appear in the merged list, meaning its commit has not been merged into main — deleting it would lose work.']
anthropic/claude-fable-5:medium FAIL 100% 109 in → 101 out (40 reasoning)
none The "merged into main" list shows only `main` itself. The `feature-experimental` branch (at commit 93db705) is not in the merged list, meaning it contains work not yet in main, so it should not be deleted.
Failure: Extra: ['The "merged into main" list shows only `main` itself. The `feature-experimental` branch (at commit 93db705) is not in the merged list, meaning it contains work not yet in main, so it should not be deleted.']
anthropic/claude-fable-5:xhigh FAIL 100% 109 in → 128 out (34 reasoning)
none The `git branch --merged main` output only shows `main` itself, which should never be deleted. The `feature-experimental` branch does not appear in the merged list, meaning its commit (`ced1951 Experiment`) contains work not yet merged into main — deleting it would lose that work.
Failure: Extra: ['The `git branch --merged main` output only shows `main` itself, which should never be deleted. The `feature-experimental` branch does not appear in the merged list, meaning its commit (`ced1951 Experiment`) contains work not yet merged into main — deleting it would lose that work.']
anthropic/claude-haiku-4.5:high PASS 100% 117 in → 142 out (141 reasoning)
none
anthropic/claude-haiku-4.5:low PASS 100% 118 in → 186 out (193 reasoning)
none
anthropic/claude-haiku-4.5:medium PASS 100% 114 in → 257 out (261 reasoning)
none
anthropic/claude-haiku-4.5:none PASS 100% 87 in → 4 out (0 reasoning)
none
anthropic/claude-haiku-4.5:xhigh PASS 100% 114 in → 180 out (180 reasoning)
none
anthropic/claude-opus-4.6:high PASS 100% 87 in → 133 out (112 reasoning)
none
anthropic/claude-opus-4.6:low PASS 100% 86 in → 4 out (0 reasoning)
none
anthropic/claude-opus-4.6:max FAIL 100% 86 in → 166 out (132 reasoning)
none The `feature-experimental` branch is **not** listed as merged into `main`, so it should not be deleted.
Failure: Extra: ['The `feature-experimental` branch is **not** listed as merged into `main`, so it should not be deleted.']
anthropic/claude-opus-4.6:medium PASS 100% 86 in → 41 out (24 reasoning)
none
anthropic/claude-opus-4.6:none FAIL 100% 86 in → 75 out (0 reasoning)
Looking at the information provided: - `feature-experimental` (commit 7e74602) is **not** listed as merged into main. - The only branch listed as merged into main is `main` itself. Since `feature-experimental` is not fully merged into main, there are no branches that should be deleted. none
Failure: Extra: ['- The only branch listed as merged into main is `main` itself.', '- `feature-experimental` (commit 7e74602) is **not** listed as merged into main.', 'Looking at the information provided:', 'Since `feature-experimental` is not fully merged into main, there are no branches that should be deleted.']
anthropic/claude-opus-4.6:xhigh PASS 100% 89 in → 119 out (91 reasoning)
none
anthropic/claude-opus-4.7:high PASS 100% 114 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:low PASS 100% 115 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:max PASS 100% 114 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:medium PASS 100% 114 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:none PASS 100% 113 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:xhigh PASS 100% 114 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.8:high FAIL 100% 109 in → 208 out (37 reasoning)
none The `feature-experimental` branch is **not** in the "Merged into main" list, meaning it contains unmerged work (commit `4d498a7 "Experiment"`). Deleting it would lose that work. The only branch shown as merged is `main` itself, which obviously shouldn't be deleted. Therefore, no branches are safe to delete.
Failure: Extra: ['The `feature-experimental` branch is **not** in the "Merged into main" list, meaning it contains unmerged work (commit `4d498a7 "Experiment"`). Deleting it would lose that work.', "The only branch shown as merged is `main` itself, which obviously shouldn't be deleted.", 'Therefore, no branches are safe to delete.']
anthropic/claude-opus-4.8:low PASS 100% 110 in → 3 out (0 reasoning)
none
anthropic/claude-opus-4.8:max FAIL 100% 111 in → 508 out (75 reasoning)
none **Reasoning:** - `feature-experimental` (89f6e08) is **not** listed under "Merged into main," so it contains unmerged work and should not be deleted. - `main` is the primary branch itself, so it wouldn't be deleted. No branches are safe to delete based on the merge status shown.
Failure: Extra: ['**Reasoning:**', '- `feature-experimental` (89f6e08) is **not** listed under "Merged into main," so it contains unmerged work and should not be deleted.', "- `main` is the primary branch itself, so it wouldn't be deleted.", 'No branches are safe to delete based on the merge status shown.']
anthropic/claude-opus-4.8:medium PASS 100% 111 in → 3 out (0 reasoning)
none
anthropic/claude-opus-4.8:none FAIL 100% 108 in → 59 out (0 reasoning)
Based on the information provided, the only branch merged into main is `main` itself. The `feature-experimental` branch is **not** listed as merged into main, so it should not be deleted. none
Failure: Extra: ['Based on the information provided, the only branch merged into main is `main` itself. The `feature-experimental` branch is **not** listed as merged into main, so it should not be deleted.']
anthropic/claude-opus-4.8:xhigh FAIL 100% 111 in → 272 out (39 reasoning)
none The only branch listed as merged into `main` is `main` itself (which you would never delete). The `feature-experimental` branch is **not** in the merged list, so it contains unmerged work and should not be deleted.
Failure: Extra: ['The only branch listed as merged into `main` is `main` itself (which you would never delete). The `feature-experimental` branch is **not** in the merged list, so it contains unmerged work and should not be deleted.']
anthropic/claude-sonnet-4.6:high PASS 100% 88 in → 118 out (102 reasoning)
none
anthropic/claude-sonnet-4.6:low PASS 100% 88 in → 4 out (0 reasoning)
none
anthropic/claude-sonnet-4.6:max PASS 100% 87 in → 125 out (115 reasoning)
none
anthropic/claude-sonnet-4.6:medium PASS 100% 86 in → 4 out (0 reasoning)
none
anthropic/claude-sonnet-4.6:none PASS 100% 87 in → 4 out (0 reasoning)
none
anthropic/claude-sonnet-4.6:xhigh PASS 100% 88 in → 113 out (100 reasoning)
none
anthropic/claude-sonnet-5:high FAIL 100% 112 in → 111 out (0 reasoning)
Based on the information provided, here are my findings: **Branches merged into main:** Only `main` itself appears in the merged list. **Answer:** none The `feature-experimental` branch is not listed as merged into main, so it should NOT be deleted (it likely contains unmerged commits with the "Experiment" work that would be lost).
Failure: Extra: ['**Answer:**', '**Branches merged into main:**', 'Based on the information provided, here are my findings:', 'Only `main` itself appears in the merged list.', 'The `feature-experimental` branch is not listed as merged into main, so it should NOT be deleted (it likely contains unmerged commits with the "Experiment" work that would be lost).']
anthropic/claude-sonnet-5:low PASS 100% 111 in → 3 out (0 reasoning)
none
anthropic/claude-sonnet-5:medium PASS 100% 110 in → 48 out (29 reasoning)
none
anthropic/claude-sonnet-5:xhigh PASS 100% 109 in → 3 out (0 reasoning)
none
arcee-ai/trinity-large-thinking:high PASS 100% 81 in → 650 out (663 reasoning)
none
arcee-ai/trinity-large-thinking:low PASS 100% 83 in → 1,149 out (1,180 reasoning)
none
arcee-ai/trinity-large-thinking:medium PASS 100% 79 in → 1,349 out (1,372 reasoning)
none
arcee-ai/trinity-large-thinking:xhigh PASS 100% 80 in → 1,747 out (1,795 reasoning)
none
arcee-ai/trinity-mini:high PASS 100% 79 in → 293 out (339 reasoning)
none
arcee-ai/trinity-mini:high__json_schema PASS 100% 77 in → 531 out (636 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
arcee-ai/trinity-mini:low PASS 100% 79 in → 538 out (664 reasoning)
none
arcee-ai/trinity-mini:low__json_schema PASS 100% 82 in → 322 out (374 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
arcee-ai/trinity-mini:medium PASS 100% 78 in → 389 out (462 reasoning)
none
arcee-ai/trinity-mini:medium__json_schema PASS 100% 76 in → 373 out (416 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 77 in → 393 out (456 reasoning)
none
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 79 in → 483 out (562 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 164 in → 125 out (135 reasoning)
none
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 158 in → 251 out (253 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 155 in → 72 out (70 reasoning)
none
deepseek/deepseek-v4-flash-0731:max PASS 100% 157 in → 85 out (83 reasoning)
none
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 156 in → 93 out (90 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete":["none"] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 80 in → 230 out (228 reasoning)
none
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 77 in → 152 out (217 reasoning)
none
deepseek/deepseek-v4-flash:high PASS 100% 78 in → 146 out (143 reasoning)
none
deepseek/deepseek-v4-flash:low PASS 100% 76 in → 131 out (128 reasoning)
none
deepseek/deepseek-v4-flash:medium PASS 100% 75 in → 200 out (197 reasoning)
none
deepseek/deepseek-v4-flash:none PASS 100% 79 in → 2 out (0 reasoning)
none
deepseek/deepseek-v4-flash:xhigh PASS 100% 155 in → 176 out (197 reasoning)
none
deepseek/deepseek-v4-pro:high PASS 100% 81 in → 191 out (188 reasoning)
none
deepseek/deepseek-v4-pro:low PASS 100% 82 in → 325 out (322 reasoning)
none
deepseek/deepseek-v4-pro:medium PASS 100% 76 in → 119 out (116 reasoning)
none
deepseek/deepseek-v4-pro:xhigh PASS 100% 157 in → 703 out (700 reasoning)
none
google/gemini-3-flash-preview:high PASS 100% 77 in → 195 out (194 reasoning)
none
google/gemini-3-flash-preview:low PASS 100% 76 in → 172 out (171 reasoning)
none
google/gemini-3-flash-preview:medium PASS 100% 77 in → 239 out (238 reasoning)
none
google/gemini-3-flash-preview:xhigh PASS 100% 75 in → 195 out (194 reasoning)
none
google/gemini-3.1-flash-lite-preview:high PASS 100% 78 in → 363 out (362 reasoning)
none
google/gemini-3.1-flash-lite-preview:low PASS 100% 75 in → 109 out (108 reasoning)
none
google/gemini-3.1-flash-lite-preview:medium PASS 100% 78 in → 175 out (174 reasoning)
none
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 79 in → 434 out (433 reasoning)
none
google/gemini-3.1-pro-preview:high PASS 100% 77 in → 148 out (147 reasoning)
none
google/gemini-3.1-pro-preview:low PASS 100% 74 in → 242 out (241 reasoning)
none
google/gemini-3.1-pro-preview:medium PASS 100% 77 in → 296 out (295 reasoning)
none
google/gemini-3.1-pro-preview:xhigh PASS 100% 73 in → 273 out (272 reasoning)
none
google/gemini-3.5-flash-lite:high PASS 100% 78 in → 377 out (376 reasoning)
none
google/gemini-3.5-flash-lite:low PASS 100% 77 in → 1 out (0 reasoning)
none
google/gemini-3.5-flash-lite:max PASS 100% 76 in → 275 out (274 reasoning)
none
google/gemini-3.5-flash-lite:medium PASS 100% 77 in → 211 out (210 reasoning)
none
google/gemini-3.5-flash-lite:xhigh PASS 100% 79 in → 307 out (306 reasoning)
none
google/gemini-3.5-flash:high PASS 100% 77 in → 313 out (312 reasoning)
none
google/gemini-3.5-flash:low PASS 100% 77 in → 283 out (282 reasoning)
none
google/gemini-3.5-flash:medium PASS 100% 77 in → 296 out (295 reasoning)
none
google/gemini-3.5-flash:xhigh PASS 100% 76 in → 349 out (348 reasoning)
none
google/gemini-3.6-flash:high PASS 100% 78 in → 282 out (281 reasoning)
none
google/gemini-3.6-flash:low PASS 100% 76 in → 123 out (122 reasoning)
none
google/gemini-3.6-flash:max PASS 100% 75 in → 292 out (291 reasoning)
none
google/gemini-3.6-flash:medium PASS 100% 78 in → 296 out (295 reasoning)
none
google/gemini-3.6-flash:xhigh PASS 100% 76 in → 314 out (313 reasoning)
none
google/gemma-4-26b-a4b-it:high PASS 100% 92 in → 262 out (244 reasoning)
none
google/gemma-4-26b-a4b-it:low PASS 100% 90 in → 466 out (432 reasoning)
none
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 89 in → 257 out (218 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 94 in → 232 out (211 reasoning)
none
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 91 in → 297 out (269 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 89 in → 2 out (0 reasoning)
none
google/gemma-4-26b-a4b-it:xhigh PASS 100% 92 in → 241 out (221 reasoning)
none
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 89 in → 271 out (238 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-31b-it:high PASS 100% 93 in → 157 out (150 reasoning)
none
google/gemma-4-31b-it:high__json_schema PASS 100% 91 in → 236 out (198 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-31b-it:low PASS 100% 92 in → 191 out (1 reasoning)
none
google/gemma-4-31b-it:medium PASS 100% 93 in → 133 out (126 reasoning)
none
google/gemma-4-31b-it:medium__json_schema PASS 100% 92 in → 242 out (207 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-31b-it:none PASS 100% 90 in → 2 out (0 reasoning)
none
google/gemma-4-31b-it:none__json_schema PASS 100% 89 in → 11 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
google/gemma-4-31b-it:xhigh PASS 100% 94 in → 203 out (1 reasoning)
none
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 91 in → 214 out (170 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
minimax/minimax-m2.5:high PASS 100% 114 in → 247 out (245 reasoning)
none
minimax/minimax-m2.5:low PASS 100% 109 in → 424 out (511 reasoning)
none
minimax/minimax-m2.5:xhigh PASS 100% 109 in → 326 out (358 reasoning)
none
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 117 in → 359 out (414 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
minimax/minimax-m2.7:high PASS 100% 111 in → 315 out (313 reasoning)
none
minimax/minimax-m2.7:low PASS 100% 111 in → 328 out (325 reasoning)
none
minimax/minimax-m2.7:medium PASS 100% 110 in → 279 out (320 reasoning)
none
minimax/minimax-m2.7:xhigh PASS 100% 115 in → 283 out (281 reasoning)
none
minimax/minimax-m3:high PASS 100% 248 in → 172 out (198 reasoning)
none
minimax/minimax-m3:low PASS 100% 250 in → 216 out (247 reasoning)
none
minimax/minimax-m3:medium PASS 100% 249 in → 638 out (751 reasoning)
none
minimax/minimax-m3:xhigh PASS 100% 248 in → 124 out (135 reasoning)
none
mistralai/devstral-2512 PASS 100% 82 in → 2 out
none
mistralai/mistral-medium-3-5:high PASS 100% 92 in → 184 out (211 reasoning)
none
mistralai/mistral-medium-3-5:low PASS 100% 92 in → 194 out (219 reasoning)
none
mistralai/mistral-medium-3-5:medium PASS 100% 92 in → 275 out (306 reasoning)
none
mistralai/mistral-medium-3-5:none PASS 100% 94 in → 2 out (0 reasoning)
none
mistralai/mistral-medium-3-5:xhigh PASS 100% 94 in → 132 out (147 reasoning)
none
moonshotai/kimi-k2.6:high PASS 100% 79 in → 199 out (214 reasoning)
none
moonshotai/kimi-k2.6:low PASS 100% 77 in → 425 out (422 reasoning)
none
moonshotai/kimi-k2.6:low__json_schema PASS 100% 81 in → 267 out (256 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
moonshotai/kimi-k2.6:medium PASS 100% 78 in → 726 out (723 reasoning)
none
moonshotai/kimi-k2.6:xhigh PASS 100% 79 in → 364 out (413 reasoning)
none
moonshotai/kimi-k2.7-code:high PASS 100% 79 in → 149 out (146 reasoning)
none
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 80 in → 130 out (112 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
moonshotai/kimi-k2.7-code:low PASS 100% 80 in → 120 out (117 reasoning)
none
moonshotai/kimi-k2.7-code:medium PASS 100% 80 in → 257 out (254 reasoning)
none
moonshotai/kimi-k2.7-code:xhigh PASS 100% 79 in → 77 out (74 reasoning)
none
moonshotai/kimi-k3:high PASS 100% 156 in → 116 out (100 reasoning)
none
moonshotai/kimi-k3:high__json_schema PASS 100% 155 in → 66 out (50 reasoning)
none
moonshotai/kimi-k3:low PASS 100% 156 in → 169 out (153 reasoning)
none
moonshotai/kimi-k3:low__json_schema PASS 100% 155 in → 173 out (157 reasoning)
none
moonshotai/kimi-k3:max PASS 100% 155 in → 110 out (94 reasoning)
none
moonshotai/kimi-k3:max__json_schema PASS 100% 158 in → 115 out (99 reasoning)
none
moonshotai/kimi-k3:medium PASS 100% 155 in → 175 out (159 reasoning)
none
moonshotai/kimi-k3:medium__json_schema PASS 100% 157 in → 163 out (147 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 92 in → 507 out (589 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 94 in → 139 out (144 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 96 in → 175 out (192 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 95 in → 183 out (190 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 96 in → 261 out (274 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 94 in → 2 out (0 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 95 in → 375 out (412 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 92 in → 212 out (221 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 92 in → 91 out (90 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 95 in → 108 out (97 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 94 in → 181 out (169 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 95 in → 2 out (0 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 92 in → 68 out (66 reasoning)
none
openai/gpt-5.4-mini:high PASS 100% 77 in → 103 out (96 reasoning)
none
openai/gpt-5.4-mini:low PASS 100% 77 in → 58 out (51 reasoning)
none
openai/gpt-5.4-mini:medium PASS 100% 78 in → 98 out (91 reasoning)
none
openai/gpt-5.4-mini:medium__json_schema PASS 100% 122 in → 320 out (302 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.4-mini:none PASS 100% 79 in → 5 out (0 reasoning)
none
openai/gpt-5.4-mini:xhigh PASS 100% 79 in → 232 out (225 reasoning)
none
openai/gpt-5.4-nano:high PASS 100% 77 in → 64 out (57 reasoning)
none
openai/gpt-5.4-nano:low PASS 100% 75 in → 21 out (14 reasoning)
none
openai/gpt-5.4-nano:medium PASS 100% 75 in → 40 out (33 reasoning)
none
openai/gpt-5.4-nano:medium__json_schema PASS 100% 120 in → 62 out (44 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.4-nano:none PASS 100% 76 in → 5 out (0 reasoning)
none
openai/gpt-5.4-nano:none__json_schema PASS 100% 119 in → 16 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.4-nano:xhigh PASS 100% 77 in → 206 out (199 reasoning)
none
openai/gpt-5.4:high PASS 100% 79 in → 101 out (94 reasoning)
none
openai/gpt-5.4:low PASS 100% 79 in → 32 out (25 reasoning)
none
openai/gpt-5.4:low__json_schema PASS 100% 122 in → 43 out (25 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.4:medium PASS 100% 76 in → 76 out (69 reasoning)
none
openai/gpt-5.4:none PASS 100% 77 in → 5 out (0 reasoning)
none
openai/gpt-5.4:xhigh PASS 100% 78 in → 109 out (102 reasoning)
none
openai/gpt-5.4:xhigh__json_schema PASS 100% 119 in → 106 out (88 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.5:high PASS 100% 77 in → 65 out (58 reasoning)
none
openai/gpt-5.5:low PASS 100% 77 in → 33 out (26 reasoning)
none
openai/gpt-5.5:medium PASS 100% 76 in → 52 out (45 reasoning)
none
openai/gpt-5.5:none PASS 100% 77 in → 5 out (0 reasoning)
none
openai/gpt-5.5:xhigh PASS 100% 79 in → 92 out (85 reasoning)
none
openai/gpt-5.6-luna:high PASS 100% 78 in → 38 out (31 reasoning)
none
openai/gpt-5.6-luna:high__json_schema PASS 100% 118 in → 58 out (40 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.6-luna:low PASS 100% 76 in → 27 out (20 reasoning)
none
openai/gpt-5.6-luna:low__json_schema PASS 100% 118 in → 46 out (28 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.6-luna:max PASS 100% 78 in → 68 out (61 reasoning)
none
openai/gpt-5.6-luna:max__json_schema PASS 100% 121 in → 112 out (94 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.6-luna:medium PASS 100% 77 in → 28 out (21 reasoning)
none
openai/gpt-5.6-luna:none PASS 100% 77 in → 5 out (0 reasoning)
none
openai/gpt-5.6-luna:xhigh PASS 100% 76 in → 62 out (55 reasoning)
none
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 121 in → 88 out (70 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.6-sol:high PASS 100% 77 in → 5 out (0 reasoning)
none
openai/gpt-5.6-sol:low PASS 100% 76 in → 5 out (0 reasoning)
none
openai/gpt-5.6-sol:max PASS 100% 77 in → 38 out (31 reasoning)
none
openai/gpt-5.6-sol:medium PASS 100% 77 in → 5 out (0 reasoning)
none
openai/gpt-5.6-sol:none PASS 100% 73 in → 5 out (0 reasoning)
none
openai/gpt-5.6-sol:xhigh PASS 100% 76 in → 5 out (0 reasoning)
none
openai/gpt-5.6-terra:high PASS 100% 78 in → 29 out (22 reasoning)
none
openai/gpt-5.6-terra:low PASS 100% 75 in → 21 out (14 reasoning)
none
openai/gpt-5.6-terra:max PASS 100% 75 in → 82 out (75 reasoning)
none
openai/gpt-5.6-terra:medium PASS 100% 79 in → 28 out (21 reasoning)
none
openai/gpt-5.6-terra:none PASS 100% 76 in → 5 out (0 reasoning)
none
openai/gpt-5.6-terra:xhigh PASS 100% 76 in → 38 out (31 reasoning)
none
openai/gpt-oss-120b:high PASS 100% 151 in → 328 out (0 reasoning)
none
openai/gpt-oss-120b:low PASS 100% 125 in → 77 out (78 reasoning)
none
openai/gpt-oss-120b:medium PASS 100% 123 in → 126 out (128 reasoning)
none
openai/gpt-oss-120b:medium__json_schema PASS 100% 134 in → 166 out (174 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-oss-120b:xhigh PASS 100% 124 in → 244 out (280 reasoning)
none
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 134 in → 176 out (173 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
openai/gpt-oss-20b:high PASS 100% 125 in → 495 out (562 reasoning)
none
openai/gpt-oss-20b:low PASS 100% 123 in → 38 out (30 reasoning)
none
openai/gpt-oss-20b:medium PASS 100% 139 in → 114 out (119 reasoning)
none
openai/gpt-oss-20b:xhigh PASS 100% 139 in → 336 out (375 reasoning)
none
poolside/laguna-m.1:high PASS 100% 97 in → 223 out (219 reasoning)
none
poolside/laguna-m.1:high__json_schema PASS 100% 98 in → 635 out (625 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-m.1:low PASS 100% 99 in → 351 out (347 reasoning)
none
poolside/laguna-m.1:medium PASS 100% 100 in → 777 out (773 reasoning)
none
poolside/laguna-m.1:medium__json_schema PASS 100% 100 in → 815 out (801 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
poolside/laguna-xs-2.1:high PASS 100% 97 in → 300 out (298 reasoning)
none
poolside/laguna-xs-2.1:low PASS 100% 97 in → 192 out (190 reasoning)
none
poolside/laguna-xs-2.1:medium PASS 100% 97 in → 192 out (190 reasoning)
none
poolside/laguna-xs-2.1:xhigh PASS 100% 97 in → 148 out (146 reasoning)
none
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 99 in → 412 out (401 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs.2:high PASS 100% 134 in → 492 out (488 reasoning)
none
poolside/laguna-xs.2:low PASS 100% 133 in → 146 out (142 reasoning)
none
poolside/laguna-xs.2:medium PASS 100% 135 in → 189 out (185 reasoning)
none
poolside/laguna-xs.2:none PASS 100% 135 in → 3 out (0 reasoning)
none
poolside/laguna-xs.2:xhigh PASS 100% 136 in → 237 out (233 reasoning)
none
qwen/qwen3.6-27b:high PASS 100% 90 in → 315 out (307 reasoning)
none
qwen/qwen3.6-27b:low PASS 100% 91 in → 823 out (779 reasoning)
none
qwen/qwen3.6-27b:medium PASS 100% 91 in → 866 out (840 reasoning)
none
qwen/qwen3.6-27b:none PASS 100% 91 in → 2 out (0 reasoning)
none
qwen/qwen3.6-27b:xhigh PASS 100% 90 in → 1,170 out (1,185 reasoning)
none
qwen/qwen3.6-35b-a3b:high PASS 100% 90 in → 747 out (742 reasoning)
none
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 84 in → 1,033 out (1,002 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
qwen/qwen3.6-35b-a3b:low PASS 100% 88 in → 1,132 out (1,151 reasoning)
none
qwen/qwen3.6-35b-a3b:medium PASS 100% 90 in → 579 out (552 reasoning)
none
qwen/qwen3.6-35b-a3b:none PASS 100% 91 in → 2 out (0 reasoning)
none
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 90 in → 949 out (944 reasoning)
none
qwen/qwen3.6-flash:high PASS 100% 90 in → 1,160 out (1,155 reasoning)
none
qwen/qwen3.6-flash:low PASS 100% 90 in → 1,055 out (1,050 reasoning)
none
qwen/qwen3.6-flash:medium PASS 100% 89 in → 760 out (754 reasoning)
none
qwen/qwen3.6-flash:xhigh PASS 100% 90 in → 1,362 out (1,356 reasoning)
none
qwen/qwen3.7-flash:high PASS 100% 89 in → 866 out (861 reasoning)
none
qwen/qwen3.7-flash:low PASS 100% 88 in → 1,213 out (1,207 reasoning)
none
qwen/qwen3.7-flash:max PASS 100% 89 in → 877 out (871 reasoning)
none
qwen/qwen3.7-flash:medium PASS 100% 88 in → 1,057 out (1,052 reasoning)
none
qwen/qwen3.7-flash:xhigh PASS 100% 88 in → 586 out (580 reasoning)
none
qwen/qwen3.7-max:high PASS 100% 88 in → 373 out (367 reasoning)
none
qwen/qwen3.7-max:low PASS 100% 90 in → 211 out (206 reasoning)
none
qwen/qwen3.7-max:medium PASS 100% 90 in → 394 out (389 reasoning)
none
qwen/qwen3.7-max:none PASS 100% 89 in → 1 out (0 reasoning)
none
qwen/qwen3.7-max:xhigh PASS 100% 90 in → 502 out (497 reasoning)
none
qwen/qwen3.7-plus:high PASS 100% 91 in → 555 out (549 reasoning)
none
qwen/qwen3.7-plus:low PASS 100% 90 in → 508 out (502 reasoning)
none
qwen/qwen3.7-plus:medium PASS 100% 89 in → 221 out (215 reasoning)
none
qwen/qwen3.7-plus:none PASS 100% 92 in → 1 out (0 reasoning)
none
qwen/qwen3.7-plus:xhigh PASS 100% 88 in → 367 out (362 reasoning)
none
tencent/hy3:high PASS 100% 86 in → 442 out (439 reasoning)
none
tencent/hy3:high__json_schema PASS 100% 89 in → 621 out (660 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
tencent/hy3:low PASS 100% 89 in → 328 out (325 reasoning)
none
tencent/hy3:low__json_schema PASS 100% 88 in → 339 out (369 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
tencent/hy3:medium PASS 100% 86 in → 420 out (417 reasoning)
none
tencent/hy3:medium__json_schema PASS 100% 85 in → 336 out (350 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
tencent/hy3:none PASS 100% 89 in → 2 out (0 reasoning)
none
tencent/hy3:none__json_schema PASS 100% 93 in → 12 out (0 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
tencent/hy3:xhigh PASS 100% 88 in → 327 out (324 reasoning)
none
tencent/hy3:xhigh__json_schema PASS 100% 86 in → 441 out (459 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
thinkingmachines/inkling-small:high PASS 100% 88 in → 135 out (139 reasoning)
none
thinkingmachines/inkling-small:high__json_schema PASS 100% 87 in → 145 out (125 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
thinkingmachines/inkling-small:low PASS 100% 84 in → 71 out (67 reasoning)
none
thinkingmachines/inkling-small:max PASS 100% 87 in → 180 out (203 reasoning)
none
thinkingmachines/inkling-small:medium PASS 100% 86 in → 129 out (132 reasoning)
none
thinkingmachines/inkling-small:none PASS 100% 86 in → 4 out (0 reasoning)
none
thinkingmachines/inkling-small:xhigh PASS 100% 87 in → 211 out (204 reasoning)
none
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 86 in → 268 out (251 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
thinkingmachines/inkling:high PASS 100% 84 in → 366 out (444 reasoning)
none
thinkingmachines/inkling:high__json_schema PASS 100% 86 in → 406 out (392 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
thinkingmachines/inkling:low PASS 100% 84 in → 54 out (47 reasoning)
none
thinkingmachines/inkling:max PASS 100% 86 in → 230 out (222 reasoning)
none
thinkingmachines/inkling:medium PASS 100% 88 in → 191 out (222 reasoning)
none
thinkingmachines/inkling:xhigh PASS 100% 85 in → 372 out (365 reasoning)
none
thinkingmachines/inkling:xhigh__json_schema PASS 100% 85 in → 440 out (420 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
x-ai/grok-4.3:high PASS 100% 258 in → 451 out (450 reasoning)
none
x-ai/grok-4.3:low PASS 100% 262 in → 269 out (268 reasoning)
none
x-ai/grok-4.3:max PASS 100% 256 in → 312 out (311 reasoning)
none
x-ai/grok-4.3:medium PASS 100% 263 in → 348 out (347 reasoning)
none
x-ai/grok-4.3:none PASS 100% 253 in → 1 out (0 reasoning)
none
x-ai/grok-4.3:xhigh PASS 100% 255 in → 291 out (290 reasoning)
none
x-ai/grok-4.5:high PASS 100% 278 in → 260 out (259 reasoning)
none
x-ai/grok-4.5:low PASS 100% 277 in → 189 out (188 reasoning)
none
x-ai/grok-4.5:max PASS 100% 277 in → 286 out (285 reasoning)
none
x-ai/grok-4.5:medium PASS 100% 277 in → 248 out (247 reasoning)
none
x-ai/grok-4.5:xhigh PASS 100% 278 in → 295 out (294 reasoning)
none
z-ai/glm-4.7-flash:high PASS 100% 80 in → 1,619 out (1,648 reasoning)
none
z-ai/glm-4.7-flash:low PASS 100% 79 in → 6,121 out (5,989 reasoning)
none
z-ai/glm-4.7-flash:medium PASS 100% 80 in → 712 out (768 reasoning)
none
z-ai/glm-4.7-flash:xhigh PASS 100% 77 in → 896 out (913 reasoning)
none
z-ai/glm-5.2:high PASS 100% 84 in → 215 out (218 reasoning)
none
z-ai/glm-5.2:low PASS 100% 87 in → 104 out (109 reasoning)
none
z-ai/glm-5.2:medium PASS 100% 85 in → 293 out (305 reasoning)
none
z-ai/glm-5.2:medium__json_schema PASS 100% 86 in → 103 out (113 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
z-ai/glm-5.2:none PASS 100% 79 in → 1 out (0 reasoning)
none
z-ai/glm-5.2:xhigh PASS 100% 87 in → 476 out (465 reasoning)
none
anthropic/claude-fable-5:high__json_schema FAIL 0% 351 in → 34 out (23 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:low__json_schema FAIL 0% 350 in → 25 out (12 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:max__json_schema FAIL 0% 349 in → 199 out (67 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:medium__json_schema FAIL 0% 349 in → 27 out (12 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:xhigh__json_schema FAIL 0% 351 in → 53 out (38 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:high__json_schema FAIL 0% 306 in → 381 out (398 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:low__json_schema FAIL 0% 308 in → 388 out (408 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:medium__json_schema FAIL 0% 305 in → 591 out (575 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 275 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:xhigh__json_schema FAIL 0% 306 in → 412 out (429 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 275 in → 107 out (96 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:low__json_schema FAIL 0% 275 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:max__json_schema FAIL 0% 276 in → 162 out (144 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:medium__json_schema FAIL 0% 276 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:none__json_schema FAIL 0% 276 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 275 in → 83 out (62 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:high__json_schema FAIL 0% 354 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:low__json_schema FAIL 0% 355 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:max__json_schema FAIL 0% 355 in → 235 out (46 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:medium__json_schema FAIL 0% 355 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:none__json_schema FAIL 0% 357 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:xhigh__json_schema FAIL 0% 353 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:high__json_schema FAIL 0% 349 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:low__json_schema FAIL 0% 353 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 350 in → 208 out (44 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:medium__json_schema FAIL 0% 353 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:none__json_schema FAIL 0% 350 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:xhigh__json_schema FAIL 0% 348 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:high__json_schema FAIL 0% 276 in → 152 out (132 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:low__json_schema FAIL 0% 277 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:max__json_schema FAIL 0% 276 in → 213 out (202 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:medium__json_schema FAIL 0% 276 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:none__json_schema FAIL 0% 276 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:xhigh__json_schema FAIL 0% 274 in → 140 out (122 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:high__json_schema FAIL 0% 349 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 350 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 350 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:none FAIL 0% 110 in → 224 out (0 reasoning)
I don't have access to the repository, so I can't inspect the actual commit history, branch data, or merge status. Based solely on the text you've shared, `feature-experimental` is **not** listed under "Merged into main" — only `main` itself is listed there. This suggests `feature-experimental` is likely *not* fully merged into `main`, so it should not be deleted based on this information. To give an accurate answer, please run and share the output of: ``` git branch --merged main ``` This will list all branches (excluding main itself) that are fully merged into main. Any branch shown there (other than main) is safe to delete. Based on what you've provided, that list appears empty (or shows only main), which means: **none**
Failure: Missing: ['none'] Extra: ['**none**', 'Based solely on the text you\'ve shared, `feature-experimental` is **not** listed under "Merged into main" — only `main` itself is listed there. This suggests `feature-experimental` is likely *not* fully merged into `main`, so it should not be deleted based on this information.', "I don't have access to the repository, so I can't inspect the actual commit history, branch data, or merge status.", "This will list all branches (excluding main itself) that are fully merged into main. Any branch shown there (other than main) is safe to delete. Based on what you've provided, that list appears empty (or shows only main), which means:", 'To give an accurate answer, please run and share the output of:', '```', 'git branch --merged main']
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 352 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 346 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 0% 78 in → 90 out (85 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-flash-0731:medium__json_schema FAIL 0% 78 in → 49 out (65 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": []}
Failure: Missing: ['none']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 78 in → 3 out (0 reasoning)
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 78 in → 13 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-experimental"]}
Failure: Missing: ['none'] Extra: ['feature-experimental']
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 80 in → 126 out (157 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-flash:high__json_schema FAIL 0% 184 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
deepseek/deepseek-v4-flash:low__json_schema FAIL 0% 79 in → 220 out (200 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-experimental" ] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
deepseek/deepseek-v4-flash:medium__json_schema FAIL 0% 80 in → 208 out (196 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 79 in → 16 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-experimental"] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0% 157 in → 247 out (261 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-pro:high__json_schema FAIL 0% 276 in → 287 out (277 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 77 in → 15 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-experimental"] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 277 in → 241 out (231 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
deepseek/deepseek-v4-pro:none FAIL 0% 81 in → 3 out (0 reasoning)
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 78 in → 19 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-experimental" ] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 357 in → 885 out (875 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3-flash-preview:high__json_schema FAIL 0% 155 in → 474 out (465 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3-flash-preview:low__json_schema FAIL 0% 157 in → 569 out (560 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3-flash-preview:medium__json_schema FAIL 0% 158 in → 726 out (718 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3-flash-preview:xhigh__json_schema FAIL 0% 159 in → 592 out (584 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:high__json_schema FAIL 0% 159 in → 580 out (567 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 0% 76 in → 129 out (116 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 76 in → 278 out (269 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:xhigh__json_schema FAIL 0% 79 in → 670 out (657 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:high__json_schema FAIL 0% 157 in → 515 out (507 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 158 in → 283 out (274 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:medium__json_schema FAIL 0% 157 in → 379 out (371 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:xhigh__json_schema FAIL 0% 157 in → 448 out (440 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:high__json_schema FAIL 0% 158 in → 677 out (669 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 77 in → 13 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:max__json_schema FAIL 0% 76 in → 448 out (440 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:medium__json_schema FAIL 0% 79 in → 493 out (485 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:xhigh__json_schema FAIL 0% 78 in → 490 out (482 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash:high__json_schema FAIL 0% 154 in → 352 out (344 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash:low__json_schema FAIL 0% 155 in → 158 out (145 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.5-flash:medium__json_schema FAIL 0% 155 in → 232 out (224 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash:xhigh__json_schema FAIL 0% 158 in → 526 out (518 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:high__json_schema FAIL 0% 78 in → 279 out (271 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:low__json_schema FAIL 0% 74 in → 149 out (141 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:max__json_schema FAIL 0% 157 in → 417 out (409 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:medium__json_schema FAIL 0% 156 in → 248 out (240 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:xhigh__json_schema FAIL 0% 159 in → 298 out (290 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemma-4-26b-a4b-it:high__json_schema FAIL 0% 92 in → 272 out (240 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemma-4-26b-a4b-it:none__json_schema FAIL 0% 90 in → 13 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemma-4-31b-it:low__json_schema FAIL 0% 95 in → 218 out (209 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
ibm-granite/granite-4.1-8b FAIL 0% 84 in → 69 out
feature-Experimental (Note: The branch name in the list appears as "feature-Experimental" with a hyphen, but in the description it is referred to as "feature-experimental" with a lowercase 'e'. Assuming the intended branch to be merged is the one listed as "feature-Experimental", that is the branch to delete.)
Failure: Missing: ['none'] Extra: ['(Note: The branch name in the list appears as "feature-Experimental" with a hyphen, but in the description it is referred to as "feature-experimental" with a lowercase \'e\'. Assuming the intended branch to be merged is the one listed as "feature-Experimental", that is the branch to delete.)', 'feature-Experimental']
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-experimental"] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
inclusionai/ling-2.6-flash FAIL 0% 99 in → 4 out
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-experimental" ] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
liquid/lfm-2-24b-a2b FAIL 0% 87 in → 5 out
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
Invalid JSON. Output: feature-experimental
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:high__json_schema FAIL 0% 109 in → 542 out (636 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.5:low__json_schema FAIL 0% 111 in → 634 out (627 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.5:medium FAIL 0% 114 in → 236 out (232 reasoning)
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
minimax/minimax-m2.5:medium__json_schema FAIL 0% 112 in → 409 out (446 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.7:high__json_schema FAIL 0% 111 in → 808 out (914 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
minimax/minimax-m2.7:low__json_schema FAIL 0% 262 in → 760 out (744 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.7:medium__json_schema FAIL 0% 260 in → 866 out (856 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
minimax/minimax-m2.7:xhigh__json_schema FAIL 0% 260 in → 382 out (370 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m3:high__json_schema FAIL 0% 235 in → 15 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-experimental"] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
minimax/minimax-m3:low__json_schema FAIL 0% 236 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m3:medium__json_schema FAIL 0% 237 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m3:xhigh__json_schema FAIL 0% 236 in → 15 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-experimental"] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
feature-experimental
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-experimental"]}
Failure: Missing: ['none'] Extra: ['feature-experimental']
mistralai/mistral-medium-3-5:high__json_schema FAIL 0% 82 in → 212 out (229 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:low__json_schema FAIL 0% 82 in → 549 out (621 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:medium__json_schema FAIL 0% 83 in → 129 out (125 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:none__json_schema FAIL 0% 81 in → 13 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-experimental"]}
Failure: Missing: ['none'] Extra: ['feature-experimental']
mistralai/mistral-medium-3-5:xhigh__json_schema FAIL 0% 80 in → 484 out (554 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
moonshotai/kimi-k2.6:high__json_schema FAIL 0% 77 in → 295 out (241 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:low__json_schema FAIL 0% 79 in → 159 out (172 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
moonshotai/kimi-k2.7-code:medium__json_schema FAIL 0% 174 in → 135 out (125 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
moonshotai/kimi-k2.7-code:xhigh__json_schema FAIL 0% 171 in → 147 out (137 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
(empty output)
Failure: OpenAI-compatible response contained provider error: Provider returned error
moonshotai/kimi-k3:xhigh__json_schema FAIL 0% 265 in → 136 out (114 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
nvidia/nemotron-3-nano-30b-a3b:high FAIL 0% 93 in → 192 out (212 reasoning)
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 94 in → 18 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-experimental" ] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
nvidia/nemotron-3-super-120b-a12b:high__json_schema FAIL 0% 96 in → 168 out (147 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:low__json_schema FAIL 0% 94 in → 119 out (102 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:medium__json_schema FAIL 0% 95 in → 96 out (83 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:none__json_schema FAIL 0% 93 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema FAIL 0% 95 in → 129 out (112 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
openai/gpt-5.4-mini:high__json_schema FAIL 0% 119 in → 336 out (320 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-mini:low__json_schema FAIL 0% 119 in → 79 out (63 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-mini:none__json_schema FAIL 0% 119 in → 24 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-experimental"]}
Failure: Missing: ['none'] Extra: ['feature-experimental']
openai/gpt-5.4-mini:xhigh__json_schema FAIL 0% 119 in → 1,261 out (1,245 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-nano:high__json_schema FAIL 0% 120 in → 132 out (116 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-nano:low__json_schema FAIL 0% 117 in → 30 out (14 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-nano:xhigh__json_schema FAIL 0% 120 in → 165 out (149 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:high__json_schema FAIL 0% 120 in → 96 out (80 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:medium__json_schema FAIL 0% 120 in → 80 out (64 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:none__json_schema FAIL 0% 122 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:high__json_schema FAIL 0% 121 in → 170 out (154 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:low__json_schema FAIL 0% 122 in → 40 out (24 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:medium__json_schema FAIL 0% 120 in → 73 out (57 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:none__json_schema FAIL 0% 118 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:xhigh__json_schema FAIL 0% 122 in → 628 out (612 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-luna:medium__json_schema FAIL 0% 118 in → 105 out (89 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-luna:none__json_schema FAIL 0% 120 in → 18 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-experimental"]}
Failure: Missing: ['none'] Extra: ['feature-experimental']
openai/gpt-5.6-sol:high__json_schema FAIL 0% 119 in → 93 out (77 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:low__json_schema FAIL 0% 122 in → 105 out (89 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:max__json_schema FAIL 0% 121 in → 136 out (120 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:medium__json_schema FAIL 0% 119 in → 55 out (39 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:none__json_schema FAIL 0% 119 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:xhigh__json_schema FAIL 0% 119 in → 68 out (52 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:high__json_schema FAIL 0% 121 in → 41 out (25 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:low__json_schema FAIL 0% 121 in → 36 out (20 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:max__json_schema FAIL 0% 120 in → 51 out (35 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:medium__json_schema FAIL 0% 119 in → 45 out (29 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:none__json_schema FAIL 0% 121 in → 18 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-experimental"]}
Failure: Missing: ['none'] Extra: ['feature-experimental']
openai/gpt-5.6-terra:xhigh__json_schema FAIL 0% 118 in → 55 out (39 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-oss-120b:high__json_schema FAIL 0% 138 in → 340 out (320 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
openai/gpt-oss-120b:low__json_schema FAIL 0% 138 in → 12 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-experimental"]}
Failure: Missing: ['none'] Extra: ['feature-experimental']
Invalid JSON. Output: {"branches_to_delete":[]} # Done. The assistant responded with the correct format and content. The user request to list branch names to delete was satisfied. Since none of the branches are merged into main, the array is empty, which aligns with the developer instruction to return a JSON object. The assistant did not provide any extraneous text or explanation. The solution is succinct and correct. The system instructions were honored by providing the JSON format as required. This is the final answer. The conversation is closed. # Done. The assistant should stop now. (end) This is the final JSON object as required. No additional comments or formatting. (end) (end). The assistant has now delivered the final. (final) (final) (end). The assistant has delivered final.{"branches_to_delete":[]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 6 column 1 (char 32)
Failure: Failed to parse structured JSON response: Extra data: line 6 column 1 (char 32)
openai/gpt-oss-20b:low__json_schema FAIL 0% 139 in → 29 out (11 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
openai/gpt-oss-20b:medium__json_schema FAIL 0% 141 in → 209 out (192 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
openai/gpt-oss-20b:xhigh__json_schema FAIL 0% 126 in → 398 out (454 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
poolside/laguna-m.1:low__json_schema FAIL 0% 97 in → 522 out (513 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-m.1:none FAIL 0% 99 in → 5 out (0 reasoning)
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
poolside/laguna-m.1:none__json_schema FAIL 0% 99 in → 11 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-experimental"]}
Failure: Missing: ['none'] Extra: ['feature-experimental']
poolside/laguna-m.1:xhigh FAIL 0% 98 in → 320 out (314 reasoning)
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
poolside/laguna-m.1:xhigh__json_schema FAIL 0% 96 in → 238 out (225 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
poolside/laguna-xs-2.1:high__json_schema FAIL 0% 98 in → 240 out (231 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs-2.1:low__json_schema FAIL 0% 98 in → 184 out (175 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs-2.1:medium__json_schema FAIL 0% 99 in → 191 out (178 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
poolside/laguna-xs-2.1:none FAIL 0% 97 in → 3 out (0 reasoning)
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 97 in → 16 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-experimental"] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
poolside/laguna-xs.2:high__json_schema FAIL 0% 134 in → 318 out (309 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs.2:low__json_schema FAIL 0% 134 in → 275 out (266 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs.2:medium__json_schema FAIL 0% 135 in → 128 out (119 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs.2:none__json_schema FAIL 0% 135 in → 11 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-experimental"]}
Failure: Missing: ['none'] Extra: ['feature-experimental']
poolside/laguna-xs.2:xhigh__json_schema FAIL 0% 136 in → 318 out (309 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
qwen/qwen3.6-27b:high__json_schema FAIL 0% 89 in → 797 out (766 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-27b:low__json_schema FAIL 0% 87 in → 850 out (1 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-27b:medium__json_schema FAIL 0% 91 in → 711 out (1 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 89 in → 23 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-experimental" ] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
qwen/qwen3.6-27b:xhigh__json_schema FAIL 0% 90 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete":[] }
Failure: Missing: ['none']
qwen/qwen3.6-35b-a3b:low__json_schema FAIL 0% 88 in → 712 out (694 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-35b-a3b:medium__json_schema FAIL 0% 88 in → 1,028 out (1,018 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 91 in → 22 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-experimental" ] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
qwen/qwen3.6-35b-a3b:xhigh__json_schema FAIL 0% 88 in → 892 out (892 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d4eb4073-95d2-936a-8c49-f0f16b149c05","request_id":"d4eb4073-95d2-936a-8c49-f0f16b149c05"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e74ab947-bcd1-949a-af70-495f7b2fe3e5","request_id":"e74ab947-bcd1-949a-af70-495f7b2fe3e5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-99593edb-ea78-92ab-b2b7-3376733f5fe2","request_id":"99593edb-ea78-92ab-b2b7-3376733f5fe2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none FAIL 0% 88 in → 3 out (0 reasoning)
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-17f8ece5-8298-9d39-be04-3a0f8a0191a5","request_id":"17f8ece5-8298-9d39-be04-3a0f8a0191a5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-39039b73-1911-99e6-8701-50e1526cfdd9","request_id":"39039b73-1911-99e6-8701-50e1526cfdd9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ede1d58d-50f1-916a-bafc-25b2f9a19095"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2d83bc13-b345-926d-943e-53c9de653512"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e78ddf11-796e-92f9-9095-285406fb1724"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c62a16db-98ce-9ecb-9a99-b0e157cc3097"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none FAIL 0% 92 in → 3 out (0 reasoning)
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3dd8af56-16ac-9797-ac1b-d6aafeaf78da"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f36dd114-c70c-9223-b6a1-c05e5e14ac25"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b371981d-437f-9c86-abbb-6a878bb1aabb","request_id":"b371981d-437f-9c86-abbb-6a878bb1aabb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6b161ada-3d81-939e-b43c-436c6f080162","request_id":"6b161ada-3d81-939e-b43c-436c6f080162"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-584e9af7-108b-96c6-aa23-702002d938fa","request_id":"584e9af7-108b-96c6-aa23-702002d938fa"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-12be7faa-73fe-9ef6-8f1a-242eab148edd","request_id":"12be7faa-73fe-9ef6-8f1a-242eab148edd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-19c9a754-10eb-9cdf-a9df-e1997294dc33","request_id":"19c9a754-10eb-9cdf-a9df-e1997294dc33"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fb761a22-f9f6-934b-81d4-979635c74b23","request_id":"fb761a22-f9f6-934b-81d4-979635c74b23"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f2fa589b-875e-97ca-8c01-3b0bc7ed37c2","request_id":"f2fa589b-875e-97ca-8c01-3b0bc7ed37c2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-db6296a3-147c-9693-ac3a-35c40213941f","request_id":"db6296a3-147c-9693-ac3a-35c40213941f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-88cb0beb-6e29-957f-958e-fee7d7de06cd","request_id":"88cb0beb-6e29-957f-958e-fee7d7de06cd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-594c4878-113b-9255-9b2b-afc8c2a242ad","request_id":"594c4878-113b-9255-9b2b-afc8c2a242ad"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:low__json_schema FAIL 0% 86 in → 36 out (22 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
thinkingmachines/inkling-small:max__json_schema FAIL 0% 85 in → 229 out (216 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
thinkingmachines/inkling-small:medium__json_schema FAIL 0% 85 in → 118 out (104 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output: none{ "branches_to_delete": [] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0% 86 in → 38 out (16 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-experimental" ] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
thinkingmachines/inkling:max__json_schema FAIL 0% 87 in → 405 out (393 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
thinkingmachines/inkling:medium__json_schema FAIL 0% 86 in → 337 out (323 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
thinkingmachines/inkling:none FAIL 0% 86 in → 4 out (0 reasoning)
none<|end_message|>
Failure: Missing: ['none'] Extra: ['none<|end_message|>']
Invalid JSON. Output: none{"branches_to_delete": []}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:high__json_schema FAIL 0% 319 in → 626 out (620 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:low__json_schema FAIL 0% 326 in → 358 out (349 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
x-ai/grok-4.3:max__json_schema FAIL 0% 320 in → 483 out (477 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:medium__json_schema FAIL 0% 327 in → 366 out (357 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
x-ai/grok-4.3:none__json_schema FAIL 0% 318 in → 6 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:xhigh__json_schema FAIL 0% 321 in → 544 out (538 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:high__json_schema FAIL 0% 360 in → 436 out (428 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:low__json_schema FAIL 0% 359 in → 214 out (206 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:max__json_schema FAIL 0% 359 in → 305 out (297 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:medium__json_schema FAIL 0% 361 in → 285 out (277 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:xhigh__json_schema FAIL 0% 363 in → 375 out (367 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
z-ai/glm-4.7-flash:high__json_schema FAIL 0% 78 in → 573 out (560 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
z-ai/glm-4.7-flash:low__json_schema FAIL 0% 78 in → 632 out (629 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 81 in → 3 out (0 reasoning)
feature-experimental
Failure: Missing: ['none'] Extra: ['feature-experimental']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 78 in → 10 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output: none
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:high__json_schema FAIL 0% 268 in → 349 out (338 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
z-ai/glm-5.2:low__json_schema FAIL 0% 88 in → 206 out (196 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
z-ai/glm-5.2:none__json_schema FAIL 0% 81 in → 18 out (0 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-experimental" ] }
Failure: Missing: ['none'] Extra: ['feature-experimental']
z-ai/glm-5.2:xhigh__json_schema FAIL 0% 87 in → 649 out (681 reasoning)
feature-experimental
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-experimental"]}
Failure: Missing: ['none'] Extra: ['feature-experimental']