Only main branch
Tests ability to handle a repository with only the main branch (trivial case). Evaluates the minimum-viable-repo edge case.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Hello' > hello.txt
  5. 05 git add hello.txt
  6. 06 git commit -m 'Initial commit'
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
none

Scoped model quality, cost, API time, and token usage for branch_cleanup/f011.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high FAIL 100% 93 in → 71 out (21 reasoning)
none The repository only contains the `main` branch itself. There are no other branches merged into main that could be deleted (and you should never delete the current/default branch).
Failure: Extra: ['The repository only contains the `main` branch itself. There are no other branches merged into main that could be deleted (and you should never delete the current/default branch).']
anthropic/claude-fable-5:low FAIL 100% 92 in → 73 out (17 reasoning)
none The only branch in the repository is `main` itself, which cannot be deleted (it's the current/default branch). There are no other branches merged into main that could be cleaned up.
Failure: Extra: ["The only branch in the repository is `main` itself, which cannot be deleted (it's the current/default branch). There are no other branches merged into main that could be cleaned up."]
anthropic/claude-fable-5:max PASS 100% 92 in → 177 out (42 reasoning)
none
anthropic/claude-fable-5:medium FAIL 100% 94 in → 80 out (17 reasoning)
none The only branch in the repository is `main` itself. There are no other branches, merged or otherwise, so there is nothing to delete (and you should never delete `main`, the current branch).
Failure: Extra: ['The only branch in the repository is `main` itself. There are no other branches, merged or otherwise, so there is nothing to delete (and you should never delete `main`, the current branch).']
anthropic/claude-fable-5:xhigh FAIL 100% 94 in → 96 out (42 reasoning)
none The only branch in the repository is `main` itself, which is the current branch and should never be deleted. There are no other branches that have been merged and could be safely removed.
Failure: Extra: ['The only branch in the repository is `main` itself, which is the current branch and should never be deleted. There are no other branches that have been merged and could be safely removed.']
anthropic/claude-haiku-4.5:high PASS 100% 100 in → 182 out (195 reasoning)
none
anthropic/claude-haiku-4.5:low PASS 100% 102 in → 134 out (136 reasoning)
none
anthropic/claude-haiku-4.5:medium PASS 100% 102 in → 137 out (142 reasoning)
none
anthropic/claude-haiku-4.5:none PASS 100% 73 in → 4 out (0 reasoning)
none
anthropic/claude-haiku-4.5:xhigh PASS 100% 103 in → 116 out (116 reasoning)
none
anthropic/claude-opus-4.6:high PASS 100% 73 in → 99 out (89 reasoning)
none
anthropic/claude-opus-4.6:low PASS 100% 72 in → 4 out (0 reasoning)
none
anthropic/claude-opus-4.6:max PASS 100% 73 in → 109 out (103 reasoning)
none
anthropic/claude-opus-4.6:medium PASS 100% 75 in → 46 out (28 reasoning)
none
anthropic/claude-opus-4.6:none FAIL 100% 73 in → 38 out (0 reasoning)
Looking at the repository, the only branch that exists is `main`, and it is merged into itself (as expected). There are no other branches to evaluate. none
Failure: Extra: ['Looking at the repository, the only branch that exists is `main`, and it is merged into itself (as expected).', 'There are no other branches to evaluate.']
anthropic/claude-opus-4.6:xhigh PASS 100% 74 in → 100 out (89 reasoning)
none
anthropic/claude-opus-4.7:high PASS 100% 99 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:low PASS 100% 98 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:max PASS 100% 98 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:medium PASS 100% 98 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:none PASS 100% 96 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:xhigh PASS 100% 98 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.8:high FAIL 100% 94 in → 162 out (36 reasoning)
none The only branch in this repository is `main` itself, which is the primary branch and should never be deleted. There are no other feature or topic branches that have been merged into main, so there is nothing to clean up.
Failure: Extra: ['The only branch in this repository is `main` itself, which is the primary branch and should never be deleted. There are no other feature or topic branches that have been merged into main, so there is nothing to clean up.']
anthropic/claude-opus-4.8:low PASS 100% 92 in → 21 out (19 reasoning)
none
anthropic/claude-opus-4.8:max PASS 100% 93 in → 268 out (54 reasoning)
none
anthropic/claude-opus-4.8:medium PASS 100% 93 in → 20 out (19 reasoning)
none
anthropic/claude-opus-4.8:none PASS 100% 93 in → 3 out (0 reasoning)
none
anthropic/claude-opus-4.8:xhigh FAIL 100% 93 in → 205 out (35 reasoning)
none There are no branches to delete. The only branch in the repository is `main` itself, and there are no other merged branches that could be safely removed.
Failure: Extra: ['There are no branches to delete. The only branch in the repository is `main` itself, and there are no other merged branches that could be safely removed.']
anthropic/claude-sonnet-4.6:high PASS 100% 72 in → 111 out (105 reasoning)
none
anthropic/claude-sonnet-4.6:low PASS 100% 72 in → 4 out (0 reasoning)
none
anthropic/claude-sonnet-4.6:max PASS 100% 73 in → 99 out (92 reasoning)
none
anthropic/claude-sonnet-4.6:medium PASS 100% 73 in → 4 out (0 reasoning)
none
anthropic/claude-sonnet-4.6:none PASS 100% 74 in → 4 out (0 reasoning)
none
anthropic/claude-sonnet-4.6:xhigh PASS 100% 73 in → 104 out (95 reasoning)
none
anthropic/claude-sonnet-5:high FAIL 100% 92 in → 99 out (0 reasoning)
Based on the information provided, here are my findings: **Branches merged into main:** - main (this is main itself, not a candidate for deletion) **Analysis:** No other branches were listed besides `main`. Since there are no additional branches to evaluate for deletion, there is nothing to delete. none
Failure: Extra: ['**Analysis:**', '**Branches merged into main:**', '- main (this is main itself, not a candidate for deletion)', 'Based on the information provided, here are my findings:', 'No other branches were listed besides `main`. Since there are no additional branches to evaluate for deletion, there is nothing to delete.']
anthropic/claude-sonnet-5:low PASS 100% 93 in → 3 out (0 reasoning)
none
anthropic/claude-sonnet-5:medium PASS 100% 91 in → 3 out (0 reasoning)
none
anthropic/claude-sonnet-5:none PASS 100% 93 in → 3 out (0 reasoning)
none
anthropic/claude-sonnet-5:xhigh PASS 100% 93 in → 90 out (37 reasoning)
none
arcee-ai/trinity-large-thinking:high PASS 100% 70 in → 1,181 out (1,204 reasoning)
none
arcee-ai/trinity-large-thinking:low PASS 100% 70 in → 1,729 out (1,777 reasoning)
none
arcee-ai/trinity-large-thinking:medium PASS 100% 69 in → 1,472 out (1,506 reasoning)
none
arcee-ai/trinity-large-thinking:xhigh PASS 100% 71 in → 1,398 out (1,423 reasoning)
none
arcee-ai/trinity-mini:high PASS 100% 68 in → 810 out (942 reasoning)
none
arcee-ai/trinity-mini:high__json_schema PASS 100% 70 in → 209 out (231 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete":[ "none" ] }
arcee-ai/trinity-mini:low PASS 100% 70 in → 328 out (388 reasoning)
none
arcee-ai/trinity-mini:low__json_schema PASS 100% 70 in → 367 out (410 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
arcee-ai/trinity-mini:medium PASS 100% 70 in → 320 out (380 reasoning)
none
arcee-ai/trinity-mini:medium__json_schema PASS 100% 70 in → 296 out (330 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 71 in → 242 out (277 reasoning)
none
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 69 in → 365 out (419 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 147 in → 74 out (72 reasoning)
none
deepseek/deepseek-v4-flash-0731:low PASS 100% 74 in → 68 out (79 reasoning)
none
deepseek/deepseek-v4-flash-0731:max PASS 100% 147 in → 71 out (75 reasoning)
none
deepseek/deepseek-v4-flash-0731:medium PASS 100% 73 in → 30 out (32 reasoning)
none
deepseek/deepseek-v4-flash-0731:none PASS 100% 68 in → 1 out (0 reasoning)
none
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 67 in → 177 out (179 reasoning)
none
deepseek/deepseek-v4-flash:high PASS 100% 68 in → 65 out (62 reasoning)
none
deepseek/deepseek-v4-flash:low PASS 100% 67 in → 105 out (104 reasoning)
none
deepseek/deepseek-v4-flash:medium PASS 100% 69 in → 40 out (47 reasoning)
none
deepseek/deepseek-v4-flash:none PASS 100% 68 in → 2 out (0 reasoning)
none
deepseek/deepseek-v4-flash:xhigh PASS 100% 70 in → 56 out (64 reasoning)
none
deepseek/deepseek-v4-pro:high PASS 100% 69 in → 120 out (117 reasoning)
none
deepseek/deepseek-v4-pro:low PASS 100% 69 in → 116 out (113 reasoning)
none
deepseek/deepseek-v4-pro:medium PASS 100% 68 in → 121 out (118 reasoning)
none
deepseek/deepseek-v4-pro:none FAIL 100% 67 in → 35 out (0 reasoning)
Based on the information provided: * **main** is the only branch that exists. * There are no other branches listed that could have been merged into it. none
Failure: Extra: ['* **main** is the only branch that exists.', '* There are no other branches listed that could have been merged into it.', 'Based on the information provided:']
deepseek/deepseek-v4-pro:xhigh PASS 100% 67 in → 99 out (109 reasoning)
none
google/gemini-3-flash-preview:high PASS 100% 65 in → 192 out (191 reasoning)
none
google/gemini-3-flash-preview:low PASS 100% 64 in → 235 out (234 reasoning)
none
google/gemini-3-flash-preview:medium PASS 100% 65 in → 183 out (182 reasoning)
none
google/gemini-3-flash-preview:xhigh PASS 100% 65 in → 182 out (181 reasoning)
none
google/gemini-3.1-flash-lite-preview:high PASS 100% 63 in → 301 out (300 reasoning)
none
google/gemini-3.1-flash-lite-preview:low PASS 100% 66 in → 122 out (121 reasoning)
none
google/gemini-3.1-flash-lite-preview:medium PASS 100% 64 in → 230 out (229 reasoning)
none
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 65 in → 232 out (231 reasoning)
none
google/gemini-3.1-pro-preview:high PASS 100% 65 in → 103 out (102 reasoning)
none
google/gemini-3.1-pro-preview:low PASS 100% 64 in → 169 out (168 reasoning)
none
google/gemini-3.1-pro-preview:medium PASS 100% 63 in → 231 out (230 reasoning)
none
google/gemini-3.1-pro-preview:xhigh PASS 100% 65 in → 264 out (263 reasoning)
none
google/gemini-3.5-flash-lite:high PASS 100% 64 in → 259 out (258 reasoning)
none
google/gemini-3.5-flash-lite:low PASS 100% 65 in → 1 out (0 reasoning)
none
google/gemini-3.5-flash-lite:max PASS 100% 65 in → 222 out (221 reasoning)
none
google/gemini-3.5-flash-lite:medium PASS 100% 66 in → 219 out (218 reasoning)
none
google/gemini-3.5-flash-lite:xhigh PASS 100% 66 in → 271 out (270 reasoning)
none
google/gemini-3.5-flash:high PASS 100% 64 in → 271 out (270 reasoning)
none
google/gemini-3.5-flash:low PASS 100% 65 in → 64 out (63 reasoning)
none
google/gemini-3.5-flash:medium PASS 100% 64 in → 238 out (237 reasoning)
none
google/gemini-3.5-flash:xhigh PASS 100% 65 in → 295 out (294 reasoning)
none
google/gemini-3.6-flash:high PASS 100% 66 in → 205 out (204 reasoning)
none
google/gemini-3.6-flash:low PASS 100% 65 in → 103 out (102 reasoning)
none
google/gemini-3.6-flash:max PASS 100% 66 in → 254 out (253 reasoning)
none
google/gemini-3.6-flash:medium PASS 100% 64 in → 200 out (199 reasoning)
none
google/gemini-3.6-flash:xhigh PASS 100% 64 in → 183 out (182 reasoning)
none
google/gemma-4-26b-a4b-it:high PASS 100% 80 in → 312 out (310 reasoning)
none
google/gemma-4-26b-a4b-it:low PASS 100% 79 in → 230 out (213 reasoning)
none
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 80 in → 249 out (216 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 79 in → 286 out (284 reasoning)
none
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 80 in → 255 out (219 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 77 in → 2 out (0 reasoning)
none
google/gemma-4-26b-a4b-it:xhigh PASS 100% 79 in → 228 out (206 reasoning)
none
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 80 in → 251 out (221 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-31b-it:high PASS 100% 78 in → 179 out (170 reasoning)
none
google/gemma-4-31b-it:high__json_schema PASS 100% 79 in → 206 out (175 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-31b-it:low PASS 100% 80 in → 166 out (153 reasoning)
none
google/gemma-4-31b-it:low__json_schema PASS 100% 80 in → 207 out (1 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-31b-it:medium PASS 100% 80 in → 200 out (185 reasoning)
none
google/gemma-4-31b-it:medium__json_schema PASS 100% 80 in → 207 out (171 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-31b-it:none PASS 100% 77 in → 2 out (0 reasoning)
none
google/gemma-4-31b-it:none__json_schema PASS 100% 77 in → 11 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
google/gemma-4-31b-it:xhigh PASS 100% 80 in → 131 out (1 reasoning)
none
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 81 in → 195 out (169 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
ibm-granite/granite-4.1-8b PASS 100% 76 in → 2 out
none
inclusionai/ling-2.6-flash PASS 100% 85 in → 2 out
none
minimax/minimax-m2.5:high PASS 100% 100 in → 171 out (200 reasoning)
none
minimax/minimax-m2.5:high__json_schema PASS 100% 99 in → 131 out (141 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
minimax/minimax-m2.5:low PASS 100% 98 in → 327 out (390 reasoning)
none
minimax/minimax-m2.5:medium PASS 100% 100 in → 182 out (209 reasoning)
none
minimax/minimax-m2.5:xhigh PASS 100% 102 in → 182 out (180 reasoning)
none
minimax/minimax-m2.7:high PASS 100% 101 in → 99 out (97 reasoning)
none
minimax/minimax-m2.7:low PASS 100% 101 in → 122 out (120 reasoning)
none
minimax/minimax-m2.7:medium PASS 100% 100 in → 155 out (167 reasoning)
none
minimax/minimax-m2.7:xhigh PASS 100% 98 in → 131 out (152 reasoning)
none
minimax/minimax-m3:high PASS 100% 238 in → 95 out (115 reasoning)
none
minimax/minimax-m3:low PASS 100% 237 in → 88 out (102 reasoning)
none
minimax/minimax-m3:medium PASS 100% 238 in → 104 out (125 reasoning)
none
minimax/minimax-m3:xhigh PASS 100% 237 in → 88 out (107 reasoning)
none
minimax/minimax-m3:xhigh__json_schema PASS 100% 224 in → 11 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
mistralai/devstral-2512 PASS 100% 66 in → 2 out
none
mistralai/mistral-medium-3-5:high PASS 100% 81 in → 118 out (130 reasoning)
none
mistralai/mistral-medium-3-5:low PASS 100% 80 in → 167 out (196 reasoning)
none
mistralai/mistral-medium-3-5:medium PASS 100% 80 in → 177 out (221 reasoning)
none
mistralai/mistral-medium-3-5:none PASS 100% 81 in → 2 out (0 reasoning)
none
mistralai/mistral-medium-3-5:xhigh PASS 100% 81 in → 594 out (691 reasoning)
none
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 66 in → 211 out (242 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
moonshotai/kimi-k2.6:high PASS 100% 69 in → 251 out (248 reasoning)
none
moonshotai/kimi-k2.6:low PASS 100% 67 in → 338 out (335 reasoning)
none
moonshotai/kimi-k2.6:low__json_schema PASS 100% 69 in → 191 out (222 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
moonshotai/kimi-k2.6:medium PASS 100% 68 in → 206 out (203 reasoning)
none
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 66 in → 130 out (116 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
moonshotai/kimi-k2.6:xhigh PASS 100% 67 in → 221 out (218 reasoning)
none
moonshotai/kimi-k2.7-code:high PASS 100% 68 in → 103 out (116 reasoning)
none
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 67 in → 112 out (109 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
moonshotai/kimi-k2.7-code:low PASS 100% 69 in → 92 out (89 reasoning)
none
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 67 in → 89 out (88 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
moonshotai/kimi-k2.7-code:medium PASS 100% 69 in → 128 out (125 reasoning)
none
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 68 in → 70 out (56 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 68 in → 163 out (181 reasoning)
none
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 68 in → 136 out (128 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
moonshotai/kimi-k3:high PASS 100% 145 in → 102 out (86 reasoning)
none
moonshotai/kimi-k3:high__json_schema PASS 100% 144 in → 145 out (129 reasoning)
none
moonshotai/kimi-k3:low PASS 100% 144 in → 113 out (97 reasoning)
none
moonshotai/kimi-k3:low__json_schema PASS 100% 146 in → 95 out (79 reasoning)
none
moonshotai/kimi-k3:max PASS 100% 144 in → 118 out (102 reasoning)
none
moonshotai/kimi-k3:max__json_schema PASS 100% 146 in → 143 out (127 reasoning)
none
moonshotai/kimi-k3:medium PASS 100% 145 in → 127 out (111 reasoning)
none
moonshotai/kimi-k3:medium__json_schema PASS 100% 147 in → 116 out (100 reasoning)
none
moonshotai/kimi-k3:xhigh PASS 100% 145 in → 74 out (58 reasoning)
none
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 147 in → 103 out (87 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 80 in → 142 out (152 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 82 in → 268 out (280 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 80 in → 199 out (215 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 78 in → 165 out (178 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 82 in → 85 out (94 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 80 in → 351 out (349 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 80 in → 2 out (0 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 81 in → 296 out (329 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 81 in → 40 out (36 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 81 in → 111 out (104 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 81 in → 231 out (221 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 82 in → 2 out (0 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 82 in → 54 out (48 reasoning)
none
openai/gpt-5.4-mini:high PASS 100% 68 in → 68 out (61 reasoning)
none
openai/gpt-5.4-mini:low PASS 100% 67 in → 28 out (21 reasoning)
none
openai/gpt-5.4-mini:low__json_schema PASS 100% 108 in → 108 out (90 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.4-mini:medium PASS 100% 66 in → 47 out (40 reasoning)
none
openai/gpt-5.4-mini:none PASS 100% 67 in → 5 out (0 reasoning)
none
openai/gpt-5.4-mini:xhigh PASS 100% 65 in → 67 out (60 reasoning)
none
openai/gpt-5.4-nano:high PASS 100% 66 in → 34 out (27 reasoning)
none
openai/gpt-5.4-nano:low PASS 100% 66 in → 22 out (15 reasoning)
none
openai/gpt-5.4-nano:low__json_schema PASS 100% 108 in → 34 out (16 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.4-nano:medium PASS 100% 67 in → 29 out (22 reasoning)
none
openai/gpt-5.4-nano:none PASS 100% 66 in → 5 out (0 reasoning)
none
openai/gpt-5.4-nano:none__json_schema PASS 100% 110 in → 16 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.4-nano:xhigh PASS 100% 67 in → 115 out (108 reasoning)
none
openai/gpt-5.4:high PASS 100% 66 in → 109 out (102 reasoning)
none
openai/gpt-5.4:high__json_schema PASS 100% 109 in → 89 out (71 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.4:low PASS 100% 66 in → 48 out (41 reasoning)
none
openai/gpt-5.4:medium PASS 100% 66 in → 59 out (52 reasoning)
none
openai/gpt-5.4:none PASS 100% 65 in → 5 out (0 reasoning)
none
openai/gpt-5.4:xhigh PASS 100% 67 in → 110 out (103 reasoning)
none
openai/gpt-5.5:high PASS 100% 66 in → 45 out (38 reasoning)
none
openai/gpt-5.5:low PASS 100% 65 in → 5 out (0 reasoning)
none
openai/gpt-5.5:medium PASS 100% 66 in → 44 out (37 reasoning)
none
openai/gpt-5.5:none PASS 100% 66 in → 5 out (0 reasoning)
none
openai/gpt-5.5:xhigh PASS 100% 64 in → 50 out (43 reasoning)
none
openai/gpt-5.6-luna:high PASS 100% 66 in → 36 out (29 reasoning)
none
openai/gpt-5.6-luna:low PASS 100% 66 in → 16 out (9 reasoning)
none
openai/gpt-5.6-luna:max PASS 100% 67 in → 47 out (40 reasoning)
none
openai/gpt-5.6-luna:medium PASS 100% 64 in → 30 out (23 reasoning)
none
openai/gpt-5.6-luna:none PASS 100% 68 in → 5 out (0 reasoning)
none
openai/gpt-5.6-luna:xhigh PASS 100% 67 in → 44 out (37 reasoning)
none
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 109 in → 55 out (37 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.6-sol:high PASS 100% 65 in → 5 out (0 reasoning)
none
openai/gpt-5.6-sol:low PASS 100% 65 in → 5 out (0 reasoning)
none
openai/gpt-5.6-sol:max PASS 100% 65 in → 46 out (39 reasoning)
none
openai/gpt-5.6-sol:medium PASS 100% 66 in → 5 out (0 reasoning)
none
openai/gpt-5.6-sol:none PASS 100% 64 in → 5 out (0 reasoning)
none
openai/gpt-5.6-sol:xhigh PASS 100% 64 in → 5 out (0 reasoning)
none
openai/gpt-5.6-terra:high PASS 100% 67 in → 20 out (13 reasoning)
none
openai/gpt-5.6-terra:low PASS 100% 66 in → 30 out (23 reasoning)
none
openai/gpt-5.6-terra:max PASS 100% 66 in → 54 out (47 reasoning)
none
openai/gpt-5.6-terra:medium PASS 100% 65 in → 24 out (17 reasoning)
none
openai/gpt-5.6-terra:none PASS 100% 64 in → 5 out (0 reasoning)
none
openai/gpt-5.6-terra:xhigh PASS 100% 66 in → 39 out (32 reasoning)
none
openai/gpt-oss-120b:high PASS 100% 126 in → 179 out (197 reasoning)
none
openai/gpt-oss-120b:high__json_schema PASS 100% 124 in → 177 out (179 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
openai/gpt-oss-120b:low PASS 100% 127 in → 96 out (86 reasoning)
none
openai/gpt-oss-120b:low__json_schema PASS 100% 129 in → 109 out (101 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
openai/gpt-oss-120b:medium PASS 100% 113 in → 104 out (109 reasoning)
none
openai/gpt-oss-120b:medium__json_schema PASS 100% 127 in → 117 out (94 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
openai/gpt-oss-120b:xhigh PASS 100% 125 in → 166 out (188 reasoning)
none
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 128 in → 152 out (157 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
openai/gpt-oss-20b:high PASS 100% 115 in → 258 out (291 reasoning)
none
openai/gpt-oss-20b:high__json_schema PASS 100% 126 in → 9 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-oss-20b:low PASS 100% 130 in → 15 out (5 reasoning)
none
openai/gpt-oss-20b:medium PASS 100% 133 in → 100 out (102 reasoning)
none
openai/gpt-oss-20b:xhigh PASS 100% 128 in → 280 out (310 reasoning)
none
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 126 in → 121 out (107 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
poolside/laguna-m.1:high PASS 100% 79 in → 304 out (300 reasoning)
none
poolside/laguna-m.1:high__json_schema PASS 100% 77 in → 169 out (159 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-m.1:low PASS 100% 77 in → 322 out (318 reasoning)
none
poolside/laguna-m.1:low__json_schema PASS 100% 78 in → 334 out (324 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-m.1:medium PASS 100% 78 in → 280 out (276 reasoning)
none
poolside/laguna-m.1:none FAIL 100% 79 in → 49 out (0 reasoning)
Based on the information provided, there is only one branch listed (main), and it is already the main branch. There are no other branches shown that have been merged into main. Therefore, no branches need to be deleted. none
Failure: Extra: ['Based on the information provided, there is only one branch listed (main), and it is already the main branch. There are no other branches shown that have been merged into main. Therefore, no branches need to be deleted.']
poolside/laguna-m.1:none__json_schema PASS 100% 78 in → 13 out (0 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
poolside/laguna-m.1:xhigh PASS 100% 76 in → 175 out (171 reasoning)
none
poolside/laguna-m.1:xhigh__json_schema PASS 100% 78 in → 232 out (222 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs-2.1:high PASS 100% 78 in → 214 out (212 reasoning)
none
poolside/laguna-xs-2.1:high__json_schema PASS 100% 77 in → 10 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs-2.1:low PASS 100% 78 in → 258 out (256 reasoning)
none
poolside/laguna-xs-2.1:low__json_schema PASS 100% 79 in → 167 out (157 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs-2.1:medium PASS 100% 79 in → 156 out (154 reasoning)
none
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 78 in → 182 out (172 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs-2.1:none PASS 100% 76 in → 1 out (0 reasoning)
none
poolside/laguna-xs-2.1:xhigh PASS 100% 79 in → 141 out (139 reasoning)
none
poolside/laguna-xs.2:high PASS 100% 116 in → 168 out (164 reasoning)
none
poolside/laguna-xs.2:low PASS 100% 114 in → 189 out (185 reasoning)
none
poolside/laguna-xs.2:low__json_schema PASS 100% 113 in → 162 out (152 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs.2:medium PASS 100% 115 in → 130 out (126 reasoning)
none
poolside/laguna-xs.2:medium__json_schema PASS 100% 116 in → 176 out (166 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs.2:none PASS 100% 116 in → 3 out (0 reasoning)
none
poolside/laguna-xs.2:none__json_schema PASS 100% 113 in → 9 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs.2:xhigh PASS 100% 116 in → 131 out (127 reasoning)
none
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 114 in → 187 out (172 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
qwen/qwen3.6-27b:high PASS 100% 75 in → 345 out (339 reasoning)
none
qwen/qwen3.6-27b:low PASS 100% 77 in → 526 out (516 reasoning)
none
qwen/qwen3.6-27b:medium PASS 100% 77 in → 484 out (469 reasoning)
none
qwen/qwen3.6-27b:none PASS 100% 78 in → 2 out (0 reasoning)
none
qwen/qwen3.6-27b:xhigh PASS 100% 75 in → 647 out (636 reasoning)
none
qwen/qwen3.6-35b-a3b:high PASS 100% 75 in → 571 out (565 reasoning)
none
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 77 in → 269 out (259 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
qwen/qwen3.6-35b-a3b:low PASS 100% 76 in → 561 out (545 reasoning)
none
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 76 in → 597 out (571 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 74 in → 333 out (327 reasoning)
none
qwen/qwen3.6-35b-a3b:none PASS 100% 77 in → 1 out (0 reasoning)
none
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 76 in → 500 out (479 reasoning)
none
qwen/qwen3.6-flash:high PASS 100% 75 in → 677 out (671 reasoning)
none
qwen/qwen3.6-flash:low PASS 100% 76 in → 578 out (572 reasoning)
none
qwen/qwen3.6-flash:medium PASS 100% 76 in → 764 out (758 reasoning)
none
qwen/qwen3.6-flash:none PASS 100% 79 in → 1 out (0 reasoning)
none
qwen/qwen3.6-flash:xhigh PASS 100% 75 in → 727 out (721 reasoning)
none
qwen/qwen3.7-flash:high PASS 100% 76 in → 376 out (370 reasoning)
none
qwen/qwen3.7-flash:low PASS 100% 76 in → 322 out (316 reasoning)
none
qwen/qwen3.7-flash:max PASS 100% 75 in → 448 out (442 reasoning)
none
qwen/qwen3.7-flash:medium PASS 100% 76 in → 335 out (329 reasoning)
none
qwen/qwen3.7-flash:none PASS 100% 78 in → 1 out (0 reasoning)
none
qwen/qwen3.7-flash:xhigh PASS 100% 73 in → 820 out (814 reasoning)
none
qwen/qwen3.7-max:high PASS 100% 77 in → 405 out (399 reasoning)
none
qwen/qwen3.7-max:low PASS 100% 76 in → 509 out (503 reasoning)
none
qwen/qwen3.7-max:medium PASS 100% 76 in → 367 out (361 reasoning)
none
qwen/qwen3.7-max:none PASS 100% 78 in → 1 out (0 reasoning)
none
qwen/qwen3.7-max:xhigh PASS 100% 76 in → 363 out (358 reasoning)
none
qwen/qwen3.7-plus:high PASS 100% 77 in → 445 out (440 reasoning)
none
qwen/qwen3.7-plus:low PASS 100% 76 in → 487 out (481 reasoning)
none
qwen/qwen3.7-plus:medium PASS 100% 75 in → 455 out (449 reasoning)
none
qwen/qwen3.7-plus:none PASS 100% 78 in → 1 out (0 reasoning)
none
qwen/qwen3.7-plus:xhigh PASS 100% 76 in → 98 out (92 reasoning)
none
tencent/hy3:high PASS 100% 77 in → 253 out (250 reasoning)
none
tencent/hy3:high__json_schema PASS 100% 74 in → 235 out (241 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
tencent/hy3:low PASS 100% 77 in → 262 out (276 reasoning)
none
tencent/hy3:low__json_schema PASS 100% 76 in → 270 out (260 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
tencent/hy3:medium PASS 100% 76 in → 238 out (235 reasoning)
none
tencent/hy3:medium__json_schema PASS 100% 76 in → 226 out (225 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
tencent/hy3:none PASS 100% 78 in → 2 out (0 reasoning)
none
tencent/hy3:none__json_schema PASS 100% 81 in → 12 out (0 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
tencent/hy3:xhigh PASS 100% 76 in → 224 out (221 reasoning)
none
tencent/hy3:xhigh__json_schema PASS 100% 75 in → 429 out (459 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
thinkingmachines/inkling-small:high PASS 100% 76 in → 184 out (204 reasoning)
none
thinkingmachines/inkling-small:low PASS 100% 74 in → 40 out (33 reasoning)
none
thinkingmachines/inkling-small:max PASS 100% 74 in → 137 out (130 reasoning)
none
thinkingmachines/inkling-small:medium PASS 100% 75 in → 195 out (188 reasoning)
none
thinkingmachines/inkling-small:none PASS 100% 72 in → 4 out (0 reasoning)
none
thinkingmachines/inkling-small:xhigh PASS 100% 76 in → 149 out (142 reasoning)
none
thinkingmachines/inkling:high PASS 100% 75 in → 122 out (114 reasoning)
none
thinkingmachines/inkling:low PASS 100% 74 in → 53 out (46 reasoning)
none
thinkingmachines/inkling:low__json_schema PASS 100% 74 in → 26 out (6 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
thinkingmachines/inkling:max PASS 100% 76 in → 141 out (157 reasoning)
none
thinkingmachines/inkling:medium PASS 100% 74 in → 126 out (119 reasoning)
none
thinkingmachines/inkling:none PASS 100% 72 in → 4 out (0 reasoning)
none
thinkingmachines/inkling:xhigh PASS 100% 73 in → 229 out (268 reasoning)
none
x-ai/grok-4.3:high PASS 100% 244 in → 208 out (207 reasoning)
none
x-ai/grok-4.3:low PASS 100% 254 in → 177 out (176 reasoning)
none
x-ai/grok-4.3:max PASS 100% 246 in → 165 out (164 reasoning)
none
x-ai/grok-4.3:medium PASS 100% 250 in → 196 out (195 reasoning)
none
x-ai/grok-4.3:none PASS 100% 242 in → 1 out (0 reasoning)
none
x-ai/grok-4.3:xhigh PASS 100% 246 in → 229 out (228 reasoning)
none
x-ai/grok-4.5:high PASS 100% 267 in → 142 out (141 reasoning)
none
x-ai/grok-4.5:low PASS 100% 267 in → 186 out (185 reasoning)
none
x-ai/grok-4.5:max PASS 100% 268 in → 231 out (230 reasoning)
none
x-ai/grok-4.5:medium PASS 100% 268 in → 166 out (165 reasoning)
none
x-ai/grok-4.5:xhigh PASS 100% 266 in → 187 out (186 reasoning)
none
z-ai/glm-4.7-flash:high PASS 100% 69 in → 1,129 out (1,181 reasoning)
none
z-ai/glm-4.7-flash:low PASS 100% 68 in → 872 out (864 reasoning)
none
z-ai/glm-4.7-flash:medium PASS 100% 68 in → 393 out (393 reasoning)
none
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 68 in → 386 out (361 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
z-ai/glm-4.7-flash:xhigh PASS 100% 66 in → 344 out (329 reasoning)
none
z-ai/glm-5.2:high PASS 100% 74 in → 200 out (197 reasoning)
none
z-ai/glm-5.2:low PASS 100% 75 in → 88 out (86 reasoning)
none
z-ai/glm-5.2:medium PASS 100% 75 in → 61 out (58 reasoning)
none
z-ai/glm-5.2:none PASS 100% 70 in → 2 out (0 reasoning)
none
z-ai/glm-5.2:none__json_schema PASS 100% 68 in → 16 out (0 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
z-ai/glm-5.2:xhigh PASS 100% 75 in → 253 out (250 reasoning)
none
anthropic/claude-fable-5:high__json_schema FAIL 0% 334 in → 35 out (23 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:low__json_schema FAIL 0% 331 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:max__json_schema FAIL 0% 331 in → 148 out (59 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:medium__json_schema FAIL 0% 331 in → 30 out (15 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:xhigh__json_schema FAIL 0% 332 in → 52 out (31 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:high__json_schema FAIL 0% 290 in → 218 out (224 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:low__json_schema FAIL 0% 290 in → 231 out (244 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:medium__json_schema FAIL 0% 293 in → 380 out (376 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 262 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:xhigh__json_schema FAIL 0% 292 in → 254 out (265 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 264 in → 55 out (31 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:low__json_schema FAIL 0% 264 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:max__json_schema FAIL 0% 264 in → 103 out (88 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:medium__json_schema FAIL 0% 262 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:none__json_schema FAIL 0% 262 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 262 in → 62 out (40 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:high__json_schema FAIL 0% 338 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:low__json_schema FAIL 0% 337 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:max__json_schema FAIL 0% 337 in → 132 out (53 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:medium__json_schema FAIL 0% 339 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:none__json_schema FAIL 0% 336 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:xhigh__json_schema FAIL 0% 338 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:high__json_schema FAIL 0% 332 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:low__json_schema FAIL 0% 333 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 331 in → 145 out (29 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:medium__json_schema FAIL 0% 332 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:none__json_schema FAIL 0% 333 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:xhigh__json_schema FAIL 0% 331 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:high__json_schema FAIL 0% 262 in → 92 out (74 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:low__json_schema FAIL 0% 264 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:max__json_schema FAIL 0% 264 in → 149 out (134 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:medium__json_schema FAIL 0% 264 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:none__json_schema FAIL 0% 262 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:xhigh__json_schema FAIL 0% 263 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:high__json_schema FAIL 0% 332 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 332 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 332 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 331 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 332 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash-0731:high__json_schema FAIL 0% 148 in → 137 out (132 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 0% 69 in → 37 out (39 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output: { { "branches_to_delete": [] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)
deepseek/deepseek-v4-flash-0731:medium__json_schema FAIL 0% 67 in → 157 out (149 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 69 in → 10 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 70 in → 56 out (55 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output: none
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:low__json_schema FAIL 0% 69 in → 68 out (57 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 69 in → 10 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0% 174 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
deepseek/deepseek-v4-pro:high__json_schema FAIL 0% 265 in → 264 out (254 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 268 in → 255 out (245 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 267 in → 322 out (312 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 69 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 346 in → 768 out (758 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3-flash-preview:high__json_schema FAIL 0% 66 in → 299 out (291 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3-flash-preview:low__json_schema FAIL 0% 142 in → 205 out (197 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3-flash-preview:medium__json_schema FAIL 0% 146 in → 297 out (284 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3-flash-preview:xhigh__json_schema FAIL 0% 65 in → 183 out (175 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:high__json_schema FAIL 0% 66 in → 594 out (585 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 0% 64 in → 107 out (94 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 66 in → 230 out (217 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:xhigh__json_schema FAIL 0% 146 in → 658 out (649 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:high__json_schema FAIL 0% 145 in → 599 out (591 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 146 in → 207 out (198 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:medium__json_schema FAIL 0% 146 in → 502 out (494 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:xhigh__json_schema FAIL 0% 142 in → 641 out (633 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:high__json_schema FAIL 0% 65 in → 373 out (365 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 65 in → 13 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:max__json_schema FAIL 0% 145 in → 252 out (239 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:medium__json_schema FAIL 0% 64 in → 275 out (262 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:xhigh__json_schema FAIL 0% 65 in → 360 out (352 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash:high__json_schema FAIL 0% 145 in → 327 out (319 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash:low__json_schema FAIL 0% 146 in → 89 out (76 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.5-flash:medium__json_schema FAIL 0% 145 in → 223 out (215 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash:xhigh__json_schema FAIL 0% 146 in → 445 out (437 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:high__json_schema FAIL 0% 146 in → 259 out (251 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:low__json_schema FAIL 0% 146 in → 13 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.6-flash:max__json_schema FAIL 0% 65 in → 414 out (406 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:medium__json_schema FAIL 0% 65 in → 211 out (203 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:xhigh__json_schema FAIL 0% 146 in → 307 out (299 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemma-4-26b-a4b-it:high__json_schema FAIL 0% 79 in → 229 out (202 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemma-4-26b-a4b-it:none__json_schema FAIL 0% 77 in → 13 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
main
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "main" ] }
Failure: Missing: ['none'] Extra: ['main']
liquid/lfm-2-24b-a2b FAIL 0% 74 in → 2 out
main
Failure: Missing: ['none'] Extra: ['main']
Invalid JSON. Output: main
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:low__json_schema FAIL 0% 101 in → 242 out (255 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.5:medium__json_schema FAIL 0% 99 in → 233 out (225 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.5:xhigh__json_schema FAIL 0% 100 in → 128 out (116 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:low__json_schema FAIL 0% 251 in → 357 out (345 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.7:medium__json_schema FAIL 0% 250 in → 203 out (193 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high__json_schema FAIL 0% 223 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m3:low__json_schema FAIL 0% 226 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
minimax/minimax-m3:medium__json_schema FAIL 0% 224 in → 10 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:high__json_schema FAIL 0% 68 in → 141 out (159 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:low__json_schema FAIL 0% 69 in → 649 out (743 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:medium__json_schema FAIL 0% 69 in → 372 out (435 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:none__json_schema FAIL 0% 68 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
moonshotai/kimi-k2.6:high__json_schema FAIL 0% 67 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0% 67 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 81 in → 32 out (0 reasoning)
main
JSON Schema Structured Output
(raw) { "branches_to_delete" : [ "main" ] }
Failure: Missing: ['none'] Extra: ['main']
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 0% 78 in → 718 out (743 reasoning)
debcfc0
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "debcfc0" ] }
Failure: Missing: ['none'] Extra: ['debcfc0']
nvidia/nemotron-3-super-120b-a12b:high__json_schema FAIL 0% 82 in → 117 out (102 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:low__json_schema FAIL 0% 81 in → 164 out (147 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:medium__json_schema FAIL 0% 81 in → 67 out (51 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:none__json_schema FAIL 0% 82 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema FAIL 0% 80 in → 162 out (146 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
openai/gpt-5.4-mini:high__json_schema FAIL 0% 110 in → 380 out (364 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-mini:medium__json_schema FAIL 0% 108 in → 104 out (88 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[] }
Failure: Missing: ['none']
openai/gpt-5.4-mini:none__json_schema FAIL 0% 109 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-mini:xhigh__json_schema FAIL 0% 110 in → 873 out (857 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-nano:high__json_schema FAIL 0% 110 in → 329 out (313 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-nano:medium__json_schema FAIL 0% 109 in → 30 out (14 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-nano:xhigh__json_schema FAIL 0% 109 in → 417 out (401 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:low__json_schema FAIL 0% 110 in → 38 out (22 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:medium__json_schema FAIL 0% 107 in → 86 out (70 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:none__json_schema FAIL 0% 107 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:xhigh__json_schema FAIL 0% 108 in → 96 out (80 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:high__json_schema FAIL 0% 109 in → 73 out (57 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:low__json_schema FAIL 0% 109 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:medium__json_schema FAIL 0% 109 in → 49 out (33 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:none__json_schema FAIL 0% 109 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:xhigh__json_schema FAIL 0% 111 in → 72 out (56 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-luna:high__json_schema FAIL 0% 108 in → 232 out (216 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-luna:low__json_schema FAIL 0% 110 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-luna:max__json_schema FAIL 0% 108 in → 210 out (194 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-luna:medium__json_schema FAIL 0% 108 in → 106 out (90 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-luna:none__json_schema FAIL 0% 108 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:high__json_schema FAIL 0% 108 in → 76 out (60 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:low__json_schema FAIL 0% 110 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:max__json_schema FAIL 0% 108 in → 147 out (131 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:medium__json_schema FAIL 0% 109 in → 54 out (38 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:none__json_schema FAIL 0% 111 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:xhigh__json_schema FAIL 0% 109 in → 87 out (71 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:high__json_schema FAIL 0% 109 in → 36 out (20 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:low__json_schema FAIL 0% 110 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:max__json_schema FAIL 0% 110 in → 56 out (40 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:medium__json_schema FAIL 0% 108 in → 35 out (19 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:none__json_schema FAIL 0% 108 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:xhigh__json_schema FAIL 0% 109 in → 44 out (28 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-oss-20b:low__json_schema FAIL 0% 126 in → 28 out (11 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
openai/gpt-oss-20b:medium__json_schema FAIL 0% 116 in → 196 out (202 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
poolside/laguna-m.1:medium__json_schema FAIL 0% 77 in → 344 out (335 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 79 in → 59 out (0 reasoning)
feature/login bugfix/header docs/readme-update hotfix/critical-patch refactor/utils test/unit-tests
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature/login", "bugfix/header", "docs/readme-update", "hotfix/critical-patch", "refactor/utils", "test/unit-tests" ] }
Failure: Missing: ['none'] Extra: ['bugfix/header', 'docs/readme-update', 'feature/login', 'hotfix/critical-patch', 'refactor/utils', 'test/unit-tests']
poolside/laguna-xs-2.1:xhigh__json_schema FAIL 0% 79 in → 273 out (264 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs.2:high__json_schema FAIL 0% 115 in → 176 out (167 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
qwen/qwen3.6-27b:high__json_schema FAIL 0% 76 in → 773 out (738 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-27b:low__json_schema FAIL 0% 77 in → 595 out (565 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-27b:medium__json_schema FAIL 0% 76 in → 359 out (348 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 77 in → 20 out (0 reasoning)
main
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "main" ] }
Failure: Missing: ['none'] Extra: ['main']
qwen/qwen3.6-27b:xhigh__json_schema FAIL 0% 77 in → 549 out (540 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-35b-a3b:medium__json_schema FAIL 0% 74 in → 651 out (632 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 78 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
qwen/qwen3.6-35b-a3b:xhigh__json_schema FAIL 0% 77 in → 427 out (405 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-34dac6c7-ac4e-9982-b1b8-31cf3d21a682","request_id":"34dac6c7-ac4e-9982-b1b8-31cf3d21a682"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f7c8e618-1119-9775-aad6-38dfa33cafc5","request_id":"f7c8e618-1119-9775-aad6-38dfa33cafc5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b711d50e-29f0-90c1-86cb-384bb8caa0ee","request_id":"b711d50e-29f0-90c1-86cb-384bb8caa0ee"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ae9c3ee6-4e10-9c69-8497-8c67ead9cf54","request_id":"ae9c3ee6-4e10-9c69-8497-8c67ead9cf54"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cc06673a-34f3-9bd2-8784-d3113f58e5a7","request_id":"cc06673a-34f3-9bd2-8784-d3113f58e5a7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e82d9e03-0aba-96c7-94ab-2a8355a4cb20"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2a39f13a-4fd0-9d01-a2c3-603e65bb75c4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-38cdb1a5-7a0c-996b-afe5-5262ab7b88ea"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c69fdfd8-dadc-9146-b1d1-96d24e40e8fe"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b56ddd72-5295-99ea-ae01-d6ebb85be6de"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1b385774-b266-96bc-a9e0-d70af0da13e5"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-629826c4-58ed-9f35-b4b8-56c95426ad11","request_id":"629826c4-58ed-9f35-b4b8-56c95426ad11"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-750dcd1d-b6bd-9df7-9b93-33d18cfc4fd1","request_id":"750dcd1d-b6bd-9df7-9b93-33d18cfc4fd1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ab9085a3-9785-94a0-a8cd-a601b647e377","request_id":"ab9085a3-9785-94a0-a8cd-a601b647e377"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-be32415d-1eee-904c-9cbf-886d62fb82e8","request_id":"be32415d-1eee-904c-9cbf-886d62fb82e8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9cc20426-b76b-9b06-9938-888acb4f4571","request_id":"9cc20426-b76b-9b06-9938-888acb4f4571"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-96fb5d3f-9118-9ab9-8b88-9396267b2ca1","request_id":"96fb5d3f-9118-9ab9-8b88-9396267b2ca1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0a630bed-00f8-945b-b2a5-2f5f03db77d2","request_id":"0a630bed-00f8-945b-b2a5-2f5f03db77d2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4a3c5d8b-b929-9b72-b9c7-8c8c0a3ff5bb","request_id":"4a3c5d8b-b929-9b72-b9c7-8c8c0a3ff5bb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9beaaf1a-2796-9aed-a5ff-12f5679922d2","request_id":"9beaaf1a-2796-9aed-a5ff-12f5679922d2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-09ae6528-716a-95e3-bb8c-2d1e786551cd","request_id":"09ae6528-716a-95e3-bb8c-2d1e786551cd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:high__json_schema FAIL 0% 73 in → 116 out (103 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
thinkingmachines/inkling-small:low__json_schema FAIL 0% 75 in → 119 out (106 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
thinkingmachines/inkling-small:max__json_schema FAIL 0% 74 in → 119 out (106 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
thinkingmachines/inkling-small:medium__json_schema FAIL 0% 74 in → 122 out (108 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output: none{ "branches_to_delete": [] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling-small:xhigh__json_schema FAIL 0% 75 in → 120 out (107 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
Invalid JSON. Output: none
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: none
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: none
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0% 75 in → 10 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
thinkingmachines/inkling:xhigh__json_schema FAIL 0% 75 in → 156 out (143 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
x-ai/grok-4.3:high__json_schema FAIL 0% 310 in → 418 out (412 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:low__json_schema FAIL 0% 315 in → 272 out (266 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:max__json_schema FAIL 0% 309 in → 418 out (412 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:medium__json_schema FAIL 0% 316 in → 302 out (296 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:none__json_schema FAIL 0% 308 in → 6 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:xhigh__json_schema FAIL 0% 309 in → 338 out (332 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:high__json_schema FAIL 0% 347 in → 381 out (373 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:low__json_schema FAIL 0% 348 in → 284 out (276 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:max__json_schema FAIL 0% 348 in → 400 out (392 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:medium__json_schema FAIL 0% 350 in → 455 out (447 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:xhigh__json_schema FAIL 0% 350 in → 262 out (250 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
z-ai/glm-4.7-flash:high__json_schema FAIL 0% 68 in → 339 out (331 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output: none
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 68 in → 1 out (0 reasoning)
main
Failure: Missing: ['none'] Extra: ['main']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 67 in → 10 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0% 68 in → 462 out (438 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
z-ai/glm-5.2:high__json_schema FAIL 0% 258 in → 55 out (44 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
z-ai/glm-5.2:low__json_schema FAIL 0% 74 in → 71 out (58 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [ ] }
Failure: Missing: ['none']
z-ai/glm-5.2:medium__json_schema FAIL 0% 76 in → 214 out (205 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
z-ai/glm-5.2:xhigh__json_schema FAIL 0% 75 in → 237 out (231 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']