No branches to delete
Tests ability to recognize when no branches need deletion (edge case). Evaluates handling of the empty-result scenario in branch cleanup.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Hello' > greeting.txt
  5. 05 git add greeting.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 git checkout -b feature-a
  8. 08 echo 'Feature A' > a.txt
  9. 09 git add a.txt
  10. 10 git commit -m 'Feature A'
  11. 11 git checkout main
  12. 12 git checkout -b feature-b
  13. 13 echo 'Feature B' > b.txt
  14. 14 git add b.txt
  15. 15 git commit -m 'Feature B'
  16. 16 git checkout main
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
none

Scoped model quality, cost, API time, and token usage for branch_cleanup/f003.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high FAIL 100% 124 in → 113 out (25 reasoning)
none Neither `feature-a` nor `feature-b` appears in the list of branches merged into main — only `main` itself does. Their commits ("Feature A" and "Feature B") are not reachable from main, so deleting them would lose work.
Failure: Extra: ['Neither `feature-a` nor `feature-b` appears in the list of branches merged into main — only `main` itself does. Their commits ("Feature A" and "Feature B") are not reachable from main, so deleting them would lose work.']
anthropic/claude-fable-5:low FAIL 100% 127 in → 101 out (12 reasoning)
none Neither `feature-a` nor `feature-b` appears in the "merged into main" list — only `main` itself does. Since their commits (8d60261, dd07ddd) differ from main's tip (3de1dc9) and aren't merged, deleting them would lose work.
Failure: Extra: ['Neither `feature-a` nor `feature-b` appears in the "merged into main" list — only `main` itself does. Since their commits (8d60261, dd07ddd) differ from main\'s tip (3de1dc9) and aren\'t merged, deleting them would lose work.']
anthropic/claude-fable-5:max PASS 100% 127 in → 272 out (58 reasoning)
none
anthropic/claude-fable-5:medium FAIL 100% 126 in → 108 out (30 reasoning)
none **Explanation:** The "Merged into main" list only shows `main` itself. Neither `feature-a` nor `feature-b` appears in the merged list, meaning they contain commits not yet merged into main. Deleting them would lose unmerged work.
Failure: Extra: ['**Explanation:** The "Merged into main" list only shows `main` itself. Neither `feature-a` nor `feature-b` appears in the merged list, meaning they contain commits not yet merged into main. Deleting them would lose unmerged work.']
anthropic/claude-fable-5:xhigh FAIL 100% 124 in → 139 out (34 reasoning)
none **Explanation:** The `git branch --merged main` output shows only `main` itself. Neither `feature-a` (cecddc3) nor `feature-b` (40cf191) appear in the merged list — they contain commits not yet merged into main, so deleting them would lose work.
Failure: Extra: ['**Explanation:** The `git branch --merged main` output shows only `main` itself. Neither `feature-a` (cecddc3) nor `feature-b` (40cf191) appear in the merged list — they contain commits not yet merged into main, so deleting them would lose work.']
anthropic/claude-haiku-4.5:high PASS 100% 128 in → 190 out (189 reasoning)
none
anthropic/claude-haiku-4.5:low PASS 100% 126 in → 155 out (156 reasoning)
none
anthropic/claude-haiku-4.5:medium PASS 100% 129 in → 155 out (138 reasoning)
none
anthropic/claude-haiku-4.5:none PASS 100% 98 in → 4 out (0 reasoning)
none
anthropic/claude-haiku-4.5:xhigh PASS 100% 128 in → 181 out (177 reasoning)
none
anthropic/claude-opus-4.6:high PASS 100% 100 in → 124 out (92 reasoning)
none
anthropic/claude-opus-4.6:low FAIL 100% 100 in → 43 out (0 reasoning)
Looking at the "Merged into main" section, only `main` itself is listed as merged into main. Neither `feature-a` nor `feature-b` are merged. none
Failure: Extra: ['Looking at the "Merged into main" section, only `main` itself is listed as merged into main. Neither `feature-a` nor `feature-b` are merged.']
anthropic/claude-opus-4.6:max PASS 100% 97 in → 164 out (147 reasoning)
none
anthropic/claude-opus-4.6:medium FAIL 100% 102 in → 46 out (0 reasoning)
Looking at the "Merged into main" section, only `main` itself is listed as merged into main. Neither `feature-a` nor `feature-b` are listed as merged. none
Failure: Extra: ['Looking at the "Merged into main" section, only `main` itself is listed as merged into main. Neither `feature-a` nor `feature-b` are listed as merged.']
anthropic/claude-opus-4.6:none FAIL 100% 99 in → 50 out (0 reasoning)
Looking at the information provided: - `feature-a` and `feature-b` are **not** listed as merged into main. - Only `main` itself is listed under "Merged into main." none
Failure: Extra: ['- Only `main` itself is listed under "Merged into main."', '- `feature-a` and `feature-b` are **not** listed as merged into main.', 'Looking at the information provided:']
anthropic/claude-opus-4.6:xhigh PASS 100% 101 in → 102 out (71 reasoning)
none
anthropic/claude-opus-4.7:high PASS 100% 127 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:low PASS 100% 128 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:max PASS 100% 131 in → 137 out (35 reasoning)
none
anthropic/claude-opus-4.7:medium PASS 100% 129 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:none PASS 100% 132 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:xhigh PASS 100% 129 in → 5 out (0 reasoning)
none
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 371 in → 19 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
anthropic/claude-opus-4.8:high FAIL 100% 126 in → 136 out (41 reasoning)
none The only branch listed as merged into main is `main` itself. Neither `feature-a` nor `feature-b` appears in the "Merged into main" list, so they contain unmerged work and should not be deleted.
Failure: Extra: ['The only branch listed as merged into main is `main` itself. Neither `feature-a` nor `feature-b` appears in the "Merged into main" list, so they contain unmerged work and should not be deleted.']
anthropic/claude-opus-4.8:low PASS 100% 124 in → 68 out (62 reasoning)
none
anthropic/claude-opus-4.8:max FAIL 100% 128 in → 576 out (71 reasoning)
none The "Merged into main" list shows only `main` itself. This means: - **feature-a** — NOT merged into main (has unmerged commits) - **feature-b** — NOT merged into main (has unmerged commits) - **main** — this is the main branch itself, which should never be deleted Since neither feature branch is fully merged into main, deleting them would risk losing work. No branches are safe to delete.
Failure: Extra: ['- **feature-a** — NOT merged into main (has unmerged commits)', '- **feature-b** — NOT merged into main (has unmerged commits)', '- **main** — this is the main branch itself, which should never be deleted', 'Since neither feature branch is fully merged into main, deleting them would risk losing work. No branches are safe to delete.', 'The "Merged into main" list shows only `main` itself. This means:']
anthropic/claude-opus-4.8:medium FAIL 100% 129 in → 114 out (34 reasoning)
none Based on the "Merged into main" list, only `main` itself appears—meaning neither `feature-a` nor `feature-b` has been merged into main yet. Therefore, no branches should be deleted. none
Failure: Extra: ['Based on the "Merged into main" list, only `main` itself appears—meaning neither `feature-a` nor `feature-b` has been merged into main yet. Therefore, no branches should be deleted.']
anthropic/claude-opus-4.8:none FAIL 100% 124 in → 64 out (0 reasoning)
Based on the information provided, only `main` is listed as merged into main. Neither `feature-a` nor `feature-b` appears in the "Merged into main" list, so they are not fully merged. none
Failure: Extra: ['Based on the information provided, only `main` is listed as merged into main. Neither `feature-a` nor `feature-b` appears in the "Merged into main" list, so they are not fully merged.']
anthropic/claude-opus-4.8:xhigh FAIL 100% 124 in → 160 out (59 reasoning)
none Neither `feature-a` nor `feature-b` appears in the "Merged into main" list (which only contains `main` itself), so no branches are fully merged into main and safe to delete.
Failure: Extra: ['Neither `feature-a` nor `feature-b` appears in the "Merged into main" list (which only contains `main` itself), so no branches are fully merged into main and safe to delete.']
anthropic/claude-sonnet-4.6:high PASS 100% 101 in → 77 out (61 reasoning)
none
anthropic/claude-sonnet-4.6:low PASS 100% 99 in → 4 out (0 reasoning)
none
anthropic/claude-sonnet-4.6:max PASS 100% 102 in → 87 out (76 reasoning)
none
anthropic/claude-sonnet-4.6:medium PASS 100% 100 in → 48 out (32 reasoning)
none
anthropic/claude-sonnet-4.6:none PASS 100% 100 in → 4 out (0 reasoning)
none
anthropic/claude-sonnet-4.6:xhigh PASS 100% 102 in → 77 out (61 reasoning)
none
anthropic/claude-sonnet-5:high PASS 100% 131 in → 54 out (32 reasoning)
none
anthropic/claude-sonnet-5:high__json_schema PASS 100% 366 in → 17 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
anthropic/claude-sonnet-5:low PASS 100% 128 in → 3 out (0 reasoning)
none
anthropic/claude-sonnet-5:low__json_schema PASS 100% 361 in → 17 out (0 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
anthropic/claude-sonnet-5:none FAIL 100% 123 in → 55 out (0 reasoning)
none Only `main` is listed as merged into main—`feature-a` and `feature-b` don't appear in that list, so they are not fully merged and should not be deleted.
Failure: Extra: ["Only `main` is listed as merged into main—`feature-a` and `feature-b` don't appear in that list, so they are not fully merged and should not be deleted."]
anthropic/claude-sonnet-5:xhigh PASS 100% 126 in → 95 out (52 reasoning)
none
arcee-ai/trinity-large-thinking:high PASS 100% 90 in → 1,180 out (1,204 reasoning)
none
arcee-ai/trinity-large-thinking:low PASS 100% 93 in → 836 out (851 reasoning)
none
arcee-ai/trinity-large-thinking:medium PASS 100% 91 in → 1,064 out (1,092 reasoning)
none
arcee-ai/trinity-large-thinking:xhigh PASS 100% 92 in → 2,021 out (2,070 reasoning)
none
arcee-ai/trinity-mini:high PASS 100% 91 in → 590 out (711 reasoning)
none
arcee-ai/trinity-mini:high__json_schema PASS 100% 92 in → 270 out (300 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
arcee-ai/trinity-mini:low PASS 100% 91 in → 296 out (345 reasoning)
none
arcee-ai/trinity-mini:low__json_schema PASS 100% 92 in → 684 out (798 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
arcee-ai/trinity-mini:medium PASS 100% 89 in → 946 out (1,138 reasoning)
none
arcee-ai/trinity-mini:medium__json_schema PASS 100% 92 in → 428 out (493 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 89 in → 562 out (662 reasoning)
none
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 92 in → 327 out (366 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 168 in → 120 out (118 reasoning)
none
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 94 in → 126 out (163 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 92 in → 52 out (50 reasoning)
none
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 88 in → 246 out (332 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 182 in → 109 out (107 reasoning)
none
deepseek/deepseek-v4-flash-0731:medium PASS 100% 93 in → 438 out (567 reasoning)
none
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 169 in → 117 out (115 reasoning)
none
deepseek/deepseek-v4-flash:high PASS 100% 92 in → 129 out (125 reasoning)
none
deepseek/deepseek-v4-flash:low PASS 100% 92 in → 137 out (136 reasoning)
none
deepseek/deepseek-v4-flash:medium PASS 100% 91 in → 158 out (155 reasoning)
none
deepseek/deepseek-v4-flash:xhigh PASS 100% 171 in → 517 out (585 reasoning)
none
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 172 in → 1,481 out (1,485 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
deepseek/deepseek-v4-pro:low PASS 100% 89 in → 158 out (156 reasoning)
none
deepseek/deepseek-v4-pro:medium PASS 100% 89 in → 216 out (238 reasoning)
none
deepseek/deepseek-v4-pro:xhigh PASS 100% 87 in → 469 out (466 reasoning)
none
google/gemini-3-flash-preview:high PASS 100% 90 in → 222 out (221 reasoning)
none
google/gemini-3-flash-preview:low PASS 100% 94 in → 233 out (232 reasoning)
none
google/gemini-3-flash-preview:medium PASS 100% 89 in → 325 out (324 reasoning)
none
google/gemini-3-flash-preview:xhigh PASS 100% 89 in → 242 out (241 reasoning)
none
google/gemini-3.1-flash-lite-preview:high PASS 100% 91 in → 389 out (388 reasoning)
none
google/gemini-3.1-flash-lite-preview:low PASS 100% 92 in → 118 out (117 reasoning)
none
google/gemini-3.1-flash-lite-preview:medium PASS 100% 90 in → 262 out (261 reasoning)
none
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 93 in → 395 out (394 reasoning)
none
google/gemini-3.1-pro-preview:high PASS 100% 92 in → 139 out (138 reasoning)
none
google/gemini-3.1-pro-preview:low PASS 100% 90 in → 149 out (148 reasoning)
none
google/gemini-3.1-pro-preview:medium PASS 100% 91 in → 214 out (213 reasoning)
none
google/gemini-3.1-pro-preview:xhigh PASS 100% 93 in → 326 out (325 reasoning)
none
google/gemini-3.5-flash-lite:high PASS 100% 93 in → 515 out (514 reasoning)
none
google/gemini-3.5-flash-lite:low PASS 100% 95 in → 1 out (0 reasoning)
none
google/gemini-3.5-flash-lite:max PASS 100% 93 in → 467 out (466 reasoning)
none
google/gemini-3.5-flash-lite:medium PASS 100% 91 in → 416 out (415 reasoning)
none
google/gemini-3.5-flash-lite:xhigh PASS 100% 93 in → 315 out (314 reasoning)
none
google/gemini-3.5-flash:high PASS 100% 92 in → 388 out (387 reasoning)
none
google/gemini-3.5-flash:low PASS 100% 91 in → 119 out (118 reasoning)
none
google/gemini-3.5-flash:medium PASS 100% 93 in → 370 out (369 reasoning)
none
google/gemini-3.5-flash:xhigh PASS 100% 94 in → 385 out (384 reasoning)
none
google/gemini-3.6-flash:high PASS 100% 93 in → 335 out (334 reasoning)
none
google/gemini-3.6-flash:low PASS 100% 91 in → 149 out (148 reasoning)
none
google/gemini-3.6-flash:max PASS 100% 94 in → 405 out (404 reasoning)
none
google/gemini-3.6-flash:medium PASS 100% 93 in → 221 out (220 reasoning)
none
google/gemini-3.6-flash:xhigh PASS 100% 93 in → 372 out (371 reasoning)
none
google/gemma-4-26b-a4b-it:high PASS 100% 108 in → 793 out (739 reasoning)
none
google/gemma-4-26b-a4b-it:low PASS 100% 106 in → 432 out (385 reasoning)
none
google/gemma-4-26b-a4b-it:medium PASS 100% 109 in → 616 out (533 reasoning)
none
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 108 in → 415 out (336 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 106 in → 2 out (0 reasoning)
none
google/gemma-4-26b-a4b-it:xhigh PASS 100% 109 in → 294 out (248 reasoning)
none
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 108 in → 297 out (236 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-31b-it:high PASS 100% 107 in → 240 out (200 reasoning)
none
google/gemma-4-31b-it:low PASS 100% 106 in → 230 out (205 reasoning)
none
google/gemma-4-31b-it:medium PASS 100% 107 in → 221 out (184 reasoning)
none
google/gemma-4-31b-it:medium__json_schema PASS 100% 110 in → 212 out (1 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
google/gemma-4-31b-it:none PASS 100% 107 in → 2 out (0 reasoning)
none
google/gemma-4-31b-it:xhigh PASS 100% 108 in → 212 out (1 reasoning)
none
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 107 in → 308 out (1 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
minimax/minimax-m2.5:high PASS 100% 120 in → 328 out (382 reasoning)
none
minimax/minimax-m2.5:low PASS 100% 125 in → 244 out (242 reasoning)
none
minimax/minimax-m2.5:medium PASS 100% 122 in → 724 out (788 reasoning)
none
minimax/minimax-m2.5:xhigh PASS 100% 119 in → 1,601 out (1,882 reasoning)
none
minimax/minimax-m2.7:high PASS 100% 125 in → 304 out (302 reasoning)
none
minimax/minimax-m2.7:low PASS 100% 124 in → 388 out (385 reasoning)
none
minimax/minimax-m2.7:xhigh PASS 100% 124 in → 873 out (871 reasoning)
none
minimax/minimax-m3:high PASS 100% 261 in → 308 out (361 reasoning)
none
minimax/minimax-m3:low PASS 100% 260 in → 165 out (196 reasoning)
none
minimax/minimax-m3:medium PASS 100% 262 in → 158 out (166 reasoning)
none
minimax/minimax-m3:xhigh PASS 100% 261 in → 292 out (323 reasoning)
none
mistralai/devstral-2512 PASS 100% 96 in → 2 out
none
mistralai/mistral-medium-3-5:high PASS 100% 107 in → 857 out (932 reasoning)
none
mistralai/mistral-medium-3-5:low PASS 100% 104 in → 1,115 out (1,246 reasoning)
none
mistralai/mistral-medium-3-5:medium PASS 100% 106 in → 372 out (392 reasoning)
none
mistralai/mistral-medium-3-5:none PASS 100% 105 in → 2 out (0 reasoning)
none
mistralai/mistral-medium-3-5:xhigh PASS 100% 107 in → 1,533 out (1,728 reasoning)
none
moonshotai/kimi-k2.6:high PASS 100% 90 in → 362 out (359 reasoning)
none
moonshotai/kimi-k2.6:low PASS 100% 90 in → 218 out (219 reasoning)
none
moonshotai/kimi-k2.6:medium PASS 100% 92 in → 384 out (381 reasoning)
none
moonshotai/kimi-k2.6:xhigh PASS 100% 90 in → 112 out (110 reasoning)
none
moonshotai/kimi-k2.7-code:high PASS 100% 92 in → 112 out (109 reasoning)
none
moonshotai/kimi-k2.7-code:low PASS 100% 90 in → 223 out (220 reasoning)
none
moonshotai/kimi-k2.7-code:medium PASS 100% 92 in → 106 out (110 reasoning)
none
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 93 in → 162 out (148 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 89 in → 229 out (226 reasoning)
none
moonshotai/kimi-k3:high PASS 100% 169 in → 178 out (162 reasoning)
none
moonshotai/kimi-k3:high__json_schema PASS 100% 170 in → 134 out (118 reasoning)
none
moonshotai/kimi-k3:low PASS 100% 169 in → 134 out (118 reasoning)
none
moonshotai/kimi-k3:low__json_schema PASS 100% 170 in → 134 out (118 reasoning)
none
moonshotai/kimi-k3:max PASS 100% 167 in → 142 out (126 reasoning)
none
moonshotai/kimi-k3:max__json_schema PASS 100% 170 in → 140 out (124 reasoning)
none
moonshotai/kimi-k3:medium PASS 100% 166 in → 117 out (101 reasoning)
none
moonshotai/kimi-k3:medium__json_schema PASS 100% 172 in → 128 out (112 reasoning)
none
moonshotai/kimi-k3:xhigh PASS 100% 171 in → 71 out (55 reasoning)
none
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 170 in → 174 out (158 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 107 in → 120 out (129 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 108 in → 296 out (318 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 106 in → 192 out (215 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 109 in → 253 out (249 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 108 in → 237 out (273 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 109 in → 2 out (0 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 111 in → 839 out (972 reasoning)
none
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 109 in → 175 out (176 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 108 in → 102 out (94 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 105 in → 123 out (113 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 108 in → 117 out (112 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 103 in → 2 out (0 reasoning)
none
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 109 in → 48 out (50 reasoning)
none
openai/gpt-5.4-mini:high PASS 100% 91 in → 104 out (97 reasoning)
none
openai/gpt-5.4-mini:low PASS 100% 89 in → 45 out (38 reasoning)
none
openai/gpt-5.4-mini:low__json_schema PASS 100% 130 in → 72 out (54 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.4-mini:medium PASS 100% 91 in → 109 out (102 reasoning)
none
openai/gpt-5.4-mini:none PASS 100% 89 in → 5 out (0 reasoning)
none
openai/gpt-5.4-mini:xhigh PASS 100% 89 in → 425 out (418 reasoning)
none
openai/gpt-5.4-nano:high PASS 100% 86 in → 68 out (61 reasoning)
none
openai/gpt-5.4-nano:low PASS 100% 89 in → 23 out (16 reasoning)
none
openai/gpt-5.4-nano:medium PASS 100% 88 in → 36 out (29 reasoning)
none
openai/gpt-5.4-nano:none PASS 100% 87 in → 5 out (0 reasoning)
none
openai/gpt-5.4-nano:xhigh PASS 100% 87 in → 167 out (160 reasoning)
none
openai/gpt-5.4:high PASS 100% 89 in → 49 out (42 reasoning)
none
openai/gpt-5.4:high__json_schema PASS 100% 133 in → 85 out (67 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.4:low PASS 100% 91 in → 41 out (34 reasoning)
none
openai/gpt-5.4:medium PASS 100% 87 in → 59 out (52 reasoning)
none
openai/gpt-5.4:none PASS 100% 87 in → 5 out (0 reasoning)
none
openai/gpt-5.4:xhigh PASS 100% 92 in → 183 out (176 reasoning)
none
openai/gpt-5.5:high PASS 100% 88 in → 78 out (71 reasoning)
none
openai/gpt-5.5:low PASS 100% 90 in → 33 out (26 reasoning)
none
openai/gpt-5.5:medium PASS 100% 89 in → 52 out (45 reasoning)
none
openai/gpt-5.5:none PASS 100% 89 in → 5 out (0 reasoning)
none
openai/gpt-5.5:xhigh PASS 100% 90 in → 83 out (76 reasoning)
none
openai/gpt-5.6-luna:high PASS 100% 88 in → 62 out (55 reasoning)
none
openai/gpt-5.6-luna:low PASS 100% 89 in → 35 out (28 reasoning)
none
openai/gpt-5.6-luna:low__json_schema PASS 100% 131 in → 68 out (50 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.6-luna:max PASS 100% 90 in → 88 out (81 reasoning)
none
openai/gpt-5.6-luna:max__json_schema PASS 100% 131 in → 89 out (71 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.6-luna:medium PASS 100% 87 in → 41 out (34 reasoning)
none
openai/gpt-5.6-luna:xhigh PASS 100% 89 in → 88 out (81 reasoning)
none
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 132 in → 83 out (65 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete":["none"]}
openai/gpt-5.6-sol:high PASS 100% 89 in → 31 out (24 reasoning)
none
openai/gpt-5.6-sol:low PASS 100% 90 in → 31 out (24 reasoning)
none
openai/gpt-5.6-sol:max PASS 100% 87 in → 41 out (34 reasoning)
none
openai/gpt-5.6-sol:medium PASS 100% 86 in → 28 out (21 reasoning)
none
openai/gpt-5.6-sol:none PASS 100% 87 in → 5 out (0 reasoning)
none
openai/gpt-5.6-sol:xhigh PASS 100% 88 in → 33 out (26 reasoning)
none
openai/gpt-5.6-terra:high PASS 100% 89 in → 45 out (38 reasoning)
none
openai/gpt-5.6-terra:low PASS 100% 89 in → 44 out (37 reasoning)
none
openai/gpt-5.6-terra:max PASS 100% 87 in → 49 out (42 reasoning)
none
openai/gpt-5.6-terra:medium PASS 100% 92 in → 39 out (32 reasoning)
none
openai/gpt-5.6-terra:none PASS 100% 89 in → 5 out (0 reasoning)
none
openai/gpt-5.6-terra:xhigh PASS 100% 88 in → 38 out (31 reasoning)
none
openai/gpt-oss-120b:high PASS 100% 150 in → 339 out (380 reasoning)
none
openai/gpt-oss-120b:low PASS 100% 150 in → 53 out (47 reasoning)
none
openai/gpt-oss-120b:medium PASS 100% 150 in → 86 out (85 reasoning)
none
openai/gpt-oss-120b:xhigh PASS 100% 150 in → 85 out (84 reasoning)
none
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 149 in → 273 out (293 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
openai/gpt-oss-20b:high PASS 100% 136 in → 393 out (430 reasoning)
none
openai/gpt-oss-20b:low PASS 100% 148 in → 45 out (39 reasoning)
none
openai/gpt-oss-20b:medium PASS 100% 147 in → 249 out (261 reasoning)
none
openai/gpt-oss-20b:xhigh PASS 100% 140 in → 425 out (480 reasoning)
none
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 153 in → 330 out (363 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
poolside/laguna-m.1:high PASS 100% 106 in → 435 out (431 reasoning)
none
poolside/laguna-m.1:high__json_schema PASS 100% 102 in → 913 out (899 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
poolside/laguna-m.1:low PASS 100% 106 in → 496 out (492 reasoning)
none
poolside/laguna-m.1:low__json_schema PASS 100% 102 in → 768 out (758 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-m.1:medium PASS 100% 104 in → 354 out (350 reasoning)
none
poolside/laguna-m.1:medium__json_schema PASS 100% 102 in → 835 out (825 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-m.1:xhigh PASS 100% 105 in → 891 out (887 reasoning)
none
poolside/laguna-xs-2.1:high PASS 100% 104 in → 443 out (441 reasoning)
none
poolside/laguna-xs-2.1:high__json_schema PASS 100% 101 in → 222 out (212 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs-2.1:low PASS 100% 102 in → 371 out (369 reasoning)
none
poolside/laguna-xs-2.1:medium PASS 100% 103 in → 308 out (306 reasoning)
none
poolside/laguna-xs-2.1:xhigh PASS 100% 105 in → 226 out (224 reasoning)
none
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 105 in → 357 out (347 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs.2:high PASS 100% 142 in → 219 out (215 reasoning)
none
poolside/laguna-xs.2:low PASS 100% 142 in → 231 out (227 reasoning)
none
poolside/laguna-xs.2:low__json_schema PASS 100% 142 in → 324 out (314 reasoning)
none
JSON Schema Structured Output
(raw) {"branches_to_delete": ["none"]}
poolside/laguna-xs.2:medium PASS 100% 143 in → 397 out (393 reasoning)
none
poolside/laguna-xs.2:none PASS 100% 141 in → 3 out (0 reasoning)
none
poolside/laguna-xs.2:xhigh PASS 100% 141 in → 346 out (342 reasoning)
none
qwen/qwen3.6-27b:high PASS 100% 103 in → 837 out (776 reasoning)
none
qwen/qwen3.6-27b:low PASS 100% 100 in → 626 out (598 reasoning)
none
qwen/qwen3.6-27b:medium PASS 100% 105 in → 740 out (721 reasoning)
none
qwen/qwen3.6-27b:none PASS 100% 102 in → 2 out (0 reasoning)
none
qwen/qwen3.6-27b:xhigh PASS 100% 103 in → 769 out (735 reasoning)
none
qwen/qwen3.6-35b-a3b:high PASS 100% 103 in → 1,012 out (1,007 reasoning)
none
qwen/qwen3.6-35b-a3b:low PASS 100% 103 in → 1,037 out (1,023 reasoning)
none
qwen/qwen3.6-35b-a3b:medium PASS 100% 100 in → 1,319 out (1,326 reasoning)
none
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 104 in → 815 out (821 reasoning)
none
qwen/qwen3.6-flash:high PASS 100% 104 in → 1,067 out (1,061 reasoning)
none
qwen/qwen3.6-flash:low PASS 100% 103 in → 2,056 out (2,050 reasoning)
none
qwen/qwen3.6-flash:medium PASS 100% 102 in → 3,037 out (3,031 reasoning)
none
qwen/qwen3.6-flash:xhigh PASS 100% 106 in → 1,172 out (1,166 reasoning)
none
qwen/qwen3.7-flash:high PASS 100% 100 in → 892 out (886 reasoning)
none
qwen/qwen3.7-flash:low PASS 100% 105 in → 841 out (835 reasoning)
none
qwen/qwen3.7-flash:low__json_schema PASS 100% 104 in → 925 out (920 reasoning)
none
qwen/qwen3.7-flash:max PASS 100% 103 in → 864 out (859 reasoning)
none
qwen/qwen3.7-flash:medium PASS 100% 105 in → 925 out (920 reasoning)
none
qwen/qwen3.7-flash:xhigh PASS 100% 103 in → 1,044 out (1,039 reasoning)
none
qwen/qwen3.7-max:high PASS 100% 104 in → 194 out (188 reasoning)
none
qwen/qwen3.7-max:low PASS 100% 102 in → 102 out (96 reasoning)
none
qwen/qwen3.7-max:medium PASS 100% 98 in → 155 out (149 reasoning)
none
qwen/qwen3.7-max:none PASS 100% 105 in → 1 out (0 reasoning)
none
qwen/qwen3.7-max:xhigh PASS 100% 100 in → 590 out (584 reasoning)
none
qwen/qwen3.7-plus:high PASS 100% 104 in → 320 out (315 reasoning)
none
qwen/qwen3.7-plus:low PASS 100% 103 in → 339 out (334 reasoning)
none
qwen/qwen3.7-plus:medium PASS 100% 102 in → 436 out (430 reasoning)
none
qwen/qwen3.7-plus:none PASS 100% 103 in → 1 out (0 reasoning)
none
qwen/qwen3.7-plus:xhigh PASS 100% 103 in → 506 out (500 reasoning)
none
tencent/hy3:high PASS 100% 98 in → 328 out (325 reasoning)
none
tencent/hy3:high__json_schema PASS 100% 100 in → 262 out (250 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
tencent/hy3:low PASS 100% 96 in → 309 out (306 reasoning)
none
tencent/hy3:low__json_schema PASS 100% 97 in → 225 out (222 reasoning)
none
tencent/hy3:medium PASS 100% 95 in → 280 out (277 reasoning)
none
tencent/hy3:medium__json_schema PASS 100% 98 in → 271 out (275 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
tencent/hy3:none PASS 100% 98 in → 2 out (0 reasoning)
none
tencent/hy3:xhigh PASS 100% 100 in → 460 out (489 reasoning)
none
tencent/hy3:xhigh__json_schema PASS 100% 95 in → 443 out (475 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
thinkingmachines/inkling-small:high PASS 100% 96 in → 148 out (141 reasoning)
none
thinkingmachines/inkling-small:low PASS 100% 99 in → 54 out (47 reasoning)
none
thinkingmachines/inkling-small:max PASS 100% 98 in → 248 out (264 reasoning)
none
thinkingmachines/inkling-small:medium PASS 100% 98 in → 140 out (133 reasoning)
none
thinkingmachines/inkling-small:medium__json_schema PASS 100% 95 in → 70 out (50 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 101 in → 131 out (124 reasoning)
none
thinkingmachines/inkling:high PASS 100% 99 in → 450 out (443 reasoning)
none
thinkingmachines/inkling:low PASS 100% 96 in → 38 out (39 reasoning)
none
thinkingmachines/inkling:max PASS 100% 96 in → 364 out (357 reasoning)
none
thinkingmachines/inkling:medium PASS 100% 99 in → 151 out (144 reasoning)
none
thinkingmachines/inkling:none PASS 100% 95 in → 5 out (0 reasoning)
none
thinkingmachines/inkling:xhigh PASS 100% 100 in → 175 out (168 reasoning)
none
thinkingmachines/inkling:xhigh__json_schema PASS 100% 95 in → 493 out (473 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "none" ] }
x-ai/grok-4.3:high PASS 100% 265 in → 164 out (163 reasoning)
none
x-ai/grok-4.3:low PASS 100% 272 in → 231 out (230 reasoning)
none
x-ai/grok-4.3:max PASS 100% 267 in → 398 out (397 reasoning)
none
x-ai/grok-4.3:medium PASS 100% 273 in → 363 out (362 reasoning)
none
x-ai/grok-4.3:none PASS 100% 268 in → 1 out (0 reasoning)
none
x-ai/grok-4.3:xhigh PASS 100% 268 in → 180 out (179 reasoning)
none
x-ai/grok-4.5:high PASS 100% 292 in → 443 out (442 reasoning)
none
x-ai/grok-4.5:low PASS 100% 291 in → 240 out (239 reasoning)
none
x-ai/grok-4.5:max PASS 100% 292 in → 199 out (198 reasoning)
none
x-ai/grok-4.5:medium PASS 100% 293 in → 188 out (187 reasoning)
none
x-ai/grok-4.5:xhigh PASS 100% 291 in → 404 out (403 reasoning)
none
z-ai/glm-4.7-flash:high PASS 100% 92 in → 511 out (526 reasoning)
none
z-ai/glm-4.7-flash:high__json_schema PASS 100% 92 in → 448 out (425 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
z-ai/glm-4.7-flash:low PASS 100% 94 in → 418 out (401 reasoning)
none
z-ai/glm-4.7-flash:low__json_schema PASS 100% 91 in → 516 out (500 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
z-ai/glm-4.7-flash:medium PASS 100% 90 in → 1,438 out (1,438 reasoning)
none
z-ai/glm-4.7-flash:xhigh PASS 100% 90 in → 520 out (517 reasoning)
none
z-ai/glm-5.2:high PASS 100% 101 in → 132 out (131 reasoning)
none
z-ai/glm-5.2:low PASS 100% 97 in → 272 out (269 reasoning)
none
z-ai/glm-5.2:low__json_schema PASS 100% 98 in → 186 out (188 reasoning)
none
JSON Schema Structured Output
(raw) { "branches_to_delete": ["none"] }
z-ai/glm-5.2:medium PASS 100% 98 in → 215 out (212 reasoning)
none
z-ai/glm-5.2:none PASS 100% 92 in → 2 out (0 reasoning)
none
z-ai/glm-5.2:xhigh PASS 100% 100 in → 339 out (335 reasoning)
none
anthropic/claude-fable-5:high__json_schema FAIL 0% 365 in → 32 out (21 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:low__json_schema FAIL 0% 365 in → 25 out (12 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:max__json_schema FAIL 0% 365 in → 240 out (79 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:medium__json_schema FAIL 0% 367 in → 31 out (20 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-fable-5:xhigh__json_schema FAIL 0% 366 in → 51 out (40 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:high__json_schema FAIL 0% 318 in → 365 out (376 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:low__json_schema FAIL 0% 319 in → 445 out (456 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:medium__json_schema FAIL 0% 316 in → 566 out (541 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 287 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-haiku-4.5:xhigh__json_schema FAIL 0% 316 in → 413 out (407 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 287 in → 82 out (57 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:low__json_schema FAIL 0% 288 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:max__json_schema FAIL 0% 290 in → 101 out (81 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:medium__json_schema FAIL 0% 286 in → 42 out (16 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:none__json_schema FAIL 0% 290 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 288 in → 82 out (57 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:high__json_schema FAIL 0% 371 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:low__json_schema FAIL 0% 365 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:max__json_schema FAIL 0% 371 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:medium__json_schema FAIL 0% 368 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.7:none__json_schema FAIL 0% 369 in → 17 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:high__json_schema FAIL 0% 360 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:low__json_schema FAIL 0% 364 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 364 in → 317 out (49 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:medium__json_schema FAIL 0% 366 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:none__json_schema FAIL 0% 364 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-opus-4.8:xhigh__json_schema FAIL 0% 361 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:high__json_schema FAIL 0% 287 in → 89 out (64 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:low__json_schema FAIL 0% 290 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:max__json_schema FAIL 0% 288 in → 161 out (142 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:medium__json_schema FAIL 0% 289 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:none__json_schema FAIL 0% 290 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-4.6:xhigh__json_schema FAIL 0% 288 in → 86 out (62 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:medium FAIL 0% 124 in → 11 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 367 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 363 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 366 in → 80 out (42 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash-0731:max__json_schema FAIL 0% 169 in → 18 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-a", "feature-b"] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
deepseek/deepseek-v4-flash-0731:medium__json_schema FAIL 0% 244 in → 922 out (1,015 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 86 in → 6 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 89 in → 18 out (0 reasoning)
feature-a
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a" ] }
Failure: Missing: ['none'] Extra: ['feature-a']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:high__json_schema FAIL 0% 93 in → 62 out (43 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-a", "feature-b"]}
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
deepseek/deepseek-v4-flash:low__json_schema FAIL 0% 95 in → 96 out (85 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-flash:medium__json_schema FAIL 0% 94 in → 395 out (371 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-b" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
deepseek/deepseek-v4-flash:none FAIL 0% 91 in → 6 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 91 in → 16 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-a", "feature-b"] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
deepseek/deepseek-v4-pro:high FAIL 0% 92 in → 3 out (0 reasoning)
feature-a
Failure: Missing: ['none'] Extra: ['feature-a']
deepseek/deepseek-v4-pro:high__json_schema FAIL 0% 95 in → 357 out (345 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 91 in → 352 out (333 reasoning)
feature-a
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a" ] }
Failure: Missing: ['none'] Extra: ['feature-a']
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 289 in → 274 out (264 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
deepseek/deepseek-v4-pro:none FAIL 0% 93 in → 5 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 289 in → 15 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-a","feature-b"]}
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 170 in → 298 out (286 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3-flash-preview:high__json_schema FAIL 0% 91 in → 542 out (534 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3-flash-preview:low__json_schema FAIL 0% 174 in → 389 out (380 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3-flash-preview:medium__json_schema FAIL 0% 172 in → 280 out (271 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3-flash-preview:xhigh__json_schema FAIL 0% 170 in → 256 out (248 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:high__json_schema FAIL 0% 90 in → 1,065 out (1,056 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 0% 173 in → 140 out (127 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 92 in → 257 out (244 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.1-flash-lite-preview:xhigh__json_schema FAIL 0% 92 in → 996 out (987 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:high__json_schema FAIL 0% 172 in → 333 out (325 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 173 in → 367 out (359 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:medium__json_schema FAIL 0% 173 in → 437 out (429 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.1-pro-preview:xhigh__json_schema FAIL 0% 175 in → 291 out (282 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:high__json_schema FAIL 0% 92 in → 625 out (617 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 173 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:max__json_schema FAIL 0% 93 in → 544 out (531 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:medium__json_schema FAIL 0% 93 in → 479 out (466 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.5-flash-lite:xhigh__json_schema FAIL 0% 94 in → 688 out (680 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash:high__json_schema FAIL 0% 175 in → 275 out (267 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash:low__json_schema FAIL 0% 172 in → 201 out (193 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash:medium__json_schema FAIL 0% 169 in → 299 out (291 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.5-flash:xhigh__json_schema FAIL 0% 175 in → 424 out (416 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:high__json_schema FAIL 0% 173 in → 391 out (383 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:low__json_schema FAIL 0% 175 in → 171 out (158 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemini-3.6-flash:max__json_schema FAIL 0% 174 in → 438 out (430 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:medium__json_schema FAIL 0% 93 in → 282 out (274 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemini-3.6-flash:xhigh__json_schema FAIL 0% 171 in → 634 out (626 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemma-4-26b-a4b-it:high__json_schema FAIL 0% 106 in → 425 out (365 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemma-4-26b-a4b-it:low__json_schema FAIL 0% 108 in → 275 out (215 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemma-4-26b-a4b-it:none__json_schema FAIL 0% 105 in → 13 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemma-4-31b-it:high__json_schema FAIL 0% 107 in → 222 out (178 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
google/gemma-4-31b-it:low__json_schema FAIL 0% 111 in → 226 out (217 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
google/gemma-4-31b-it:none__json_schema FAIL 0% 106 in → 18 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-a", "feature-b"]}
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
ibm-granite/granite-4.1-8b FAIL 0% 101 in → 6 out
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-b" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
inclusionai/ling-2.6-flash FAIL 0% 115 in → 6 out
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-b" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
liquid/lfm-2-24b-a2b FAIL 0% 98 in → 6 out
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
Invalid JSON. Output: feature-a feature-b
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:high__json_schema FAIL 0% 125 in → 539 out (586 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.5:low__json_schema FAIL 0% 124 in → 514 out (564 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.5:medium__json_schema FAIL 0% 120 in → 541 out (537 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.5:xhigh__json_schema FAIL 0% 121 in → 896 out (883 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.7:high__json_schema FAIL 0% 273 in → 585 out (569 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.7:low__json_schema FAIL 0% 120 in → 547 out (605 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
minimax/minimax-m2.7:medium FAIL 0% 105 in → 438 out (558 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
minimax/minimax-m2.7:medium__json_schema FAIL 0% 277 in → 1,163 out (1,151 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m2.7:xhigh__json_schema FAIL 0% 274 in → 829 out (817 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m3:high__json_schema FAIL 0% 251 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m3:low__json_schema FAIL 0% 248 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m3:medium__json_schema FAIL 0% 248 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
minimax/minimax-m3:xhigh__json_schema FAIL 0% 248 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
feature-a feature-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-a", "feature-b"]}
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
mistralai/mistral-medium-3-5:high__json_schema FAIL 0% 96 in → 1,112 out (1,229 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:low__json_schema FAIL 0% 94 in → 196 out (200 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:medium__json_schema FAIL 0% 97 in → 165 out (162 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:none__json_schema FAIL 0% 95 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
mistralai/mistral-medium-3-5:xhigh__json_schema FAIL 0% 98 in → 146 out (144 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
moonshotai/kimi-k2.6:high__json_schema FAIL 0% 90 in → 23 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-b" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
moonshotai/kimi-k2.6:low__json_schema FAIL 0% 282 in → 975 out (966 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
moonshotai/kimi-k2.6:medium__json_schema FAIL 0% 283 in → 1,137 out (1,127 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0% 93 in → 775 out (533 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": []}
Failure: Missing: ['none']
moonshotai/kimi-k2.7-code:high__json_schema FAIL 0% 182 in → 148 out (138 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
moonshotai/kimi-k2.7-code:low__json_schema FAIL 0% 185 in → 187 out (177 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema FAIL 0% 105 in → 152 out (151 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 110 in → 15 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-a","feature-b"]}
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
nvidia/nemotron-3-super-120b-a12b:high__json_schema FAIL 0% 109 in → 190 out (168 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:low__json_schema FAIL 0% 108 in → 116 out (100 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:medium__json_schema FAIL 0% 106 in → 67 out (51 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:none__json_schema FAIL 0% 108 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema FAIL 0% 106 in → 135 out (116 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
openai/gpt-5.4-mini:high__json_schema FAIL 0% 131 in → 407 out (391 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-mini:medium__json_schema FAIL 0% 130 in → 368 out (352 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[] }
Failure: Missing: ['none']
openai/gpt-5.4-mini:none__json_schema FAIL 0% 133 in → 20 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-a","feature-b"]}
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
openai/gpt-5.4-mini:xhigh__json_schema FAIL 0% 130 in → 532 out (516 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-nano:high__json_schema FAIL 0% 132 in → 140 out (124 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-nano:low__json_schema FAIL 0% 129 in → 40 out (24 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-nano:medium__json_schema FAIL 0% 133 in → 91 out (75 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4-nano:none__json_schema FAIL 0% 131 in → 20 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-a","feature-b"]}
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
openai/gpt-5.4-nano:xhigh__json_schema FAIL 0% 129 in → 665 out (649 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:low__json_schema FAIL 0% 131 in → 42 out (26 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:medium__json_schema FAIL 0% 132 in → 93 out (77 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:none__json_schema FAIL 0% 133 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.4:xhigh__json_schema FAIL 0% 130 in → 200 out (184 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:high__json_schema FAIL 0% 131 in → 74 out (58 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:low__json_schema FAIL 0% 135 in → 54 out (38 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:medium__json_schema FAIL 0% 132 in → 76 out (60 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:none__json_schema FAIL 0% 129 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.5:xhigh__json_schema FAIL 0% 134 in → 79 out (63 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-luna:high__json_schema FAIL 0% 132 in → 142 out (126 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-luna:medium__json_schema FAIL 0% 132 in → 194 out (178 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-luna:none FAIL 0% 86 in → 9 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
openai/gpt-5.6-luna:none__json_schema FAIL 0% 131 in → 20 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-a","feature-b"]}
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
openai/gpt-5.6-sol:high__json_schema FAIL 0% 131 in → 100 out (84 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:low__json_schema FAIL 0% 129 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:max__json_schema FAIL 0% 131 in → 165 out (149 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:medium__json_schema FAIL 0% 131 in → 61 out (45 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:none__json_schema FAIL 0% 134 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-sol:xhigh__json_schema FAIL 0% 131 in → 103 out (87 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:high__json_schema FAIL 0% 133 in → 43 out (27 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:low__json_schema FAIL 0% 134 in → 42 out (26 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:max__json_schema FAIL 0% 133 in → 59 out (43 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:medium__json_schema FAIL 0% 132 in → 35 out (19 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:none__json_schema FAIL 0% 131 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-5.6-terra:xhigh__json_schema FAIL 0% 131 in → 61 out (45 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
openai/gpt-oss-120b:high__json_schema FAIL 0% 149 in → 23 out (0 reasoning)
feature-a feature-git
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-git" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-git']
openai/gpt-oss-120b:low__json_schema FAIL 0% 148 in → 21 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-b" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
openai/gpt-oss-120b:medium__json_schema FAIL 0% 137 in → 112 out (97 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
openai/gpt-oss-20b:high__json_schema FAIL 0% 147 in → 16 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-a", "feature-b"]}
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
openai/gpt-oss-20b:low__json_schema FAIL 0% 153 in → 51 out (34 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
openai/gpt-oss-20b:medium__json_schema FAIL 0% 258 in → 260 out (241 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
poolside/laguna-m.1:none FAIL 0% 107 in → 7 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
poolside/laguna-m.1:none__json_schema FAIL 0% 104 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
poolside/laguna-m.1:xhigh__json_schema FAIL 0% 102 in → 895 out (886 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs-2.1:low__json_schema FAIL 0% 104 in → 314 out (301 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
poolside/laguna-xs-2.1:medium__json_schema FAIL 0% 104 in → 338 out (329 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs-2.1:none FAIL 0% 101 in → 5 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 105 in → 28 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-b" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
poolside/laguna-xs.2:high__json_schema FAIL 0% 139 in → 172 out (163 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs.2:medium__json_schema FAIL 0% 139 in → 345 out (336 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
poolside/laguna-xs.2:none__json_schema FAIL 0% 141 in → 14 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-a", "feature-b"]}
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
poolside/laguna-xs.2:xhigh__json_schema FAIL 0% 144 in → 450 out (437 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-27b:high__json_schema FAIL 0% 103 in → 643 out (598 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-27b:low__json_schema FAIL 0% 102 in → 797 out (738 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-27b:medium__json_schema FAIL 0% 105 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 106 in → 27 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-b" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
qwen/qwen3.6-27b:xhigh__json_schema FAIL 0% 103 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-35b-a3b:high__json_schema FAIL 0% 104 in → 848 out (793 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-35b-a3b:low__json_schema FAIL 0% 104 in → 1,359 out (1,306 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-35b-a3b:medium__json_schema FAIL 0% 98 in → 996 out (956 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
qwen/qwen3.6-35b-a3b:none FAIL 0% 105 in → 6 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 103 in → 28 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-b" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
qwen/qwen3.6-35b-a3b:xhigh__json_schema FAIL 0% 100 in → 505 out (508 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ae5fe66c-52af-9689-87d6-218e830e2ef2","request_id":"ae5fe66c-52af-9689-87d6-218e830e2ef2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3451e98b-1ecc-9a5c-906f-186bf5d030df","request_id":"3451e98b-1ecc-9a5c-906f-186bf5d030df"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-343a3f67-e0b0-9ce4-a0f6-3b0284eec4c6","request_id":"343a3f67-e0b0-9ce4-a0f6-3b0284eec4c6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none FAIL 0% 103 in → 5 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2360e788-2bb9-90ff-ab1e-1b864e945ff6","request_id":"2360e788-2bb9-90ff-ab1e-1b864e945ff6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e1078c3d-5584-9f00-9768-faaa24f2469a","request_id":"e1078c3d-5584-9f00-9768-faaa24f2469a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-aba00f18-2617-94a1-8fb1-724e8fbf393c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a65c9393-4877-9d77-956b-4f41444b5b3e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-38f9f674-a034-9297-ae71-ce2e85831d63"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none FAIL 0% 105 in → 5 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-66b4aa2a-799f-99ea-83e8-76e5d14a6d0f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-cf349e08-a111-9496-947a-84620b452633"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b7914fb3-0385-9367-86e2-68ef769436fe","request_id":"b7914fb3-0385-9367-86e2-68ef769436fe"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-50b9294b-c8e8-9fdd-a791-6f15657ca496","request_id":"50b9294b-c8e8-9fdd-a791-6f15657ca496"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-98b3bfe8-ec52-9dca-82f9-331acfbcbcb0","request_id":"98b3bfe8-ec52-9dca-82f9-331acfbcbcb0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c035860d-f307-972b-8797-57c02c80bd4a","request_id":"c035860d-f307-972b-8797-57c02c80bd4a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4882ba8f-22b5-9928-8327-dc23940a33e8","request_id":"4882ba8f-22b5-9928-8327-dc23940a33e8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d19da0f5-34cd-93f8-9517-37a0ce5bcc1a","request_id":"d19da0f5-34cd-93f8-9517-37a0ce5bcc1a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3147ebd1-6238-9911-b56d-e1c5986d32f3","request_id":"3147ebd1-6238-9911-b56d-e1c5986d32f3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d8a0d1cb-4f85-941f-aa85-5ef329cfb1b9","request_id":"d8a0d1cb-4f85-941f-aa85-5ef329cfb1b9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-740750b2-6746-96ac-8aa1-132ea4c23715","request_id":"740750b2-6746-96ac-8aa1-132ea4c23715"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c239a2c3-7705-9374-91ee-38a494361db2","request_id":"c239a2c3-7705-9374-91ee-38a494361db2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:none__json_schema FAIL 0% 101 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
thinkingmachines/inkling-small:high__json_schema FAIL 0% 98 in → 254 out (240 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
thinkingmachines/inkling-small:low__json_schema FAIL 0% 96 in → 47 out (33 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
thinkingmachines/inkling-small:max__json_schema FAIL 0% 98 in → 193 out (179 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
thinkingmachines/inkling-small:none FAIL 0% 95 in → 8 out (0 reasoning)
feature-a feature-b<|end_message|>
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b<|end_message|>']
Invalid JSON. Output: feature-a feature-b{ "branches_to_delete": ["feature-a", "feature-b"] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling-small:xhigh__json_schema FAIL 0% 94 in → 150 out (136 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
Invalid JSON. Output: none
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: none
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:max__json_schema FAIL 0% 98 in → 387 out (375 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
Invalid JSON. Output: none
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0% 96 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
x-ai/grok-4.3:high__json_schema FAIL 0% 332 in → 534 out (528 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:low__json_schema FAIL 0% 340 in → 298 out (291 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete": []}
Failure: Missing: ['none']
x-ai/grok-4.3:max__json_schema FAIL 0% 333 in → 452 out (446 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:medium__json_schema FAIL 0% 338 in → 539 out (530 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
x-ai/grok-4.3:none__json_schema FAIL 0% 334 in → 6 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.3:xhigh__json_schema FAIL 0% 331 in → 772 out (766 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:high__json_schema FAIL 0% 375 in → 365 out (357 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:low__json_schema FAIL 0% 373 in → 320 out (312 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:max__json_schema FAIL 0% 372 in → 512 out (504 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:medium__json_schema FAIL 0% 373 in → 339 out (331 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
x-ai/grok-4.5:xhigh__json_schema FAIL 0% 374 in → 367 out (359 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
z-ai/glm-4.7-flash:medium__json_schema FAIL 0% 91 in → 505 out (486 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
z-ai/glm-4.7-flash:none FAIL 0% 93 in → 5 out (0 reasoning)
feature-a feature-b
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 91 in → 22 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-b" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0% 88 in → 2,256 out (2,296 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['none']
z-ai/glm-5.2:high__json_schema FAIL 0% 284 in → 402 out (394 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"branches_to_delete":[]}
Failure: Missing: ['none']
Invalid JSON. Output: none
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:none__json_schema FAIL 0% 94 in → 23 out (0 reasoning)
feature-a feature-b
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-a", "feature-b" ] }
Failure: Missing: ['none'] Extra: ['feature-a', 'feature-b']
Invalid JSON. Output: { "branches_to_delete": [ "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If one or more, list the branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` feature-a", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-a` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature-a` and `feature-b` are visible in `main`.", "Now list those branch names, one per line. If none, respond with 'none'.", "All commits from `feature
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Unterminated string starting at: line 58 column 4 (char 4260)
Failure: Failed to parse structured JSON response: Unterminated string starting at: line 58 column 4 (char 4260)