Branches off branches
Tests ability to handle nested branch topology (branches off branches). Evaluates understanding of merge status when branches fork from non-main branches.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'v1' > app.py
  5. 05 git add app.py
  6. 06 git commit -m 'Initial'
  7. 07 git checkout -b feature-base
  8. 08 echo 'base feature' > base.py
  9. 09 git add base.py
  10. 10 git commit -m 'Base feature'
  11. 11 git checkout -b feature-sub
  12. 12 echo 'sub feature' > sub.py
  13. 13 git add sub.py
  14. 14 git commit -m 'Sub feature'
  15. 15 git checkout feature-base
  16. 16 git merge feature-sub
  17. 17 git checkout main
  18. 18 git merge feature-base
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
feature-base feature-sub

Scoped model quality, cost, API time, and token usage for branch_cleanup/f007.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high FAIL 100% 143 in → 108 out (22 reasoning)
feature-base feature-sub Both branches point to the same commit as main (`e7dd812`) and are listed as fully merged, so they can be safely deleted with `git branch -d feature-base feature-sub`. (`main` itself is never deleted, of course.)
Failure: Extra: ['Both branches point to the same commit as main (`e7dd812`) and are listed as fully merged, so they can be safely deleted with `git branch -d feature-base feature-sub`. (`main` itself is never deleted, of course.)']
anthropic/claude-fable-5:high__json_schema PASS 100% 379 in → 44 out (17 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
anthropic/claude-fable-5:low FAIL 100% 143 in → 56 out (8 reasoning)
feature-base feature-sub Both branches point to the same commit as main (bb8df14) and are fully merged, so they can be safely deleted.
Failure: Extra: ['Both branches point to the same commit as main (bb8df14) and are fully merged, so they can be safely deleted.']
anthropic/claude-fable-5:low__json_schema PASS 100% 380 in → 28 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
anthropic/claude-fable-5:max PASS 100% 141 in → 257 out (117 reasoning)
feature-base feature-sub
anthropic/claude-fable-5:max__json_schema PASS 100% 379 in → 196 out (79 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
anthropic/claude-fable-5:medium FAIL 100% 135 in → 87 out (17 reasoning)
feature-base feature-sub Both branches point to the same commit as `main` (046941b) and are listed as fully merged, so they can be safely deleted with `git branch -d feature-base feature-sub`.
Failure: Extra: ['Both branches point to the same commit as `main` (046941b) and are listed as fully merged, so they can be safely deleted with `git branch -d feature-base feature-sub`.']
anthropic/claude-fable-5:medium__json_schema PASS 100% 383 in → 28 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
anthropic/claude-fable-5:xhigh PASS 100% 141 in → 41 out (27 reasoning)
feature-base feature-sub
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 382 in → 56 out (17 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
anthropic/claude-haiku-4.5:high PASS 100% 136 in → 164 out (158 reasoning)
feature-base feature-sub
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 328 in → 450 out (446 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
anthropic/claude-haiku-4.5:low PASS 100% 141 in → 168 out (130 reasoning)
feature-base feature-sub
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 333 in → 367 out (333 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
anthropic/claude-haiku-4.5:medium PASS 100% 139 in → 173 out (157 reasoning)
feature-base feature-sub
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 328 in → 262 out (244 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
anthropic/claude-haiku-4.5:none PASS 100% 113 in → 10 out (0 reasoning)
feature-base feature-sub
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 300 in → 21 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 141 in → 131 out (128 reasoning)
feature-base feature-sub
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 325 in → 381 out (356 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.6:high PASS 100% 112 in → 54 out (32 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.6:high__json_schema PASS 100% 299 in → 49 out (16 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.6:low PASS 100% 112 in → 10 out (0 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.6:low__json_schema PASS 100% 292 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.6:max PASS 100% 107 in → 159 out (131 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.6:max__json_schema PASS 100% 302 in → 100 out (75 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.6:medium PASS 100% 113 in → 10 out (0 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 298 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.6:none PASS 100% 109 in → 10 out (0 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.6:none__json_schema PASS 100% 302 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 112 in → 54 out (32 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 302 in → 49 out (16 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.7:high PASS 100% 148 in → 14 out (0 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.7:high__json_schema PASS 100% 385 in → 28 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.7:low PASS 100% 149 in → 14 out (0 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.7:low__json_schema PASS 100% 382 in → 28 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.7:max PASS 100% 152 in → 127 out (60 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.7:max__json_schema PASS 100% 387 in → 29 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.7:medium PASS 100% 149 in → 14 out (0 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 388 in → 28 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.7:none PASS 100% 146 in → 14 out (0 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.7:none__json_schema PASS 100% 384 in → 28 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 146 in → 14 out (0 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 384 in → 28 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.8:high PASS 100% 140 in → 73 out (42 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.8:high__json_schema PASS 100% 380 in → 26 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.8:low PASS 100% 143 in → 42 out (21 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.8:low__json_schema PASS 100% 382 in → 26 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.8:max PASS 100% 140 in → 391 out (67 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.8:max__json_schema PASS 100% 385 in → 347 out (55 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.8:medium PASS 100% 144 in → 67 out (52 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 377 in → 26 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.8:none PASS 100% 141 in → 12 out (0 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.8:none__json_schema PASS 100% 383 in → 25 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 144 in → 183 out (36 reasoning)
feature-base feature-sub
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 382 in → 26 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-4.6:high PASS 100% 113 in → 103 out (83 reasoning)
feature-base feature-sub
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 296 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-4.6:low PASS 100% 113 in → 10 out (0 reasoning)
feature-base feature-sub
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 302 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-4.6:max PASS 100% 115 in → 102 out (83 reasoning)
feature-base feature-sub
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 299 in → 108 out (83 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 109 in → 10 out (0 reasoning)
feature-base feature-sub
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 301 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-4.6:none PASS 100% 107 in → 10 out (0 reasoning)
feature-base feature-sub
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 296 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 107 in → 102 out (83 reasoning)
feature-base feature-sub
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 302 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-5:high PASS 100% 140 in → 57 out (32 reasoning)
feature-base feature-sub
anthropic/claude-sonnet-5:high__json_schema PASS 100% 385 in → 26 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-5:low PASS 100% 138 in → 12 out (0 reasoning)
feature-base feature-sub
anthropic/claude-sonnet-5:low__json_schema PASS 100% 382 in → 26 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-5:medium FAIL 100% 143 in → 41 out (0 reasoning)
Based on the merge status provided, the following branches are fully merged into main and can be deleted: feature-base feature-sub
Failure: Extra: ['Based on the merge status provided, the following branches are fully merged into main and can be deleted:']
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 383 in → 26 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-5:none PASS 100% 147 in → 12 out (0 reasoning)
feature-base feature-sub
anthropic/claude-sonnet-5:none__json_schema PASS 100% 386 in → 25 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 149 in → 75 out (30 reasoning)
feature-base feature-sub
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 385 in → 55 out (23 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
arcee-ai/trinity-large-thinking:high PASS 100% 106 in → 467 out (465 reasoning)
feature-base feature-sub
arcee-ai/trinity-large-thinking:low PASS 100% 100 in → 284 out (279 reasoning)
feature-base feature-sub
arcee-ai/trinity-large-thinking:medium PASS 100% 100 in → 473 out (473 reasoning)
feature-base feature-sub
arcee-ai/trinity-large-thinking:xhigh PASS 100% 102 in → 452 out (453 reasoning)
feature-base feature-sub
arcee-ai/trinity-mini:high PASS 100% 105 in → 259 out (296 reasoning)
feature-base feature-sub
arcee-ai/trinity-mini:high__json_schema PASS 100% 100 in → 320 out (354 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
arcee-ai/trinity-mini:low PASS 100% 106 in → 273 out (317 reasoning)
feature-base feature-sub
arcee-ai/trinity-mini:low__json_schema PASS 100% 102 in → 340 out (375 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
arcee-ai/trinity-mini:medium PASS 100% 106 in → 301 out (344 reasoning)
feature-base feature-sub
arcee-ai/trinity-mini:medium__json_schema PASS 100% 103 in → 285 out (318 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 109 in → 199 out (231 reasoning)
feature-base feature-sub
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 100 in → 265 out (287 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete":[ "feature-base", "feature-sub" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 180 in → 194 out (204 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 98 in → 251 out (237 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 94 in → 36 out (30 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 97 in → 23 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 193 in → 146 out (140 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 180 in → 23 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 100 in → 52 out (46 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 103 in → 72 out (59 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 98 in → 5 out (0 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 99 in → 22 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 101 in → 54 out (78 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 100 in → 59 out (56 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
deepseek/deepseek-v4-flash:high PASS 100% 101 in → 117 out (111 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 98 in → 121 out (104 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
deepseek/deepseek-v4-flash:low PASS 100% 95 in → 128 out (119 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 101 in → 230 out (240 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
deepseek/deepseek-v4-flash:medium PASS 100% 101 in → 62 out (55 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash:none PASS 100% 104 in → 6 out (0 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 97 in → 23 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
deepseek/deepseek-v4-flash:xhigh PASS 100% 174 in → 274 out (322 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 181 in → 321 out (297 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
deepseek/deepseek-v4-pro:high PASS 100% 103 in → 278 out (271 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 300 in → 125 out (108 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
deepseek/deepseek-v4-pro:low PASS 100% 100 in → 310 out (304 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 300 in → 77 out (60 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
deepseek/deepseek-v4-pro:medium PASS 100% 103 in → 161 out (154 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 100 in → 250 out (226 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
deepseek/deepseek-v4-pro:none PASS 100% 101 in → 6 out (0 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 103 in → 22 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 103 in → 312 out (305 reasoning)
feature-base feature-sub
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 375 in → 500 out (483 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
google/gemini-3-flash-preview:high PASS 100% 105 in → 290 out (283 reasoning)
feature-base feature-sub
google/gemini-3-flash-preview:high__json_schema PASS 100% 183 in → 384 out (369 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3-flash-preview:low PASS 100% 101 in → 239 out (232 reasoning)
feature-base feature-sub
google/gemini-3-flash-preview:low__json_schema PASS 100% 186 in → 363 out (333 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemini-3-flash-preview:medium PASS 100% 98 in → 250 out (243 reasoning)
feature-base feature-sub
google/gemini-3-flash-preview:medium__json_schema PASS 100% 183 in → 225 out (210 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3-flash-preview:xhigh PASS 100% 99 in → 199 out (192 reasoning)
feature-base feature-sub
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 102 in → 371 out (356 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 99 in → 363 out (356 reasoning)
feature-base feature-sub
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 179 in → 468 out (453 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 100 in → 121 out (114 reasoning)
feature-base feature-sub
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 106 in → 142 out (112 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 103 in → 181 out (174 reasoning)
feature-base feature-sub
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 182 in → 223 out (206 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 106 in → 388 out (381 reasoning)
feature-base feature-sub
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 106 in → 318 out (288 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemini-3.1-pro-preview:high PASS 100% 105 in → 206 out (199 reasoning)
feature-base feature-sub
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 183 in → 109 out (95 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.1-pro-preview:low PASS 100% 95 in → 246 out (239 reasoning)
feature-base feature-sub
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 182 in → 229 out (214 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.1-pro-preview:medium PASS 100% 101 in → 314 out (307 reasoning)
feature-base feature-sub
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 186 in → 79 out (64 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 105 in → 347 out (340 reasoning)
feature-base feature-sub
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 179 in → 480 out (465 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.5-flash-lite:high PASS 100% 106 in → 373 out (366 reasoning)
feature-base feature-sub
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 100 in → 327 out (310 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
google/gemini-3.5-flash-lite:low PASS 100% 103 in → 7 out (0 reasoning)
feature-base feature-sub
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 105 in → 30 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemini-3.5-flash-lite:max PASS 100% 105 in → 336 out (329 reasoning)
feature-base feature-sub
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 106 in → 436 out (419 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
google/gemini-3.5-flash-lite:medium PASS 100% 105 in → 197 out (190 reasoning)
feature-base feature-sub
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 106 in → 234 out (204 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemini-3.5-flash-lite:xhigh PASS 100% 99 in → 340 out (333 reasoning)
feature-base feature-sub
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 183 in → 427 out (410 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
google/gemini-3.5-flash:high PASS 100% 98 in → 445 out (438 reasoning)
feature-base feature-sub
google/gemini-3.5-flash:high__json_schema PASS 100% 182 in → 285 out (270 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.5-flash:low PASS 100% 105 in → 292 out (285 reasoning)
feature-base feature-sub
google/gemini-3.5-flash:low__json_schema PASS 100% 186 in → 185 out (170 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.5-flash:medium PASS 100% 99 in → 335 out (328 reasoning)
feature-base feature-sub
google/gemini-3.5-flash:medium__json_schema PASS 100% 180 in → 379 out (349 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemini-3.5-flash:xhigh PASS 100% 101 in → 393 out (386 reasoning)
feature-base feature-sub
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 186 in → 483 out (468 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.6-flash:high PASS 100% 106 in → 411 out (404 reasoning)
feature-base feature-sub
google/gemini-3.6-flash:high__json_schema PASS 100% 103 in → 381 out (366 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.6-flash:low PASS 100% 102 in → 120 out (113 reasoning)
feature-base feature-sub
google/gemini-3.6-flash:low__json_schema PASS 100% 105 in → 30 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemini-3.6-flash:max PASS 100% 106 in → 341 out (334 reasoning)
feature-base feature-sub
google/gemini-3.6-flash:max__json_schema PASS 100% 103 in → 469 out (454 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.6-flash:medium PASS 100% 99 in → 290 out (283 reasoning)
feature-base feature-sub
google/gemini-3.6-flash:medium__json_schema PASS 100% 100 in → 293 out (278 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemini-3.6-flash:xhigh PASS 100% 106 in → 420 out (413 reasoning)
feature-base feature-sub
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 180 in → 502 out (487 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemma-4-26b-a4b-it:high PASS 100% 120 in → 277 out (233 reasoning)
feature-base feature-sub
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 118 in → 292 out (225 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 118 in → 304 out (240 reasoning)
feature-base feature-sub
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 119 in → 283 out (211 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 117 in → 261 out (253 reasoning)
feature-base feature-sub
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 117 in → 303 out (236 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 114 in → 8 out (0 reasoning)
feature-base feature-sub
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 115 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 118 in → 309 out (258 reasoning)
feature-base feature-sub
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 114 in → 315 out (243 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemma-4-31b-it:high PASS 100% 121 in → 168 out (1 reasoning)
feature-base feature-sub
google/gemma-4-31b-it:high__json_schema PASS 100% 121 in → 206 out (154 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemma-4-31b-it:low PASS 100% 117 in → 209 out (182 reasoning)
feature-base feature-sub
google/gemma-4-31b-it:low__json_schema PASS 100% 118 in → 225 out (207 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
google/gemma-4-31b-it:medium PASS 100% 112 in → 214 out (174 reasoning)
feature-base feature-sub
google/gemma-4-31b-it:medium__json_schema PASS 100% 117 in → 239 out (172 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemma-4-31b-it:none PASS 100% 114 in → 8 out (0 reasoning)
feature-base feature-sub
google/gemma-4-31b-it:none__json_schema PASS 100% 118 in → 22 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
google/gemma-4-31b-it:xhigh PASS 100% 121 in → 294 out (1 reasoning)
feature-base feature-sub
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 117 in → 206 out (189 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
ibm-granite/granite-4.1-8b PASS 100% 111 in → 6 out
feature-base feature-sub
ibm-granite/granite-4.1-8b__json_schema PASS 100% 111 in → 24 out
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
inclusionai/ling-2.6-flash PASS 100% 124 in → 6 out
feature-base feature-sub
inclusionai/ling-2.6-flash__json_schema PASS 100% 127 in → 28 out
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
liquid/lfm-2-24b-a2b PASS 100% 109 in → 7 out
feature-base feature-sub
minimax/minimax-m2.5:high PASS 100% 129 in → 344 out (396 reasoning)
feature-base feature-sub
minimax/minimax-m2.5:high__json_schema PASS 100% 139 in → 461 out (527 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
minimax/minimax-m2.5:low PASS 100% 136 in → 165 out (158 reasoning)
feature-base feature-sub
minimax/minimax-m2.5:low__json_schema PASS 100% 129 in → 315 out (354 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
minimax/minimax-m2.5:medium PASS 100% 135 in → 147 out (140 reasoning)
feature-base feature-sub
minimax/minimax-m2.5:medium__json_schema PASS 100% 129 in → 427 out (408 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
minimax/minimax-m2.5:xhigh PASS 100% 126 in → 280 out (273 reasoning)
feature-base feature-sub
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 135 in → 433 out (479 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
minimax/minimax-m2.7:high PASS 100% 129 in → 267 out (259 reasoning)
feature-base feature-sub
minimax/minimax-m2.7:low PASS 100% 129 in → 324 out (316 reasoning)
feature-base feature-sub
minimax/minimax-m2.7:low__json_schema PASS 100% 280 in → 192 out (177 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
minimax/minimax-m2.7:medium PASS 100% 132 in → 135 out (150 reasoning)
feature-base feature-sub
minimax/minimax-m2.7:medium__json_schema PASS 100% 280 in → 652 out (637 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
minimax/minimax-m2.7:xhigh PASS 100% 137 in → 297 out (291 reasoning)
feature-base feature-sub
minimax/minimax-m3:high PASS 100% 267 in → 164 out (196 reasoning)
feature-base feature-sub
minimax/minimax-m3:low PASS 100% 267 in → 159 out (179 reasoning)
feature-base feature-sub
minimax/minimax-m3:low__json_schema PASS 100% 254 in → 16 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
minimax/minimax-m3:medium PASS 100% 273 in → 89 out (101 reasoning)
feature-base feature-sub
minimax/minimax-m3:medium__json_schema PASS 100% 257 in → 18 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
minimax/minimax-m3:xhigh PASS 100% 270 in → 129 out (143 reasoning)
feature-base feature-sub
minimax/minimax-m3:xhigh__json_schema PASS 100% 257 in → 23 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
mistralai/devstral-2512 PASS 100% 104 in → 6 out
feature-base feature-sub
mistralai/devstral-2512__json_schema PASS 100% 101 in → 16 out
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
mistralai/mistral-medium-3-5:high PASS 100% 116 in → 254 out (272 reasoning)
feature-base feature-sub
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 98 in → 216 out (226 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
mistralai/mistral-medium-3-5:low PASS 100% 113 in → 373 out (409 reasoning)
feature-base feature-sub
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 101 in → 191 out (194 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
mistralai/mistral-medium-3-5:medium PASS 100% 116 in → 254 out (264 reasoning)
feature-base feature-sub
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 104 in → 174 out (184 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
mistralai/mistral-medium-3-5:none PASS 100% 116 in → 6 out (0 reasoning)
feature-base feature-sub
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 104 in → 16 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 116 in → 209 out (209 reasoning)
feature-base feature-sub
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 107 in → 288 out (306 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
moonshotai/kimi-k2.6:high PASS 100% 97 in → 187 out (180 reasoning)
feature-base feature-sub
moonshotai/kimi-k2.6:high__json_schema PASS 100% 100 in → 284 out (292 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
moonshotai/kimi-k2.6:low PASS 100% 97 in → 285 out (278 reasoning)
feature-base feature-sub
moonshotai/kimi-k2.6:low__json_schema PASS 100% 102 in → 384 out (159 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
moonshotai/kimi-k2.6:medium PASS 100% 97 in → 270 out (268 reasoning)
feature-base feature-sub
moonshotai/kimi-k2.6:xhigh PASS 100% 94 in → 271 out (293 reasoning)
feature-base feature-sub
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 290 in → 405 out (390 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
moonshotai/kimi-k2.7-code:high PASS 100% 100 in → 105 out (98 reasoning)
feature-base feature-sub
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 97 in → 117 out (105 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
moonshotai/kimi-k2.7-code:low PASS 100% 102 in → 149 out (142 reasoning)
feature-base feature-sub
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 100 in → 202 out (211 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
moonshotai/kimi-k2.7-code:medium PASS 100% 97 in → 175 out (168 reasoning)
feature-base feature-sub
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 103 in → 184 out (191 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 103 in → 100 out (105 reasoning)
feature-base feature-sub
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 194 in → 108 out (93 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
moonshotai/kimi-k3:high PASS 100% 177 in → 172 out (152 reasoning)
feature-base feature-sub
moonshotai/kimi-k3:high__json_schema PASS 100% 177 in → 166 out (146 reasoning)
feature-base feature-sub
moonshotai/kimi-k3:low PASS 100% 174 in → 94 out (74 reasoning)
feature-base feature-sub
moonshotai/kimi-k3:low__json_schema PASS 100% 171 in → 147 out (127 reasoning)
feature-base feature-sub
moonshotai/kimi-k3:max PASS 100% 177 in → 76 out (56 reasoning)
feature-base feature-sub
moonshotai/kimi-k3:max__json_schema PASS 100% 284 in → 148 out (120 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
moonshotai/kimi-k3:medium PASS 100% 174 in → 144 out (124 reasoning)
feature-base feature-sub
moonshotai/kimi-k3:medium__json_schema PASS 100% 174 in → 141 out (121 reasoning)
feature-base feature-sub
moonshotai/kimi-k3:xhigh PASS 100% 177 in → 141 out (121 reasoning)
feature-base feature-sub
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 174 in → 114 out (94 reasoning)
feature-base feature-sub
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 111 in → 101 out (95 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 120 in → 213 out (228 reasoning)
feature-base feature-sub
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 117 in → 94 out (80 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 114 in → 100 out (106 reasoning)
feature-base feature-sub
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 117 in → 69 out (57 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 114 in → 15 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base","feature-sub"]}
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 120 in → 150 out (164 reasoning)
feature-base feature-sub
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 114 in → 221 out (222 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 117 in → 73 out (65 reasoning)
feature-base feature-sub
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 120 in → 79 out (61 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 117 in → 64 out (55 reasoning)
feature-base feature-sub
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 117 in → 88 out (78 reasoning)
feature-base feature-sub
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 117 in → 185 out (164 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 120 in → 6 out (0 reasoning)
feature-base feature-sub
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 120 in → 23 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 111 in → 136 out (146 reasoning)
feature-base feature-sub
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 117 in → 123 out (92 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
openai/gpt-5.4-mini:high PASS 100% 101 in → 78 out (67 reasoning)
feature-base feature-sub
openai/gpt-5.4-mini:high__json_schema PASS 100% 135 in → 136 out (114 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4-mini:low PASS 100% 95 in → 39 out (28 reasoning)
feature-base feature-sub
openai/gpt-5.4-mini:low__json_schema PASS 100% 144 in → 53 out (31 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4-mini:medium PASS 100% 98 in → 54 out (43 reasoning)
feature-base feature-sub
openai/gpt-5.4-mini:medium__json_schema PASS 100% 147 in → 266 out (244 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4-mini:none PASS 100% 98 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.4-mini:none__json_schema PASS 100% 144 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4-mini:xhigh PASS 100% 92 in → 73 out (62 reasoning)
feature-base feature-sub
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 141 in → 246 out (224 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4-nano:high PASS 100% 101 in → 57 out (46 reasoning)
feature-base feature-sub
openai/gpt-5.4-nano:high__json_schema PASS 100% 138 in → 45 out (23 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4-nano:low PASS 100% 92 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.4-nano:low__json_schema PASS 100% 144 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4-nano:medium PASS 100% 101 in → 40 out (29 reasoning)
feature-base feature-sub
openai/gpt-5.4-nano:medium__json_schema PASS 100% 141 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4-nano:none PASS 100% 98 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.4-nano:none__json_schema PASS 100% 144 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4-nano:xhigh PASS 100% 101 in → 99 out (88 reasoning)
feature-base feature-sub
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 135 in → 380 out (358 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4:high PASS 100% 101 in → 51 out (40 reasoning)
feature-base feature-sub
openai/gpt-5.4:high__json_schema PASS 100% 141 in → 397 out (375 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4:low PASS 100% 101 in → 56 out (45 reasoning)
feature-base feature-sub
openai/gpt-5.4:low__json_schema PASS 100% 141 in → 32 out (10 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4:medium PASS 100% 95 in → 58 out (47 reasoning)
feature-base feature-sub
openai/gpt-5.4:medium__json_schema PASS 100% 138 in → 77 out (55 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4:none PASS 100% 95 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.4:none__json_schema PASS 100% 135 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.4:xhigh PASS 100% 98 in → 90 out (79 reasoning)
feature-base feature-sub
openai/gpt-5.4:xhigh__json_schema PASS 100% 141 in → 538 out (516 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.5:high PASS 100% 95 in → 63 out (52 reasoning)
feature-base feature-sub
openai/gpt-5.5:high__json_schema PASS 100% 144 in → 177 out (155 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.5:low PASS 100% 98 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.5:low__json_schema PASS 100% 147 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.5:medium PASS 100% 101 in → 51 out (40 reasoning)
feature-base feature-sub
openai/gpt-5.5:medium__json_schema PASS 100% 144 in → 149 out (127 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.5:none PASS 100% 101 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.5:none__json_schema PASS 100% 147 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.5:xhigh PASS 100% 92 in → 87 out (76 reasoning)
feature-base feature-sub
openai/gpt-5.5:xhigh__json_schema PASS 100% 144 in → 85 out (63 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-luna:high PASS 100% 98 in → 44 out (33 reasoning)
feature-base feature-sub
openai/gpt-5.6-luna:high__json_schema PASS 100% 138 in → 44 out (22 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-luna:low PASS 100% 101 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-luna:low__json_schema PASS 100% 141 in → 45 out (23 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-luna:max PASS 100% 95 in → 47 out (36 reasoning)
feature-base feature-sub
openai/gpt-5.6-luna:max__json_schema PASS 100% 144 in → 93 out (71 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-luna:medium PASS 100% 95 in → 34 out (23 reasoning)
feature-base feature-sub
openai/gpt-5.6-luna:medium__json_schema PASS 100% 138 in → 41 out (19 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-luna:none PASS 100% 95 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-luna:none__json_schema PASS 100% 138 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-luna:xhigh PASS 100% 101 in → 41 out (30 reasoning)
feature-base feature-sub
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 141 in → 78 out (56 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-sol:high PASS 100% 104 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-sol:high__json_schema PASS 100% 138 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-sol:low PASS 100% 95 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-sol:low__json_schema PASS 100% 141 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-sol:max PASS 100% 101 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-sol:max__json_schema PASS 100% 141 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-sol:medium PASS 100% 95 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-sol:medium__json_schema PASS 100% 141 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-sol:none PASS 100% 95 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-sol:none__json_schema PASS 100% 141 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-sol:xhigh PASS 100% 98 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 138 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-terra:high PASS 100% 101 in → 31 out (20 reasoning)
feature-base feature-sub
openai/gpt-5.6-terra:high__json_schema PASS 100% 141 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-terra:low PASS 100% 95 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-terra:low__json_schema PASS 100% 144 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-terra:max PASS 100% 98 in → 55 out (44 reasoning)
feature-base feature-sub
openai/gpt-5.6-terra:max__json_schema PASS 100% 144 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-terra:medium PASS 100% 98 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-terra:medium__json_schema PASS 100% 144 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-terra:none PASS 100% 98 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-terra:none__json_schema PASS 100% 141 in → 20 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-5.6-terra:xhigh PASS 100% 101 in → 9 out (0 reasoning)
feature-base feature-sub
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 135 in → 59 out (37 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-oss-120b:high PASS 100% 157 in → 278 out (315 reasoning)
feature-base feature-sub
openai/gpt-oss-120b:high__json_schema PASS 100% 241 in → 304 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
openai/gpt-oss-120b:low PASS 100% 157 in → 60 out (52 reasoning)
feature-base feature-sub
openai/gpt-oss-120b:low__json_schema PASS 100% 149 in → 65 out (42 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
openai/gpt-oss-120b:medium PASS 100% 161 in → 114 out (123 reasoning)
feature-base feature-sub
openai/gpt-oss-120b:medium__json_schema PASS 100% 149 in → 181 out (165 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
openai/gpt-oss-120b:xhigh PASS 100% 160 in → 196 out (209 reasoning)
feature-base feature-sub
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 308 in → 586 out (564 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-oss-20b:high PASS 100% 161 in → 265 out (287 reasoning)
feature-base feature-sub
openai/gpt-oss-20b:high__json_schema PASS 100% 160 in → 468 out (509 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
openai/gpt-oss-20b:low PASS 100% 156 in → 77 out (72 reasoning)
feature-base feature-sub
openai/gpt-oss-20b:low__json_schema PASS 100% 151 in → 73 out (46 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
openai/gpt-oss-20b:medium PASS 100% 157 in → 206 out (216 reasoning)
feature-base feature-sub
openai/gpt-oss-20b:medium__json_schema PASS 100% 164 in → 130 out (120 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
openai/gpt-oss-20b:xhigh PASS 100% 157 in → 463 out (530 reasoning)
feature-base feature-sub
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 146 in → 373 out (364 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
poolside/laguna-m.1:high PASS 100% 117 in → 275 out (267 reasoning)
feature-base feature-sub
poolside/laguna-m.1:high__json_schema PASS 100% 117 in → 189 out (161 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
poolside/laguna-m.1:low PASS 100% 120 in → 152 out (144 reasoning)
feature-base feature-sub
poolside/laguna-m.1:low__json_schema PASS 100% 111 in → 412 out (384 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
poolside/laguna-m.1:medium PASS 100% 120 in → 227 out (219 reasoning)
feature-base feature-sub
poolside/laguna-m.1:medium__json_schema PASS 100% 120 in → 273 out (258 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
poolside/laguna-m.1:none PASS 100% 114 in → 7 out (0 reasoning)
feature-base feature-sub
poolside/laguna-m.1:none__json_schema PASS 100% 117 in → 15 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
poolside/laguna-m.1:xhigh PASS 100% 120 in → 203 out (195 reasoning)
feature-base feature-sub
poolside/laguna-m.1:xhigh__json_schema PASS 100% 120 in → 260 out (240 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
poolside/laguna-xs-2.1:high PASS 100% 117 in → 356 out (350 reasoning)
feature-base feature-sub
poolside/laguna-xs-2.1:high__json_schema PASS 100% 117 in → 237 out (209 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
poolside/laguna-xs-2.1:low PASS 100% 117 in → 303 out (297 reasoning)
feature-base feature-sub
poolside/laguna-xs-2.1:low__json_schema PASS 100% 114 in → 197 out (169 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
poolside/laguna-xs-2.1:medium PASS 100% 120 in → 216 out (210 reasoning)
feature-base feature-sub
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 117 in → 95 out (80 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
poolside/laguna-xs-2.1:none PASS 100% 117 in → 5 out (0 reasoning)
feature-base feature-sub
poolside/laguna-xs-2.1:none__json_schema PASS 100% 111 in → 19 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
poolside/laguna-xs-2.1:xhigh PASS 100% 117 in → 174 out (168 reasoning)
feature-base feature-sub
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 117 in → 276 out (247 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
poolside/laguna-xs.2:high PASS 100% 157 in → 109 out (101 reasoning)
feature-base feature-sub
poolside/laguna-xs.2:high__json_schema PASS 100% 151 in → 254 out (239 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
poolside/laguna-xs.2:low PASS 100% 151 in → 203 out (195 reasoning)
feature-base feature-sub
poolside/laguna-xs.2:low__json_schema PASS 100% 157 in → 183 out (168 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
poolside/laguna-xs.2:medium PASS 100% 151 in → 203 out (195 reasoning)
feature-base feature-sub
poolside/laguna-xs.2:medium__json_schema PASS 100% 151 in → 215 out (200 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
poolside/laguna-xs.2:none PASS 100% 157 in → 7 out (0 reasoning)
feature-base feature-sub
poolside/laguna-xs.2:none__json_schema PASS 100% 151 in → 14 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
poolside/laguna-xs.2:xhigh PASS 100% 154 in → 309 out (301 reasoning)
feature-base feature-sub
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 157 in → 168 out (153 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
qwen/qwen3.6-27b:high PASS 100% 112 in → 855 out (813 reasoning)
feature-base feature-sub
qwen/qwen3.6-27b:high__json_schema PASS 100% 115 in → 846 out (808 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
qwen/qwen3.6-27b:low PASS 100% 112 in → 737 out (716 reasoning)
feature-base feature-sub
qwen/qwen3.6-27b:low__json_schema PASS 100% 109 in → 604 out (1 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
qwen/qwen3.6-27b:medium PASS 100% 115 in → 904 out (860 reasoning)
feature-base feature-sub
qwen/qwen3.6-27b:medium__json_schema PASS 100% 115 in → 939 out (1 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
qwen/qwen3.6-27b:none PASS 100% 114 in → 6 out (0 reasoning)
feature-base feature-sub
qwen/qwen3.6-27b:none__json_schema PASS 100% 111 in → 25 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": [ "feature-base", "feature-sub" ]}
qwen/qwen3.6-27b:xhigh PASS 100% 115 in → 683 out (667 reasoning)
feature-base feature-sub
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 114 in → 27 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 115 in → 636 out (626 reasoning)
feature-base feature-sub
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 109 in → 579 out (572 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 112 in → 388 out (354 reasoning)
feature-base feature-sub
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 103 in → 637 out (600 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 109 in → 774 out (789 reasoning)
feature-base feature-sub
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 103 in → 853 out (832 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 111 in → 5 out (0 reasoning)
feature-base feature-sub
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 108 in → 27 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 103 in → 746 out (766 reasoning)
feature-base feature-sub
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 112 in → 795 out (800 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
qwen/qwen3.6-flash:high PASS 100% 115 in → 595 out (585 reasoning)
feature-base feature-sub
qwen/qwen3.6-flash:low PASS 100% 115 in → 783 out (773 reasoning)
feature-base feature-sub
qwen/qwen3.6-flash:medium PASS 100% 115 in → 705 out (695 reasoning)
feature-base feature-sub
qwen/qwen3.6-flash:none PASS 100% 117 in → 5 out (0 reasoning)
feature-base feature-sub
qwen/qwen3.6-flash:xhigh PASS 100% 109 in → 730 out (720 reasoning)
feature-base feature-sub
qwen/qwen3.7-flash:high PASS 100% 115 in → 359 out (349 reasoning)
feature-base feature-sub
qwen/qwen3.7-flash:low PASS 100% 115 in → 726 out (716 reasoning)
feature-base feature-sub
qwen/qwen3.7-flash:max PASS 100% 112 in → 622 out (612 reasoning)
feature-base feature-sub
qwen/qwen3.7-flash:medium PASS 100% 115 in → 402 out (392 reasoning)
feature-base feature-sub
qwen/qwen3.7-flash:none PASS 100% 111 in → 5 out (0 reasoning)
feature-base feature-sub
qwen/qwen3.7-flash:xhigh PASS 100% 112 in → 876 out (866 reasoning)
feature-base feature-sub
qwen/qwen3.7-max:high PASS 100% 112 in → 193 out (183 reasoning)
feature-base feature-sub
qwen/qwen3.7-max:low PASS 100% 115 in → 151 out (141 reasoning)
feature-base feature-sub
qwen/qwen3.7-max:medium PASS 100% 115 in → 152 out (142 reasoning)
feature-base feature-sub
qwen/qwen3.7-max:none PASS 100% 111 in → 5 out (0 reasoning)
feature-base feature-sub
qwen/qwen3.7-max:xhigh PASS 100% 112 in → 203 out (193 reasoning)
feature-base feature-sub
qwen/qwen3.7-plus:high PASS 100% 115 in → 211 out (201 reasoning)
feature-base feature-sub
qwen/qwen3.7-plus:low PASS 100% 115 in → 407 out (397 reasoning)
feature-base feature-sub
qwen/qwen3.7-plus:medium PASS 100% 103 in → 427 out (417 reasoning)
feature-base feature-sub
qwen/qwen3.7-plus:none PASS 100% 117 in → 5 out (0 reasoning)
feature-base feature-sub
qwen/qwen3.7-plus:xhigh PASS 100% 115 in → 521 out (511 reasoning)
feature-base feature-sub
tencent/hy3:high PASS 100% 103 in → 504 out (497 reasoning)
feature-base feature-sub
tencent/hy3:high__json_schema PASS 100% 103 in → 600 out (627 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
tencent/hy3:low PASS 100% 106 in → 304 out (297 reasoning)
feature-base feature-sub
tencent/hy3:low__json_schema PASS 100% 106 in → 330 out (343 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
tencent/hy3:medium PASS 100% 109 in → 333 out (326 reasoning)
feature-base feature-sub
tencent/hy3:medium__json_schema PASS 100% 105 in → 315 out (313 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
tencent/hy3:none PASS 100% 112 in → 6 out (0 reasoning)
feature-base feature-sub
tencent/hy3:none__json_schema PASS 100% 111 in → 23 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
tencent/hy3:xhigh PASS 100% 103 in → 462 out (455 reasoning)
feature-base feature-sub
tencent/hy3:xhigh__json_schema PASS 100% 109 in → 673 out (723 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
thinkingmachines/inkling-small:high PASS 100% 107 in → 186 out (197 reasoning)
feature-base feature-sub
thinkingmachines/inkling-small:high__json_schema PASS 100% 110 in → 144 out (118 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
thinkingmachines/inkling-small:low PASS 100% 107 in → 25 out (14 reasoning)
feature-base feature-sub
thinkingmachines/inkling-small:low__json_schema PASS 100% 104 in → 41 out (20 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
thinkingmachines/inkling-small:max PASS 100% 110 in → 173 out (162 reasoning)
feature-base feature-sub
thinkingmachines/inkling-small:max__json_schema PASS 100% 107 in → 125 out (99 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
thinkingmachines/inkling-small:medium PASS 100% 104 in → 84 out (73 reasoning)
feature-base feature-sub
thinkingmachines/inkling-small:medium__json_schema PASS 100% 110 in → 89 out (62 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
thinkingmachines/inkling-small:none PASS 100% 108 in → 8 out (0 reasoning)
feature-base feature-sub
thinkingmachines/inkling-small:xhigh PASS 100% 107 in → 75 out (64 reasoning)
feature-base feature-sub
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 104 in → 82 out (61 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
thinkingmachines/inkling:high PASS 100% 107 in → 180 out (188 reasoning)
feature-base feature-sub
thinkingmachines/inkling:high__json_schema PASS 100% 101 in → 184 out (164 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
thinkingmachines/inkling:low PASS 100% 104 in → 27 out (15 reasoning)
feature-base feature-sub
thinkingmachines/inkling:low__json_schema PASS 100% 104 in → 49 out (30 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
thinkingmachines/inkling:max PASS 100% 104 in → 151 out (140 reasoning)
feature-base feature-sub
thinkingmachines/inkling:max__json_schema PASS 100% 104 in → 177 out (157 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
thinkingmachines/inkling:medium PASS 100% 110 in → 141 out (137 reasoning)
feature-base feature-sub
thinkingmachines/inkling:medium__json_schema PASS 100% 104 in → 139 out (119 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
thinkingmachines/inkling:xhigh PASS 100% 107 in → 166 out (155 reasoning)
feature-base feature-sub
thinkingmachines/inkling:xhigh__json_schema PASS 100% 107 in → 182 out (162 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
x-ai/grok-4.3:high PASS 100% 280 in → 173 out (168 reasoning)
feature-base feature-sub
x-ai/grok-4.3:high__json_schema PASS 100% 339 in → 212 out (201 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
x-ai/grok-4.3:low PASS 100% 283 in → 331 out (326 reasoning)
feature-base feature-sub
x-ai/grok-4.3:low__json_schema PASS 100% 348 in → 290 out (274 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
x-ai/grok-4.3:max PASS 100% 274 in → 175 out (170 reasoning)
feature-base feature-sub
x-ai/grok-4.3:max__json_schema PASS 100% 345 in → 311 out (300 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
x-ai/grok-4.3:medium PASS 100% 289 in → 546 out (541 reasoning)
feature-base feature-sub
x-ai/grok-4.3:medium__json_schema PASS 100% 348 in → 419 out (403 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
x-ai/grok-4.3:none PASS 100% 272 in → 5 out (0 reasoning)
feature-base feature-sub
x-ai/grok-4.3:none__json_schema PASS 100% 343 in → 21 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
x-ai/grok-4.3:xhigh PASS 100% 277 in → 185 out (180 reasoning)
feature-base feature-sub
x-ai/grok-4.3:xhigh__json_schema PASS 100% 336 in → 358 out (347 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
x-ai/grok-4.5:high PASS 100% 306 in → 418 out (411 reasoning)
feature-base feature-sub
x-ai/grok-4.5:high__json_schema PASS 100% 388 in → 343 out (327 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
x-ai/grok-4.5:low PASS 100% 306 in → 228 out (221 reasoning)
feature-base feature-sub
x-ai/grok-4.5:low__json_schema PASS 100% 382 in → 294 out (278 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
x-ai/grok-4.5:max PASS 100% 306 in → 249 out (242 reasoning)
feature-base feature-sub
x-ai/grok-4.5:max__json_schema PASS 100% 385 in → 263 out (247 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
x-ai/grok-4.5:medium PASS 100% 306 in → 240 out (233 reasoning)
feature-base feature-sub
x-ai/grok-4.5:medium__json_schema PASS 100% 382 in → 359 out (343 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
x-ai/grok-4.5:xhigh PASS 100% 303 in → 258 out (251 reasoning)
feature-base feature-sub
x-ai/grok-4.5:xhigh__json_schema PASS 100% 382 in → 210 out (194 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-base","feature-sub"]}
z-ai/glm-4.7-flash:high PASS 100% 105 in → 474 out (455 reasoning)
feature-base feature-sub
z-ai/glm-4.7-flash:low PASS 100% 102 in → 490 out (506 reasoning)
feature-base feature-sub
z-ai/glm-4.7-flash:medium PASS 100% 99 in → 524 out (518 reasoning)
feature-base feature-sub
z-ai/glm-4.7-flash:none PASS 100% 99 in → 6 out (0 reasoning)
feature-base feature-sub
z-ai/glm-4.7-flash:none__json_schema PASS 100% 96 in → 22 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
z-ai/glm-4.7-flash:xhigh PASS 100% 99 in → 448 out (435 reasoning)
feature-base feature-sub
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 102 in → 908 out (872 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
z-ai/glm-5.2:high PASS 100% 103 in → 308 out (290 reasoning)
feature-base feature-sub
z-ai/glm-5.2:high__json_schema PASS 100% 109 in → 256 out (239 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-base", "feature-sub"] }
z-ai/glm-5.2:low PASS 100% 106 in → 346 out (339 reasoning)
feature-base feature-sub
z-ai/glm-5.2:low__json_schema PASS 100% 293 in → 96 out (73 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
z-ai/glm-5.2:medium PASS 100% 106 in → 138 out (137 reasoning)
feature-base feature-sub
z-ai/glm-5.2:medium__json_schema PASS 100% 287 in → 176 out (161 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
z-ai/glm-5.2:none PASS 100% 100 in → 6 out (0 reasoning)
feature-base feature-sub
z-ai/glm-5.2:none__json_schema PASS 100% 103 in → 16 out (0 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-base", "feature-sub"]}
z-ai/glm-5.2:xhigh PASS 100% 109 in → 419 out (401 reasoning)
feature-base feature-sub
z-ai/glm-5.2:xhigh__json_schema PASS 100% 112 in → 348 out (301 reasoning)
feature-base feature-sub
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-base", "feature-sub" ] }
thinkingmachines/inkling:none FAIL 50% 113 in → 8 out (0 reasoning)
feature-base feature-sub<|end_message|>
Failure: Missing: ['feature-sub'] Extra: ['feature-sub<|end_message|>']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: feature-base feature-sub
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high__json_schema FAIL 0% 257 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['feature-base', 'feature-sub']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:high FAIL 0% 114 in → 303 out (316 reasoning)
(empty output)
Failure: Missing: ['feature-base', 'feature-sub']
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 114 in → 2 out (0 reasoning)
none
Failure: Missing: ['feature-base', 'feature-sub'] Extra: ['none']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ab6d537e-a531-9112-98e6-0de4fb30dd6d","request_id":"ab6d537e-a531-9112-98e6-0de4fb30dd6d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3eece34e-45aa-93b2-946a-9b5ca45505d0","request_id":"3eece34e-45aa-93b2-946a-9b5ca45505d0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-87aad9b6-23a6-9bd8-9efd-254f70fe044b","request_id":"87aad9b6-23a6-9bd8-9efd-254f70fe044b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0235e520-8ad0-9eaf-ab0f-f8cd391015e7","request_id":"0235e520-8ad0-9eaf-ab0f-f8cd391015e7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-62a15fa5-f773-9afc-b721-a7d0f7b2206f","request_id":"62a15fa5-f773-9afc-b721-a7d0f7b2206f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-077b1a27-d83e-963f-a97e-18dc409b0431"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-58d3cf40-9a92-900c-81f4-0e3f069e96e9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-794c4bf4-4a46-9163-9bf7-cefe0e67d9f2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b7d1be97-0882-95db-99c9-b3ff96e64f08"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a23fb6a5-ed54-9a5d-8804-16be5101d424"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6e6b7ed0-8efd-95db-bc0b-8235a6b795b3"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5a94aaa5-13b3-937e-97ab-f6be4785065f","request_id":"5a94aaa5-13b3-937e-97ab-f6be4785065f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b8c87e23-687e-93e9-a8a0-6e65a2f4d717","request_id":"b8c87e23-687e-93e9-a8a0-6e65a2f4d717"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0319772f-39e2-9e53-949c-a524c15f0c7d","request_id":"0319772f-39e2-9e53-949c-a524c15f0c7d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-569c881b-204e-9e1a-98f9-b5ff2dc7ed04","request_id":"569c881b-204e-9e1a-98f9-b5ff2dc7ed04"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2e7fb12b-341c-98e0-becc-a26ce59ba57a","request_id":"2e7fb12b-341c-98e0-becc-a26ce59ba57a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-80c25b21-af8d-9b7a-8176-b570602ffedb","request_id":"80c25b21-af8d-9b7a-8176-b570602ffedb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-95627299-9724-9fbc-b158-f2abdd4fce16","request_id":"95627299-9724-9fbc-b158-f2abdd4fce16"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-63aa2e63-5504-9506-9f4a-12301c1a471b","request_id":"63aa2e63-5504-9506-9f4a-12301c1a471b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5967b2f4-ecc3-9dd8-92b9-f7598bb93e05","request_id":"5967b2f4-ecc3-9dd8-92b9-f7598bb93e05"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-743ed2ed-afa0-904f-a816-9c63b944ed55","request_id":"743ed2ed-afa0-904f-a816-9c63b944ed55"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: feature-base feature-sub{ "branches_to_delete": [ "feature-base", "feature-sub" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: feature-base feature-sub{ "branches_to_delete": [ "feature-base", "feature-sub" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: feature-base feature-sub
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: feature-base feature-sub
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)