Feature branches merged and unmerged
Tests ability to distinguish merged from unmerged feature branches. Evaluates selective branch cleanup with mixed merge status.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.py
  5. 05 git add app.py
  6. 06 git commit -m 'Initial app'
  7. 07 git checkout -b feature-auth
  8. 08 echo 'auth' > auth.py
  9. 09 git add auth.py
  10. 10 git commit -m 'Add auth'
  11. 11 git checkout main
  12. 12 git merge feature-auth
  13. 13 git checkout -b feature-db
  14. 14 echo 'db' > db.py
  15. 15 git add db.py
  16. 16 git commit -m 'Add db'
  17. 17 git checkout main
  18. 18 git merge feature-db
  19. 19 git checkout -b feature-api
  20. 20 echo 'api' > api.py
  21. 21 git add api.py
  22. 22 git commit -m 'Add api'
  23. 23 git checkout main
  24. 24 git checkout -b feature-ui
  25. 25 echo 'ui' > ui.py
  26. 26 git add ui.py
  27. 27 git commit -m 'Add ui'
  28. 28 git checkout main
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
feature-auth feature-db

Scoped model quality, cost, API time, and token usage for branch_cleanup/f004.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 166 in → 52 out (43 reasoning)
feature-auth feature-db
anthropic/claude-fable-5:high__json_schema PASS 100% 407 in → 75 out (47 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
anthropic/claude-fable-5:low PASS 100% 166 in → 29 out (21 reasoning)
feature-auth feature-db
anthropic/claude-fable-5:low__json_schema PASS 100% 415 in → 26 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-fable-5:max PASS 100% 170 in → 327 out (69 reasoning)
feature-auth feature-db
anthropic/claude-fable-5:max__json_schema PASS 100% 415 in → 231 out (80 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
anthropic/claude-fable-5:medium PASS 100% 171 in → 47 out (33 reasoning)
feature-auth feature-db
anthropic/claude-fable-5:medium__json_schema PASS 100% 412 in → 50 out (16 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
anthropic/claude-fable-5:xhigh PASS 100% 173 in → 79 out (51 reasoning)
feature-auth feature-db
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 413 in → 77 out (33 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
anthropic/claude-haiku-4.5:high PASS 100% 160 in → 213 out (184 reasoning)
feature-auth feature-db
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 355 in → 279 out (231 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
anthropic/claude-haiku-4.5:low PASS 100% 166 in → 259 out (218 reasoning)
feature-auth feature-db
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 350 in → 361 out (331 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
anthropic/claude-haiku-4.5:medium PASS 100% 169 in → 169 out (159 reasoning)
feature-auth feature-db
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 351 in → 373 out (346 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
anthropic/claude-haiku-4.5:none PASS 100% 139 in → 10 out (0 reasoning)
feature-auth feature-db
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 330 in → 21 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 166 in → 244 out (191 reasoning)
feature-auth feature-db
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 354 in → 337 out (277 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
anthropic/claude-opus-4.6:high PASS 100% 136 in → 122 out (97 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.6:high__json_schema PASS 100% 325 in → 53 out (20 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.6:low PASS 100% 135 in → 10 out (0 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.6:low__json_schema PASS 100% 323 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.6:max PASS 100% 136 in → 113 out (94 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.6:max__json_schema PASS 100% 324 in → 109 out (78 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 326 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.6:none PASS 100% 136 in → 10 out (0 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.6:none__json_schema PASS 100% 322 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 135 in → 109 out (87 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 327 in → 53 out (20 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.7:high PASS 100% 180 in → 14 out (0 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.7:high__json_schema PASS 100% 413 in → 28 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.7:low PASS 100% 172 in → 14 out (0 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.7:low__json_schema PASS 100% 417 in → 28 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.7:max PASS 100% 172 in → 182 out (53 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.7:max__json_schema PASS 100% 415 in → 28 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.7:medium PASS 100% 172 in → 14 out (0 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 412 in → 28 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.7:none PASS 100% 174 in → 14 out (0 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.7:none__json_schema PASS 100% 418 in → 28 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 173 in → 14 out (0 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 421 in → 28 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.8:high PASS 100% 173 in → 94 out (37 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.8:high__json_schema PASS 100% 410 in → 26 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.8:low PASS 100% 174 in → 12 out (0 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.8:low__json_schema PASS 100% 412 in → 26 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.8:max FAIL 100% 172 in → 1,284 out (151 reasoning)
Based on the "Merged into main" list (from `git branch --merged main`), the branches fully merged into main are `feature-auth`, `feature-db`, and `main` itself. Excluding `main` (which you never delete — it's the main branch and current branch): feature-auth feature-db **Reasoning:** - `feature-db` (897bbf3) points to the exact same commit as `main` — definitely merged. - `feature-auth` (36fff3c) appears in the merged list, meaning its commit is an ancestor of `main` — safe to delete. - `feature-api` (70bc3f0) and `feature-ui` (db99c9c) are **not** in the merged list, so they contain unmerged work — keep them. - `main` is excluded from deletion.
Failure: Extra: ['**Reasoning:**', '- `feature-api` (70bc3f0) and `feature-ui` (db99c9c) are **not** in the merged list, so they contain unmerged work — keep them.', '- `feature-auth` (36fff3c) appears in the merged list, meaning its commit is an ancestor of `main` — safe to delete.', '- `feature-db` (897bbf3) points to the exact same commit as `main` — definitely merged.', '- `main` is excluded from deletion.', 'Based on the "Merged into main" list (from `git branch --merged main`), the branches fully merged into main are `feature-auth`, `feature-db`, and `main` itself. Excluding `main` (which you never delete — it\'s the main branch and current branch):']
anthropic/claude-opus-4.8:max__json_schema PASS 100% 409 in → 291 out (59 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.8:medium PASS 100% 169 in → 141 out (46 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 407 in → 26 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.8:none FAIL 100% 173 in → 45 out (0 reasoning)
Based on the "Merged into main" list, the branches fully merged into main (excluding main itself) are: feature-auth feature-db
Failure: Extra: ['Based on the "Merged into main" list, the branches fully merged into main (excluding main itself) are:']
anthropic/claude-opus-4.8:none__json_schema PASS 100% 407 in → 25 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 169 in → 147 out (49 reasoning)
feature-auth feature-db
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 408 in → 26 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-4.6:high PASS 100% 141 in → 96 out (75 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 326 in → 77 out (45 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-4.6:low PASS 100% 136 in → 10 out (0 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 329 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-4.6:max PASS 100% 134 in → 103 out (83 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 324 in → 111 out (80 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 137 in → 10 out (0 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 326 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-4.6:none PASS 100% 136 in → 10 out (0 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 320 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 137 in → 100 out (79 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 325 in → 75 out (42 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-5:high PASS 100% 162 in → 120 out (60 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-5:high__json_schema PASS 100% 413 in → 26 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-5:low PASS 100% 169 in → 12 out (0 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-5:low__json_schema PASS 100% 409 in → 26 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-5:medium PASS 100% 166 in → 43 out (28 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 412 in → 26 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-5:none PASS 100% 171 in → 12 out (0 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-5:none__json_schema PASS 100% 412 in → 25 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 167 in → 167 out (69 reasoning)
feature-auth feature-db
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 404 in → 26 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
arcee-ai/trinity-large-thinking:high PASS 100% 123 in → 437 out (436 reasoning)
feature-auth feature-db
arcee-ai/trinity-large-thinking:low PASS 100% 124 in → 491 out (492 reasoning)
feature-auth feature-db
arcee-ai/trinity-large-thinking:medium PASS 100% 119 in → 438 out (436 reasoning)
feature-auth feature-db
arcee-ai/trinity-large-thinking:xhigh PASS 100% 123 in → 482 out (482 reasoning)
feature-auth feature-db
arcee-ai/trinity-mini:high PASS 100% 124 in → 276 out (312 reasoning)
feature-auth feature-db
arcee-ai/trinity-mini:high__json_schema PASS 100% 118 in → 270 out (290 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
arcee-ai/trinity-mini:low PASS 100% 120 in → 406 out (472 reasoning)
feature-auth feature-db
arcee-ai/trinity-mini:low__json_schema PASS 100% 121 in → 356 out (385 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete":[ "feature-auth", "feature-db" ] }
arcee-ai/trinity-mini:medium PASS 100% 127 in → 276 out (314 reasoning)
feature-auth feature-db
arcee-ai/trinity-mini:medium__json_schema PASS 100% 122 in → 248 out (266 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 118 in → 299 out (342 reasoning)
feature-auth feature-db
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 130 in → 272 out (298 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 205 in → 218 out (207 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 204 in → 353 out (328 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 123 in → 216 out (231 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 120 in → 589 out (756 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 198 in → 169 out (171 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 217 in → 438 out (442 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 123 in → 41 out (49 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 126 in → 339 out (341 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 120 in → 7 out (0 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 123 in → 70 out (97 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 121 in → 137 out (116 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
deepseek/deepseek-v4-flash:high PASS 100% 125 in → 126 out (128 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash:low PASS 100% 122 in → 128 out (114 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 321 in → 106 out (88 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
deepseek/deepseek-v4-flash:medium PASS 100% 124 in → 150 out (153 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 129 in → 114 out (89 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
deepseek/deepseek-v4-flash:none PASS 100% 126 in → 6 out (0 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 200 in → 17 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
deepseek/deepseek-v4-flash:xhigh PASS 100% 199 in → 188 out (180 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 204 in → 274 out (254 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
deepseek/deepseek-v4-pro:high PASS 100% 121 in → 418 out (427 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 318 in → 223 out (206 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
deepseek/deepseek-v4-pro:low PASS 100% 120 in → 254 out (262 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-pro:medium PASS 100% 122 in → 7 out (0 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 320 in → 144 out (126 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
deepseek/deepseek-v4-pro:none PASS 100% 127 in → 6 out (0 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 322 in → 17 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
deepseek/deepseek-v4-pro:xhigh PASS 100% 201 in → 929 out (921 reasoning)
feature-auth feature-db
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 398 in → 678 out (660 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
google/gemini-3-flash-preview:high PASS 100% 127 in → 236 out (229 reasoning)
feature-auth feature-db
google/gemini-3-flash-preview:high__json_schema PASS 100% 214 in → 338 out (321 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
google/gemini-3-flash-preview:low PASS 100% 123 in → 277 out (270 reasoning)
feature-auth feature-db
google/gemini-3-flash-preview:low__json_schema PASS 100% 213 in → 342 out (327 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3-flash-preview:medium PASS 100% 125 in → 273 out (266 reasoning)
feature-auth feature-db
google/gemini-3-flash-preview:medium__json_schema PASS 100% 210 in → 305 out (291 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3-flash-preview:xhigh PASS 100% 131 in → 339 out (332 reasoning)
feature-auth feature-db
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 215 in → 322 out (307 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 132 in → 643 out (636 reasoning)
feature-auth feature-db
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 216 in → 416 out (401 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 132 in → 129 out (122 reasoning)
feature-auth feature-db
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 133 in → 125 out (108 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 132 in → 388 out (381 reasoning)
feature-auth feature-db
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 212 in → 314 out (297 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 134 in → 448 out (441 reasoning)
feature-auth feature-db
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 134 in → 520 out (503 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
google/gemini-3.1-pro-preview:high PASS 100% 132 in → 411 out (404 reasoning)
feature-auth feature-db
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 210 in → 155 out (140 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.1-pro-preview:low PASS 100% 131 in → 369 out (362 reasoning)
feature-auth feature-db
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 211 in → 100 out (70 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemini-3.1-pro-preview:medium PASS 100% 126 in → 566 out (559 reasoning)
feature-auth feature-db
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 213 in → 466 out (451 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 134 in → 476 out (469 reasoning)
feature-auth feature-db
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 214 in → 578 out (563 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.5-flash-lite:high PASS 100% 131 in → 466 out (459 reasoning)
feature-auth feature-db
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 213 in → 539 out (524 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.5-flash-lite:low PASS 100% 134 in → 7 out (0 reasoning)
feature-auth feature-db
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 212 in → 30 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemini-3.5-flash-lite:max PASS 100% 132 in → 436 out (429 reasoning)
feature-auth feature-db
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 212 in → 486 out (471 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.5-flash-lite:medium PASS 100% 135 in → 362 out (355 reasoning)
feature-auth feature-db
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 131 in → 332 out (315 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 130 in → 439 out (432 reasoning)
feature-auth feature-db
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 135 in → 472 out (457 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.5-flash:high PASS 100% 128 in → 466 out (459 reasoning)
feature-auth feature-db
google/gemini-3.5-flash:high__json_schema PASS 100% 214 in → 439 out (424 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.5-flash:low PASS 100% 129 in → 323 out (316 reasoning)
feature-auth feature-db
google/gemini-3.5-flash:low__json_schema PASS 100% 212 in → 190 out (175 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.5-flash:medium PASS 100% 130 in → 560 out (553 reasoning)
feature-auth feature-db
google/gemini-3.5-flash:medium__json_schema PASS 100% 211 in → 359 out (344 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.5-flash:xhigh PASS 100% 129 in → 486 out (479 reasoning)
feature-auth feature-db
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 212 in → 448 out (433 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.6-flash:high PASS 100% 130 in → 526 out (519 reasoning)
feature-auth feature-db
google/gemini-3.6-flash:high__json_schema PASS 100% 130 in → 321 out (306 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.6-flash:low PASS 100% 134 in → 138 out (131 reasoning)
feature-auth feature-db
google/gemini-3.6-flash:low__json_schema PASS 100% 211 in → 30 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemini-3.6-flash:max PASS 100% 129 in → 358 out (351 reasoning)
feature-auth feature-db
google/gemini-3.6-flash:max__json_schema PASS 100% 212 in → 526 out (511 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.6-flash:medium PASS 100% 134 in → 245 out (238 reasoning)
feature-auth feature-db
google/gemini-3.6-flash:medium__json_schema PASS 100% 128 in → 292 out (277 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemini-3.6-flash:xhigh PASS 100% 130 in → 465 out (458 reasoning)
feature-auth feature-db
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 130 in → 504 out (489 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
google/gemma-4-26b-a4b-it:high PASS 100% 148 in → 324 out (252 reasoning)
feature-auth feature-db
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 148 in → 414 out (317 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 148 in → 331 out (255 reasoning)
feature-auth feature-db
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 146 in → 341 out (240 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 145 in → 340 out (263 reasoning)
feature-auth feature-db
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 146 in → 392 out (304 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 146 in → 8 out (0 reasoning)
feature-auth feature-db
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 142 in → 22 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 150 in → 371 out (283 reasoning)
feature-auth feature-db
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 149 in → 361 out (262 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemma-4-31b-it:high PASS 100% 147 in → 283 out (213 reasoning)
feature-auth feature-db
google/gemma-4-31b-it:high__json_schema PASS 100% 150 in → 232 out (163 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemma-4-31b-it:low PASS 100% 151 in → 276 out (196 reasoning)
feature-auth feature-db
google/gemma-4-31b-it:low__json_schema PASS 100% 148 in → 275 out (210 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemma-4-31b-it:medium PASS 100% 146 in → 302 out (238 reasoning)
feature-auth feature-db
google/gemma-4-31b-it:medium__json_schema PASS 100% 147 in → 279 out (210 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemma-4-31b-it:none PASS 100% 142 in → 8 out (0 reasoning)
feature-auth feature-db
google/gemma-4-31b-it:none__json_schema PASS 100% 143 in → 31 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
google/gemma-4-31b-it:xhigh PASS 100% 148 in → 226 out (1 reasoning)
feature-auth feature-db
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 149 in → 289 out (272 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
ibm-granite/granite-4.1-8b__json_schema PASS 100% 138 in → 17 out
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
inclusionai/ling-2.6-flash PASS 100% 148 in → 6 out
feature-auth feature-db
inclusionai/ling-2.6-flash__json_schema PASS 100% 150 in → 28 out
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
liquid/lfm-2-24b-a2b PASS 100% 136 in → 8 out
feature-auth feature-db
minimax/minimax-m2.5:high PASS 100% 168 in → 947 out (940 reasoning)
feature-auth feature-db
minimax/minimax-m2.5:high__json_schema PASS 100% 155 in → 571 out (548 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
minimax/minimax-m2.5:low PASS 100% 153 in → 789 out (883 reasoning)
feature-auth feature-db
minimax/minimax-m2.5:low__json_schema PASS 100% 155 in → 501 out (477 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
minimax/minimax-m2.5:medium PASS 100% 166 in → 457 out (450 reasoning)
feature-auth feature-db
minimax/minimax-m2.5:medium__json_schema PASS 100% 155 in → 321 out (302 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
minimax/minimax-m2.5:xhigh PASS 100% 156 in → 265 out (258 reasoning)
feature-auth feature-db
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 154 in → 522 out (596 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
minimax/minimax-m2.7:high PASS 100% 158 in → 234 out (228 reasoning)
feature-auth feature-db
minimax/minimax-m2.7:high__json_schema PASS 100% 158 in → 18 out (18 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
minimax/minimax-m2.7:low PASS 100% 154 in → 253 out (280 reasoning)
feature-auth feature-db
minimax/minimax-m2.7:low__json_schema PASS 100% 307 in → 601 out (586 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
minimax/minimax-m2.7:medium PASS 100% 158 in → 441 out (518 reasoning)
feature-auth feature-db
minimax/minimax-m2.7:medium__json_schema PASS 100% 307 in → 261 out (242 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
minimax/minimax-m2.7:xhigh PASS 100% 157 in → 418 out (410 reasoning)
feature-auth feature-db
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 308 in → 566 out (551 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
minimax/minimax-m3:high PASS 100% 298 in → 335 out (338 reasoning)
feature-auth feature-db
minimax/minimax-m3:high__json_schema PASS 100% 281 in → 17 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
minimax/minimax-m3:low PASS 100% 291 in → 138 out (141 reasoning)
feature-auth feature-db
minimax/minimax-m3:medium PASS 100% 294 in → 224 out (247 reasoning)
feature-auth feature-db
minimax/minimax-m3:xhigh PASS 100% 296 in → 126 out (144 reasoning)
feature-auth feature-db
minimax/minimax-m3:xhigh__json_schema PASS 100% 279 in → 23 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
mistralai/devstral-2512 PASS 100% 136 in → 6 out
feature-auth feature-db
mistralai/devstral-2512__json_schema PASS 100% 129 in → 16 out
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
mistralai/mistral-medium-3-5:high PASS 100% 144 in → 405 out (428 reasoning)
feature-auth feature-db
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 134 in → 215 out (190 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
mistralai/mistral-medium-3-5:low PASS 100% 144 in → 630 out (650 reasoning)
feature-auth feature-db
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 135 in → 286 out (290 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
mistralai/mistral-medium-3-5:medium PASS 100% 146 in → 437 out (457 reasoning)
feature-auth feature-db
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 133 in → 252 out (253 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
mistralai/mistral-medium-3-5:none PASS 100% 142 in → 6 out (0 reasoning)
feature-auth feature-db
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 135 in → 16 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 139 in → 273 out (272 reasoning)
feature-auth feature-db
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 134 in → 300 out (276 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
moonshotai/kimi-k2.6:high PASS 100% 124 in → 333 out (326 reasoning)
feature-auth feature-db
moonshotai/kimi-k2.6:high__json_schema PASS 100% 319 in → 954 out (939 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
moonshotai/kimi-k2.6:low PASS 100% 121 in → 503 out (496 reasoning)
feature-auth feature-db
moonshotai/kimi-k2.6:medium PASS 100% 123 in → 280 out (273 reasoning)
feature-auth feature-db
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 122 in → 557 out (595 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
moonshotai/kimi-k2.6:xhigh PASS 100% 128 in → 243 out (253 reasoning)
feature-auth feature-db
moonshotai/kimi-k2.7-code:high PASS 100% 124 in → 144 out (137 reasoning)
feature-auth feature-db
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 120 in → 175 out (178 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
moonshotai/kimi-k2.7-code:low PASS 100% 121 in → 201 out (194 reasoning)
feature-auth feature-db
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 125 in → 104 out (84 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
moonshotai/kimi-k2.7-code:medium PASS 100% 119 in → 226 out (219 reasoning)
feature-auth feature-db
moonshotai/kimi-k2.7-code:xhigh PASS 100% 125 in → 127 out (120 reasoning)
feature-auth feature-db
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 128 in → 209 out (188 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
moonshotai/kimi-k3:high PASS 100% 204 in → 150 out (130 reasoning)
feature-auth feature-db
moonshotai/kimi-k3:high__json_schema PASS 100% 197 in → 195 out (175 reasoning)
feature-auth feature-db
moonshotai/kimi-k3:low PASS 100% 201 in → 109 out (89 reasoning)
feature-auth feature-db
moonshotai/kimi-k3:low__json_schema PASS 100% 301 in → 253 out (225 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
moonshotai/kimi-k3:max PASS 100% 200 in → 130 out (110 reasoning)
feature-auth feature-db
moonshotai/kimi-k3:max__json_schema PASS 100% 200 in → 223 out (203 reasoning)
feature-auth feature-db
moonshotai/kimi-k3:medium PASS 100% 203 in → 283 out (263 reasoning)
feature-auth feature-db
moonshotai/kimi-k3:medium__json_schema PASS 100% 203 in → 188 out (168 reasoning)
feature-auth feature-db
moonshotai/kimi-k3:xhigh PASS 100% 200 in → 157 out (137 reasoning)
feature-auth feature-db
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 196 in → 174 out (154 reasoning)
feature-auth feature-db
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 143 in → 244 out (250 reasoning)
feature-auth feature-db
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 146 in → 145 out (148 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 144 in → 117 out (131 reasoning)
feature-auth feature-db
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 146 in → 96 out (84 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 150 in → 154 out (173 reasoning)
feature-auth feature-db
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 144 in → 158 out (162 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 144 in → 34 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 141 in → 141 out (154 reasoning)
feature-auth feature-db
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 137 in → 116 out (109 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 142 in → 177 out (167 reasoning)
feature-auth feature-db
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 145 in → 106 out (82 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 146 in → 108 out (99 reasoning)
feature-auth feature-db
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 143 in → 127 out (102 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 144 in → 92 out (82 reasoning)
feature-auth feature-db
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 142 in → 158 out (123 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 145 in → 6 out (0 reasoning)
feature-auth feature-db
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 144 in → 23 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 144 in → 176 out (198 reasoning)
feature-auth feature-db
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 142 in → 180 out (147 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
openai/gpt-5.4-mini:high PASS 100% 122 in → 103 out (92 reasoning)
feature-auth feature-db
openai/gpt-5.4-mini:high__json_schema PASS 100% 163 in → 274 out (252 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4-mini:low PASS 100% 121 in → 46 out (35 reasoning)
feature-auth feature-db
openai/gpt-5.4-mini:low__json_schema PASS 100% 169 in → 57 out (35 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4-mini:medium PASS 100% 120 in → 86 out (75 reasoning)
feature-auth feature-db
openai/gpt-5.4-mini:medium__json_schema PASS 100% 164 in → 118 out (96 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4-mini:none PASS 100% 118 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.4-mini:none__json_schema PASS 100% 166 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4-mini:xhigh PASS 100% 124 in → 113 out (102 reasoning)
feature-auth feature-db
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 164 in → 336 out (314 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4-nano:high PASS 100% 127 in → 52 out (41 reasoning)
feature-auth feature-db
openai/gpt-5.4-nano:high__json_schema PASS 100% 159 in → 91 out (69 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4-nano:low PASS 100% 116 in → 39 out (28 reasoning)
feature-auth feature-db
openai/gpt-5.4-nano:low__json_schema PASS 100% 161 in → 63 out (41 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4-nano:medium PASS 100% 122 in → 43 out (32 reasoning)
feature-auth feature-db
openai/gpt-5.4-nano:medium__json_schema PASS 100% 169 in → 67 out (45 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4-nano:none PASS 100% 121 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.4-nano:none__json_schema PASS 100% 164 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4-nano:xhigh PASS 100% 123 in → 99 out (88 reasoning)
feature-auth feature-db
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 165 in → 336 out (314 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4:high PASS 100% 124 in → 136 out (125 reasoning)
feature-auth feature-db
openai/gpt-5.4:high__json_schema PASS 100% 163 in → 95 out (73 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4:low PASS 100% 124 in → 66 out (55 reasoning)
feature-auth feature-db
openai/gpt-5.4:low__json_schema PASS 100% 161 in → 71 out (49 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4:medium PASS 100% 121 in → 96 out (85 reasoning)
feature-auth feature-db
openai/gpt-5.4:medium__json_schema PASS 100% 165 in → 97 out (75 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4:none PASS 100% 123 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.4:none__json_schema PASS 100% 166 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.4:xhigh PASS 100% 124 in → 142 out (131 reasoning)
feature-auth feature-db
openai/gpt-5.4:xhigh__json_schema PASS 100% 164 in → 199 out (177 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.5:high PASS 100% 119 in → 114 out (103 reasoning)
feature-auth feature-db
openai/gpt-5.5:high__json_schema PASS 100% 168 in → 101 out (79 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.5:low PASS 100% 125 in → 57 out (46 reasoning)
feature-auth feature-db
openai/gpt-5.5:low__json_schema PASS 100% 162 in → 66 out (44 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.5:medium PASS 100% 123 in → 72 out (61 reasoning)
feature-auth feature-db
openai/gpt-5.5:medium__json_schema PASS 100% 165 in → 69 out (47 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.5:none PASS 100% 122 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.5:none__json_schema PASS 100% 168 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.5:xhigh PASS 100% 121 in → 119 out (108 reasoning)
feature-auth feature-db
openai/gpt-5.5:xhigh__json_schema PASS 100% 162 in → 142 out (120 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-luna:high PASS 100% 120 in → 98 out (87 reasoning)
feature-auth feature-db
openai/gpt-5.6-luna:high__json_schema PASS 100% 163 in → 73 out (51 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-luna:low PASS 100% 120 in → 35 out (24 reasoning)
feature-auth feature-db
openai/gpt-5.6-luna:low__json_schema PASS 100% 165 in → 58 out (36 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-luna:max PASS 100% 123 in → 125 out (114 reasoning)
feature-auth feature-db
openai/gpt-5.6-luna:max__json_schema PASS 100% 165 in → 146 out (124 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-luna:medium PASS 100% 123 in → 53 out (42 reasoning)
feature-auth feature-db
openai/gpt-5.6-luna:medium__json_schema PASS 100% 166 in → 70 out (48 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-luna:none__json_schema PASS 100% 162 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-luna:xhigh PASS 100% 127 in → 62 out (51 reasoning)
feature-auth feature-db
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 170 in → 253 out (231 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-sol:high PASS 100% 124 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.6-sol:high__json_schema PASS 100% 167 in → 58 out (36 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-sol:low PASS 100% 124 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.6-sol:low__json_schema PASS 100% 167 in → 68 out (46 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-sol:max PASS 100% 121 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.6-sol:max__json_schema PASS 100% 163 in → 107 out (85 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-sol:medium PASS 100% 124 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.6-sol:medium__json_schema PASS 100% 165 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-sol:none PASS 100% 121 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.6-sol:none__json_schema PASS 100% 163 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-sol:xhigh PASS 100% 119 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 161 in → 107 out (85 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-terra:high PASS 100% 118 in → 43 out (32 reasoning)
feature-auth feature-db
openai/gpt-5.6-terra:high__json_schema PASS 100% 163 in → 147 out (125 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-terra:low PASS 100% 124 in → 60 out (49 reasoning)
feature-auth feature-db
openai/gpt-5.6-terra:low__json_schema PASS 100% 163 in → 40 out (18 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-terra:max PASS 100% 124 in → 57 out (46 reasoning)
feature-auth feature-db
openai/gpt-5.6-terra:max__json_schema PASS 100% 165 in → 85 out (63 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-terra:medium PASS 100% 122 in → 43 out (32 reasoning)
feature-auth feature-db
openai/gpt-5.6-terra:medium__json_schema PASS 100% 163 in → 81 out (59 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-terra:none PASS 100% 120 in → 9 out (0 reasoning)
feature-auth feature-db
openai/gpt-5.6-terra:none__json_schema PASS 100% 165 in → 20 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-5.6-terra:xhigh PASS 100% 123 in → 47 out (36 reasoning)
feature-auth feature-db
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 165 in → 72 out (50 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-oss-120b:high PASS 100% 181 in → 345 out (365 reasoning)
feature-auth feature-db
openai/gpt-oss-120b:high__json_schema PASS 100% 180 in → 649 out (677 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
openai/gpt-oss-120b:low PASS 100% 182 in → 186 out (201 reasoning)
feature-auth feature-db
openai/gpt-oss-120b:low__json_schema PASS 100% 184 in → 231 out (212 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
openai/gpt-oss-120b:medium PASS 100% 187 in → 136 out (138 reasoning)
feature-auth feature-db
openai/gpt-oss-120b:medium__json_schema PASS 100% 184 in → 206 out (186 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
openai/gpt-oss-120b:xhigh PASS 100% 182 in → 450 out (506 reasoning)
feature-auth feature-db
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 181 in → 454 out (428 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
openai/gpt-oss-20b:high PASS 100% 180 in → 322 out (343 reasoning)
feature-auth feature-db
openai/gpt-oss-20b:high__json_schema PASS 100% 178 in → 16 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
openai/gpt-oss-20b:low PASS 100% 188 in → 183 out (178 reasoning)
feature-auth feature-db
openai/gpt-oss-20b:low__json_schema PASS 100% 184 in → 46 out (23 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
openai/gpt-oss-20b:medium PASS 100% 186 in → 221 out (236 reasoning)
feature-auth feature-db
openai/gpt-oss-20b:medium__json_schema PASS 100% 168 in → 201 out (214 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
openai/gpt-oss-20b:xhigh PASS 100% 189 in → 519 out (559 reasoning)
feature-auth feature-db
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 178 in → 420 out (476 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
poolside/laguna-m.1:high PASS 100% 146 in → 286 out (278 reasoning)
feature-auth feature-db
poolside/laguna-m.1:high__json_schema PASS 100% 143 in → 342 out (317 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
poolside/laguna-m.1:low PASS 100% 142 in → 228 out (220 reasoning)
feature-auth feature-db
poolside/laguna-m.1:low__json_schema PASS 100% 142 in → 407 out (390 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
poolside/laguna-m.1:medium PASS 100% 141 in → 320 out (312 reasoning)
feature-auth feature-db
poolside/laguna-m.1:medium__json_schema PASS 100% 144 in → 390 out (364 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
poolside/laguna-m.1:none PASS 100% 145 in → 7 out (0 reasoning)
feature-auth feature-db
poolside/laguna-m.1:xhigh PASS 100% 140 in → 279 out (271 reasoning)
feature-auth feature-db
poolside/laguna-m.1:xhigh__json_schema PASS 100% 144 in → 464 out (440 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
poolside/laguna-xs-2.1:high PASS 100% 139 in → 261 out (255 reasoning)
feature-auth feature-db
poolside/laguna-xs-2.1:high__json_schema PASS 100% 145 in → 265 out (250 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
poolside/laguna-xs-2.1:low PASS 100% 141 in → 362 out (356 reasoning)
feature-auth feature-db
poolside/laguna-xs-2.1:low__json_schema PASS 100% 142 in → 317 out (288 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
poolside/laguna-xs-2.1:medium PASS 100% 146 in → 243 out (237 reasoning)
feature-auth feature-db
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 143 in → 362 out (347 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
poolside/laguna-xs-2.1:none PASS 100% 144 in → 5 out (0 reasoning)
feature-auth feature-db
poolside/laguna-xs-2.1:none__json_schema PASS 100% 144 in → 14 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
poolside/laguna-xs-2.1:xhigh PASS 100% 144 in → 236 out (230 reasoning)
feature-auth feature-db
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 143 in → 245 out (230 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
poolside/laguna-xs.2:high PASS 100% 182 in → 170 out (162 reasoning)
feature-auth feature-db
poolside/laguna-xs.2:high__json_schema PASS 100% 175 in → 209 out (194 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
poolside/laguna-xs.2:low PASS 100% 185 in → 284 out (276 reasoning)
feature-auth feature-db
poolside/laguna-xs.2:low__json_schema PASS 100% 178 in → 255 out (235 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
poolside/laguna-xs.2:medium PASS 100% 177 in → 252 out (244 reasoning)
feature-auth feature-db
poolside/laguna-xs.2:medium__json_schema PASS 100% 181 in → 225 out (210 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
poolside/laguna-xs.2:none PASS 100% 181 in → 7 out (0 reasoning)
feature-auth feature-db
poolside/laguna-xs.2:none__json_schema PASS 100% 180 in → 14 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
poolside/laguna-xs.2:xhigh PASS 100% 178 in → 140 out (132 reasoning)
feature-auth feature-db
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 182 in → 223 out (208 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
qwen/qwen3.6-27b:high PASS 100% 143 in → 1,285 out (1,275 reasoning)
feature-auth feature-db
qwen/qwen3.6-27b:high__json_schema PASS 100% 137 in → 886 out (833 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
qwen/qwen3.6-27b:low PASS 100% 141 in → 853 out (823 reasoning)
feature-auth feature-db
qwen/qwen3.6-27b:low__json_schema PASS 100% 140 in → 945 out (910 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
qwen/qwen3.6-27b:medium PASS 100% 144 in → 778 out (736 reasoning)
feature-auth feature-db
qwen/qwen3.6-27b:medium__json_schema PASS 100% 141 in → 741 out (736 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
qwen/qwen3.6-27b:none PASS 100% 142 in → 6 out (0 reasoning)
feature-auth feature-db
qwen/qwen3.6-27b:none__json_schema PASS 100% 141 in → 27 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 137 in → 409 out (371 reasoning)
feature-auth feature-db
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 144 in → 17 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"]}
qwen/qwen3.6-35b-a3b:high PASS 100% 141 in → 381 out (371 reasoning)
feature-auth feature-db
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 143 in → 706 out (703 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 139 in → 773 out (763 reasoning)
feature-auth feature-db
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 142 in → 602 out (594 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 136 in → 703 out (710 reasoning)
feature-auth feature-db
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 140 in → 1,086 out (1,029 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 141 in → 6 out (0 reasoning)
feature-auth feature-db
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 141 in → 12 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 139 in → 890 out (910 reasoning)
feature-auth feature-db
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 141 in → 769 out (740 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
qwen/qwen3.6-flash:high PASS 100% 141 in → 450 out (440 reasoning)
feature-auth feature-db
qwen/qwen3.6-flash:low PASS 100% 138 in → 782 out (772 reasoning)
feature-auth feature-db
qwen/qwen3.6-flash:medium PASS 100% 138 in → 1,222 out (1,212 reasoning)
feature-auth feature-db
qwen/qwen3.6-flash:none FAIL 100% 143 in → 11 out (0 reasoning)
feature-api feature-auth feature-db feature-ui
Failure: Extra: ['feature-api', 'feature-ui']
qwen/qwen3.6-flash:xhigh PASS 100% 143 in → 783 out (773 reasoning)
feature-auth feature-db
qwen/qwen3.7-flash:high PASS 100% 143 in → 635 out (625 reasoning)
feature-auth feature-db
qwen/qwen3.7-flash:low PASS 100% 136 in → 600 out (590 reasoning)
feature-auth feature-db
qwen/qwen3.7-flash:max PASS 100% 139 in → 355 out (345 reasoning)
feature-auth feature-db
qwen/qwen3.7-flash:medium PASS 100% 138 in → 795 out (785 reasoning)
feature-auth feature-db
qwen/qwen3.7-flash:none FAIL 100% 138 in → 8 out (0 reasoning)
feature-api feature-auth feature-db
Failure: Extra: ['feature-api']
qwen/qwen3.7-flash:xhigh PASS 100% 138 in → 656 out (646 reasoning)
feature-auth feature-db
qwen/qwen3.7-max:high PASS 100% 139 in → 222 out (212 reasoning)
feature-auth feature-db
qwen/qwen3.7-max:low PASS 100% 140 in → 132 out (122 reasoning)
feature-auth feature-db
qwen/qwen3.7-max:medium PASS 100% 145 in → 239 out (229 reasoning)
feature-auth feature-db
qwen/qwen3.7-max:none PASS 100% 144 in → 5 out (0 reasoning)
feature-auth feature-db
qwen/qwen3.7-max:xhigh PASS 100% 140 in → 258 out (248 reasoning)
feature-auth feature-db
qwen/qwen3.7-plus:high PASS 100% 135 in → 477 out (467 reasoning)
feature-auth feature-db
qwen/qwen3.7-plus:low PASS 100% 139 in → 467 out (457 reasoning)
feature-auth feature-db
qwen/qwen3.7-plus:medium PASS 100% 136 in → 493 out (483 reasoning)
feature-auth feature-db
qwen/qwen3.7-plus:none PASS 100% 141 in → 5 out (0 reasoning)
feature-auth feature-db
qwen/qwen3.7-plus:xhigh PASS 100% 139 in → 266 out (256 reasoning)
feature-auth feature-db
tencent/hy3:high PASS 100% 136 in → 493 out (485 reasoning)
feature-auth feature-db
tencent/hy3:high__json_schema PASS 100% 135 in → 900 out (866 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"]}
tencent/hy3:low PASS 100% 129 in → 403 out (398 reasoning)
feature-auth feature-db
tencent/hy3:low__json_schema PASS 100% 133 in → 530 out (516 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
tencent/hy3:medium PASS 100% 130 in → 351 out (336 reasoning)
feature-auth feature-db
tencent/hy3:medium__json_schema PASS 100% 131 in → 468 out (456 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
tencent/hy3:none PASS 100% 141 in → 7 out (0 reasoning)
feature-auth feature-db
tencent/hy3:none__json_schema PASS 100% 135 in → 25 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
tencent/hy3:xhigh PASS 100% 134 in → 417 out (409 reasoning)
feature-auth feature-db
tencent/hy3:xhigh__json_schema PASS 100% 133 in → 606 out (615 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
thinkingmachines/inkling-small:high PASS 100% 131 in → 189 out (178 reasoning)
feature-auth feature-db
thinkingmachines/inkling-small:high__json_schema PASS 100% 133 in → 123 out (97 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
thinkingmachines/inkling-small:low PASS 100% 129 in → 62 out (58 reasoning)
feature-auth feature-db
thinkingmachines/inkling-small:low__json_schema PASS 100% 132 in → 64 out (37 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
thinkingmachines/inkling-small:max PASS 100% 132 in → 152 out (141 reasoning)
feature-auth feature-db
thinkingmachines/inkling-small:max__json_schema PASS 100% 130 in → 273 out (246 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
thinkingmachines/inkling-small:medium PASS 100% 128 in → 99 out (96 reasoning)
feature-auth feature-db
thinkingmachines/inkling-small:medium__json_schema PASS 100% 132 in → 143 out (122 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
thinkingmachines/inkling-small:xhigh PASS 100% 127 in → 255 out (261 reasoning)
feature-auth feature-db
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 127 in → 232 out (206 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
thinkingmachines/inkling:high PASS 100% 125 in → 137 out (126 reasoning)
feature-auth feature-db
thinkingmachines/inkling:high__json_schema PASS 100% 131 in → 216 out (196 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth", "feature-db"]}
thinkingmachines/inkling:low PASS 100% 132 in → 30 out (18 reasoning)
feature-auth feature-db
thinkingmachines/inkling:max PASS 100% 129 in → 146 out (140 reasoning)
feature-auth feature-db
thinkingmachines/inkling:max__json_schema PASS 100% 127 in → 251 out (225 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
thinkingmachines/inkling:medium PASS 100% 132 in → 63 out (64 reasoning)
feature-auth feature-db
thinkingmachines/inkling:none PASS 100% 129 in → 8 out (0 reasoning)
feature-auth feature-db
thinkingmachines/inkling:xhigh PASS 100% 130 in → 172 out (161 reasoning)
feature-auth feature-db
thinkingmachines/inkling:xhigh__json_schema PASS 100% 127 in → 185 out (159 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
x-ai/grok-4.3:high PASS 100% 300 in → 399 out (394 reasoning)
feature-auth feature-db
x-ai/grok-4.3:high__json_schema PASS 100% 365 in → 614 out (603 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
x-ai/grok-4.3:low PASS 100% 308 in → 387 out (382 reasoning)
feature-auth feature-db
x-ai/grok-4.3:low__json_schema PASS 100% 372 in → 362 out (346 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
x-ai/grok-4.3:max PASS 100% 304 in → 453 out (448 reasoning)
feature-auth feature-db
x-ai/grok-4.3:max__json_schema PASS 100% 366 in → 512 out (501 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
x-ai/grok-4.3:medium PASS 100% 304 in → 616 out (611 reasoning)
feature-auth feature-db
x-ai/grok-4.3:medium__json_schema PASS 100% 377 in → 428 out (412 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": ["feature-auth", "feature-db"] }
x-ai/grok-4.3:none PASS 100% 295 in → 5 out (0 reasoning)
feature-auth feature-db
x-ai/grok-4.3:none__json_schema PASS 100% 360 in → 11 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
x-ai/grok-4.3:xhigh PASS 100% 300 in → 578 out (573 reasoning)
feature-auth feature-db
x-ai/grok-4.3:xhigh__json_schema PASS 100% 366 in → 414 out (403 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
x-ai/grok-4.5:high PASS 100% 324 in → 416 out (409 reasoning)
feature-auth feature-db
x-ai/grok-4.5:high__json_schema PASS 100% 410 in → 531 out (515 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
x-ai/grok-4.5:low PASS 100% 330 in → 276 out (269 reasoning)
feature-auth feature-db
x-ai/grok-4.5:low__json_schema PASS 100% 412 in → 332 out (316 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
x-ai/grok-4.5:max PASS 100% 327 in → 312 out (305 reasoning)
feature-auth feature-db
x-ai/grok-4.5:max__json_schema PASS 100% 414 in → 610 out (594 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
x-ai/grok-4.5:medium PASS 100% 327 in → 443 out (436 reasoning)
feature-auth feature-db
x-ai/grok-4.5:medium__json_schema PASS 100% 412 in → 471 out (455 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
x-ai/grok-4.5:xhigh PASS 100% 336 in → 481 out (474 reasoning)
feature-auth feature-db
x-ai/grok-4.5:xhigh__json_schema PASS 100% 413 in → 403 out (387 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) {"branches_to_delete":["feature-auth","feature-db"]}
z-ai/glm-4.7-flash:high PASS 100% 128 in → 479 out (425 reasoning)
feature-auth feature-db
z-ai/glm-4.7-flash:low PASS 100% 127 in → 596 out (563 reasoning)
feature-auth feature-db
z-ai/glm-4.7-flash:medium PASS 100% 124 in → 387 out (380 reasoning)
feature-auth feature-db
z-ai/glm-4.7-flash:xhigh PASS 100% 125 in → 656 out (608 reasoning)
feature-auth feature-db
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 123 in → 559 out (478 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
z-ai/glm-5.2:high PASS 100% 129 in → 280 out (265 reasoning)
feature-auth feature-db
z-ai/glm-5.2:high__json_schema PASS 100% 131 in → 293 out (273 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
z-ai/glm-5.2:low PASS 100% 130 in → 295 out (288 reasoning)
feature-auth feature-db
z-ai/glm-5.2:low__json_schema PASS 100% 132 in → 182 out (159 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
z-ai/glm-5.2:medium PASS 100% 135 in → 284 out (0 reasoning)
feature-auth feature-db
z-ai/glm-5.2:medium__json_schema PASS 100% 135 in → 207 out (183 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
z-ai/glm-5.2:none PASS 100% 129 in → 6 out (0 reasoning)
feature-auth feature-db
z-ai/glm-5.2:none__json_schema PASS 100% 123 in → 24 out (0 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
z-ai/glm-5.2:xhigh PASS 100% 133 in → 367 out (350 reasoning)
feature-auth feature-db
z-ai/glm-5.2:xhigh__json_schema PASS 100% 134 in → 359 out (316 reasoning)
feature-auth feature-db
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth", "feature-db" ] }
anthropic/claude-opus-4.6:medium FAIL 50% 141 in → 6 out (0 reasoning)
feature-db
Failure: Missing: ['feature-auth']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 50% 133 in → 18 out (0 reasoning)
feature-auth
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-auth" ] }
Failure: Missing: ['feature-db']
openai/gpt-5.6-luna:none FAIL 50% 119 in → 6 out (0 reasoning)
feature-db
Failure: Missing: ['feature-auth']
poolside/laguna-m.1:none__json_schema FAIL 50% 143 in → 10 out (0 reasoning)
feature-auth
JSON Schema Structured Output
(raw) {"branches_to_delete": ["feature-auth"]}
Failure: Missing: ['feature-db']
thinkingmachines/inkling-small:none FAIL 50% 132 in → 8 out (0 reasoning)
feature-auth feature-db<|end_message|>
Failure: Missing: ['feature-db'] Extra: ['feature-db<|end_message|>']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:high__json_schema FAIL 0% 123 in → 229 out (205 reasoning)
feature-api feature-ui
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-api", "feature-ui" ] }
Failure: Missing: ['feature-auth', 'feature-db'] Extra: ['feature-api', 'feature-ui']
Invalid JSON. Output: feature-auth feature-db
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
ibm-granite/granite-4.1-8b FAIL 0% 135 in → 6 out
feature-api feature-ui
Failure: Missing: ['feature-auth', 'feature-db'] Extra: ['feature-api', 'feature-ui']
Invalid JSON. Output: feature-auth feature-db
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:low__json_schema FAIL 0% 279 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['feature-auth', 'feature-db']
minimax/minimax-m3:medium__json_schema FAIL 0% 281 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "branches_to_delete": [] }
Failure: Missing: ['feature-auth', 'feature-db']
Invalid JSON. Output: feature-auth feature-db
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: feature-auth feature-db
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 144 in → 2 out (0 reasoning)
none
Failure: Missing: ['feature-auth', 'feature-db'] Extra: ['none']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-39114067-7a1a-922c-a644-aa4a99d10a82","request_id":"39114067-7a1a-922c-a644-aa4a99d10a82"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-990debd2-1cd5-9a58-bcdd-25c869eff398","request_id":"990debd2-1cd5-9a58-bcdd-25c869eff398"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a120e610-660b-96ca-88c6-424ed208ff45","request_id":"a120e610-660b-96ca-88c6-424ed208ff45"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dbfd5e9d-9713-9e2f-91c7-fdca4b156c97","request_id":"dbfd5e9d-9713-9e2f-91c7-fdca4b156c97"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-eff6f8c1-c82d-9dd8-9a7b-bd48a4c14265","request_id":"eff6f8c1-c82d-9dd8-9a7b-bd48a4c14265"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-01d2cab8-7498-9c05-803f-dec5dce55688"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9f72e7d2-a909-93ff-bc77-15e32640753b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ee643151-df10-9934-b31a-ebb8a3ac8f60"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5f8933cd-5231-994b-816e-3b0b23cedff9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4218a4cb-57b4-9923-ba19-836021fbd518"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6f023067-ae1c-98a2-90b1-e5e62f7f420a"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-08402140-52b8-9109-99b8-f15abb3ce20f","request_id":"08402140-52b8-9109-99b8-f15abb3ce20f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-eec51194-3be6-9ec3-a01c-32d9d5d671a7","request_id":"eec51194-3be6-9ec3-a01c-32d9d5d671a7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9d362a93-5b99-9a44-9ad2-05c51c9c511c","request_id":"9d362a93-5b99-9a44-9ad2-05c51c9c511c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2ad74106-eec3-9a53-b74b-362e4a153993","request_id":"2ad74106-eec3-9a53-b74b-362e4a153993"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3ce49c77-354e-90be-9ce0-5f8dc7ab7d95","request_id":"3ce49c77-354e-90be-9ce0-5f8dc7ab7d95"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2b1d18ff-0888-9d5c-8edb-a65d760ca32d","request_id":"2b1d18ff-0888-9d5c-8edb-a65d760ca32d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-89bb9711-b213-9e47-9c30-97d379f34a5e","request_id":"89bb9711-b213-9e47-9c30-97d379f34a5e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-08add916-5bf1-952e-9e64-624d15f9e711","request_id":"08add916-5bf1-952e-9e64-624d15f9e711"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-da0b90ce-ac4a-92c3-8070-30b84b6ec40e","request_id":"da0b90ce-ac4a-92c3-8070-30b84b6ec40e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e7a46db6-cebc-9550-af2c-0e8763bb3abb","request_id":"e7a46db6-cebc-9550-af2c-0e8763bb3abb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: feature-auth feature-db{ "branches_to_delete": ["feature-auth", "feature-db"] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: feature-auth feature-db
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: feature-auth feature-db
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: feature-auth feature-db{"branches_to_delete": ["feature-auth", "feature-db"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 128 in → 6 out (0 reasoning)
feature-api feature-ui
Failure: Missing: ['feature-auth', 'feature-db'] Extra: ['feature-api', 'feature-ui']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 128 in → 22 out (0 reasoning)
feature-api feature-ui
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-api", "feature-ui" ] }
Failure: Missing: ['feature-auth', 'feature-db'] Extra: ['feature-api', 'feature-ui']