Mixed merged and unmerged with many branches
Tests ability to process many branches (10+) with mixed merge status. Evaluates scalability of branch cleanup analysis.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial'
  7. 07 git checkout -b fix-typo
  8. 08 echo 'typo fix' > typo.txt
  9. 09 git add typo.txt
  10. 10 git commit -m 'Fix typo'
  11. 11 git checkout main
  12. 12 git merge fix-typo
  13. 13 git checkout -b fix-bug
  14. 14 echo 'bug fix' > bug.txt
  15. 15 git add bug.txt
  16. 16 git commit -m 'Fix bug'
  17. 17 git checkout main
  18. 18 git merge fix-bug
  19. 19 git checkout -b feature-v2
  20. 20 echo 'v2' > v2.txt
  21. 21 git add v2.txt
  22. 22 git commit -m 'Start v2'
  23. 23 git checkout main
  24. 24 git checkout -b feature-v3
  25. 25 echo 'v3' > v3.txt
  26. 26 git add v3.txt
  27. 27 git commit -m 'Start v3'
  28. 28 git checkout main
  29. 29 git checkout -b docs-update
  30. 30 echo 'docs' > docs.txt
  31. 31 git add docs.txt
  32. 32 git commit -m 'Update docs'
  33. 33 git checkout main
  34. 34 git merge docs-update
Prompt
Look at the branches in this repository. Which branches should be deleted because they are fully merged into main? List ONLY the branch names to delete, one per line. If none, respond with 'none'.
Expected
docs-update fix-bug fix-typo

Scoped model quality, cost, API time, and token usage for branch_cleanup/f012.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 200 in → 48 out (39 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-fable-5:high__json_schema PASS 100% 439 in → 62 out (32 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
anthropic/claude-fable-5:low PASS 100% 200 in → 28 out (23 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-fable-5:low__json_schema PASS 100% 436 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-fable-5:max PASS 100% 202 in → 436 out (192 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-fable-5:max__json_schema PASS 100% 436 in → 242 out (72 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
anthropic/claude-fable-5:medium PASS 100% 200 in → 28 out (23 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-fable-5:medium__json_schema PASS 100% 436 in → 35 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
anthropic/claude-fable-5:xhigh PASS 100% 199 in → 64 out (47 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 443 in → 65 out (33 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
anthropic/claude-haiku-4.5:high PASS 100% 189 in → 246 out (178 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 380 in → 431 out (338 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
anthropic/claude-haiku-4.5:low PASS 100% 190 in → 192 out (161 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 385 in → 205 out (166 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
anthropic/claude-haiku-4.5:medium PASS 100% 190 in → 243 out (173 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 380 in → 410 out (321 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
anthropic/claude-haiku-4.5:none PASS 100% 161 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 349 in → 27 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 192 in → 214 out (155 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 377 in → 468 out (379 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
anthropic/claude-opus-4.6:high PASS 100% 161 in → 80 out (45 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.6:high__json_schema PASS 100% 351 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.6:low PASS 100% 162 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.6:low__json_schema PASS 100% 351 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.6:max PASS 100% 160 in → 122 out (89 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.6:max__json_schema PASS 100% 349 in → 128 out (86 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.6:medium PASS 100% 159 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 348 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.6:none PASS 100% 158 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.6:none__json_schema PASS 100% 353 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 160 in → 88 out (54 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 348 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.7:high PASS 100% 206 in → 20 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.7:high__json_schema PASS 100% 442 in → 34 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.7:low PASS 100% 201 in → 20 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.7:low__json_schema PASS 100% 448 in → 34 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.7:max PASS 100% 209 in → 200 out (66 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.7:max__json_schema PASS 100% 442 in → 35 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.7:medium PASS 100% 199 in → 20 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 444 in → 34 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.7:none PASS 100% 208 in → 20 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.7:none__json_schema PASS 100% 440 in → 34 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 205 in → 20 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 441 in → 34 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.8:high PASS 100% 197 in → 84 out (31 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.8:high__json_schema PASS 100% 437 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.8:low PASS 100% 201 in → 18 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.8:low__json_schema PASS 100% 443 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.8:max FAIL 100% 194 in → 618 out (56 reasoning)
Based on the `git branch --merged main` output, these branches are fully merged into main and can be safely deleted (excluding `main` itself, which is the current/protected branch): docs-update fix-bug fix-typo **Note:** `feature-v2` and `feature-v3` are NOT in the merged list, so they contain unmerged work and should be kept.
Failure: Extra: ['**Note:** `feature-v2` and `feature-v3` are NOT in the merged list, so they contain unmerged work and should be kept.', 'Based on the `git branch --merged main` output, these branches are fully merged into main and can be safely deleted (excluding `main` itself, which is the current/protected branch):']
anthropic/claude-opus-4.8:max__json_schema PASS 100% 447 in → 352 out (77 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.8:medium PASS 100% 199 in → 65 out (25 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 443 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.8:none PASS 100% 196 in → 18 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-opus-4.8:none__json_schema PASS 100% 436 in → 31 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-opus-4.8:xhigh FAIL 100% 201 in → 317 out (76 reasoning)
Based on the "Merged into main" list, these branches are fully merged and can be safely deleted (excluding `main` itself): docs-update fix-bug fix-typo Note: `feature-v2` and `feature-v3` are NOT in the merged list, so they contain unmerged work and should be kept. `main` is excluded since it's the branch you're keeping.
Failure: Extra: ['Based on the "Merged into main" list, these branches are fully merged and can be safely deleted (excluding `main` itself):', "Note: `feature-v2` and `feature-v3` are NOT in the merged list, so they contain unmerged work and should be kept. `main` is excluded since it's the branch you're keeping."]
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 439 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-4.6:high PASS 100% 167 in → 115 out (85 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 350 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-4.6:low PASS 100% 162 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 352 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-4.6:max PASS 100% 160 in → 123 out (93 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 353 in → 122 out (80 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 157 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 349 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-4.6:none PASS 100% 163 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 352 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 161 in → 120 out (88 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 349 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-5:high PASS 100% 201 in → 123 out (52 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-5:high__json_schema PASS 100% 443 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-5:low PASS 100% 194 in → 18 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-5:low__json_schema PASS 100% 436 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-5:medium PASS 100% 201 in → 18 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 441 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-5:none PASS 100% 198 in → 18 out (0 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-5:none__json_schema PASS 100% 437 in → 31 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 197 in → 127 out (78 reasoning)
docs-update fix-bug fix-typo
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 442 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
arcee-ai/trinity-large-thinking:high PASS 100% 149 in → 535 out (521 reasoning)
docs-update fix-bug fix-typo
arcee-ai/trinity-large-thinking:low PASS 100% 144 in → 599 out (581 reasoning)
docs-update fix-bug fix-typo
arcee-ai/trinity-large-thinking:medium PASS 100% 154 in → 448 out (432 reasoning)
docs-update fix-bug fix-typo
arcee-ai/trinity-large-thinking:xhigh PASS 100% 147 in → 467 out (449 reasoning)
docs-update fix-bug fix-typo
arcee-ai/trinity-mini:high PASS 100% 147 in → 412 out (427 reasoning)
docs-update fix-bug fix-typo
arcee-ai/trinity-mini:high__json_schema PASS 100% 149 in → 376 out (376 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
arcee-ai/trinity-mini:low PASS 100% 149 in → 282 out (287 reasoning)
docs-update fix-bug fix-typo
arcee-ai/trinity-mini:low__json_schema PASS 100% 151 in → 299 out (289 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
arcee-ai/trinity-mini:medium PASS 100% 149 in → 684 out (744 reasoning)
docs-update fix-bug fix-typo
arcee-ai/trinity-mini:medium__json_schema PASS 100% 147 in → 287 out (283 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 156 in → 294 out (298 reasoning)
docs-update fix-bug fix-typo
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 147 in → 333 out (325 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 225 in → 203 out (190 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 230 in → 200 out (173 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 148 in → 312 out (346 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 150 in → 76 out (57 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 228 in → 184 out (181 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 244 in → 248 out (217 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 157 in → 242 out (233 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 153 in → 72 out (57 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 155 in → 13 out (0 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 149 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 224 in → 213 out (182 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 151 in → 209 out (172 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
deepseek/deepseek-v4-flash:high PASS 100% 149 in → 157 out (143 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 153 in → 253 out (226 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
deepseek/deepseek-v4-flash:low PASS 100% 149 in → 196 out (182 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 147 in → 171 out (123 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-flash:medium PASS 100% 152 in → 137 out (125 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 252 in → 24 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
deepseek/deepseek-v4-flash:none PASS 100% 147 in → 13 out (0 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 231 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
deepseek/deepseek-v4-flash:xhigh PASS 100% 227 in → 237 out (223 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 222 in → 232 out (201 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-pro:high PASS 100% 153 in → 311 out (297 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 150 in → 386 out (353 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-pro:low PASS 100% 153 in → 13 out (0 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 351 in → 313 out (288 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
deepseek/deepseek-v4-pro:medium PASS 100% 150 in → 318 out (304 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 152 in → 197 out (164 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-pro:none PASS 100% 150 in → 13 out (0 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 148 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 146 in → 13 out (0 reasoning)
docs-update fix-bug fix-typo
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 428 in → 923 out (898 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
google/gemini-3-flash-preview:high PASS 100% 158 in → 351 out (339 reasoning)
docs-update fix-bug fix-typo
google/gemini-3-flash-preview:high__json_schema PASS 100% 233 in → 474 out (441 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemini-3-flash-preview:low PASS 100% 159 in → 416 out (404 reasoning)
docs-update fix-bug fix-typo
google/gemini-3-flash-preview:low__json_schema PASS 100% 238 in → 403 out (370 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemini-3-flash-preview:medium PASS 100% 155 in → 383 out (371 reasoning)
docs-update fix-bug fix-typo
google/gemini-3-flash-preview:medium__json_schema PASS 100% 231 in → 530 out (510 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3-flash-preview:xhigh PASS 100% 158 in → 419 out (407 reasoning)
docs-update fix-bug fix-typo
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 234 in → 372 out (352 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 155 in → 472 out (460 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 152 in → 506 out (486 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 155 in → 127 out (115 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 159 in → 137 out (115 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 157 in → 215 out (203 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 236 in → 302 out (275 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 160 in → 578 out (566 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 157 in → 432 out (412 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.1-pro-preview:high PASS 100% 156 in → 503 out (491 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 235 in → 256 out (218 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemini-3.1-pro-preview:low PASS 100% 158 in → 259 out (247 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 234 in → 151 out (113 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemini-3.1-pro-preview:medium PASS 100% 156 in → 366 out (354 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 236 in → 113 out (90 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 156 in → 433 out (421 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 237 in → 378 out (359 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.5-flash-lite:high PASS 100% 156 in → 469 out (457 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 241 in → 459 out (436 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
google/gemini-3.5-flash-lite:low PASS 100% 157 in → 12 out (0 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 156 in → 38 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemini-3.5-flash-lite:max PASS 100% 157 in → 508 out (496 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 160 in → 427 out (407 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.5-flash-lite:medium PASS 100% 158 in → 395 out (383 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 154 in → 334 out (311 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 160 in → 459 out (447 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 235 in → 430 out (410 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.5-flash:high PASS 100% 159 in → 449 out (437 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.5-flash:high__json_schema PASS 100% 240 in → 535 out (515 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.5-flash:low PASS 100% 158 in → 365 out (353 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.5-flash:low__json_schema PASS 100% 241 in → 117 out (97 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.5-flash:medium PASS 100% 157 in → 526 out (514 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.5-flash:medium__json_schema PASS 100% 240 in → 441 out (421 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.5-flash:xhigh PASS 100% 157 in → 589 out (577 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 234 in → 276 out (256 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.6-flash:high PASS 100% 158 in → 415 out (403 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.6-flash:high__json_schema PASS 100% 158 in → 396 out (376 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.6-flash:low PASS 100% 156 in → 97 out (85 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.6-flash:low__json_schema PASS 100% 158 in → 151 out (131 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.6-flash:max PASS 100% 158 in → 420 out (408 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.6-flash:max__json_schema PASS 100% 157 in → 344 out (324 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.6-flash:medium PASS 100% 155 in → 382 out (370 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.6-flash:medium__json_schema PASS 100% 159 in → 383 out (363 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemini-3.6-flash:xhigh PASS 100% 158 in → 437 out (425 reasoning)
docs-update fix-bug fix-typo
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 242 in → 403 out (383 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
google/gemma-4-26b-a4b-it:high PASS 100% 171 in → 410 out (294 reasoning)
docs-update fix-bug fix-typo
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 173 in → 425 out (299 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 165 in → 312 out (248 reasoning)
docs-update fix-bug fix-typo
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 175 in → 455 out (315 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 172 in → 470 out (358 reasoning)
docs-update fix-bug fix-typo
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 170 in → 457 out (323 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 166 in → 13 out (0 reasoning)
docs-update fix-bug fix-typo
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 171 in → 28 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 175 in → 359 out (291 reasoning)
docs-update fix-bug fix-typo
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 175 in → 459 out (339 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
google/gemma-4-31b-it:high PASS 100% 172 in → 385 out (1 reasoning)
docs-update fix-bug fix-typo
google/gemma-4-31b-it:high__json_schema PASS 100% 174 in → 313 out (208 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemma-4-31b-it:low PASS 100% 173 in → 349 out (1 reasoning)
docs-update fix-bug fix-typo
google/gemma-4-31b-it:low__json_schema PASS 100% 171 in → 351 out (219 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
google/gemma-4-31b-it:medium PASS 100% 176 in → 373 out (360 reasoning)
docs-update fix-bug fix-typo
google/gemma-4-31b-it:medium__json_schema PASS 100% 170 in → 373 out (238 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
google/gemma-4-31b-it:none PASS 100% 167 in → 13 out (0 reasoning)
docs-update fix-bug fix-typo
google/gemma-4-31b-it:none__json_schema PASS 100% 171 in → 24 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
google/gemma-4-31b-it:xhigh PASS 100% 174 in → 347 out (1 reasoning)
docs-update fix-bug fix-typo
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 172 in → 293 out (193 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
inclusionai/ling-2.6-flash PASS 100% 179 in → 12 out
docs-update fix-bug fix-typo
inclusionai/ling-2.6-flash__json_schema PASS 100% 180 in → 37 out
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
minimax/minimax-m2.5:high PASS 100% 183 in → 441 out (428 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m2.5:low PASS 100% 181 in → 258 out (240 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m2.5:low__json_schema PASS 100% 197 in → 293 out (243 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
minimax/minimax-m2.5:medium PASS 100% 181 in → 410 out (398 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m2.5:medium__json_schema PASS 100% 191 in → 464 out (438 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
minimax/minimax-m2.5:xhigh PASS 100% 185 in → 362 out (363 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 189 in → 499 out (467 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
minimax/minimax-m2.7:high PASS 100% 188 in → 338 out (329 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m2.7:low PASS 100% 178 in → 342 out (345 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m2.7:low__json_schema PASS 100% 333 in → 357 out (325 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
minimax/minimax-m2.7:medium PASS 100% 185 in → 301 out (289 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m2.7:medium__json_schema PASS 100% 328 in → 449 out (428 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
minimax/minimax-m2.7:xhigh PASS 100% 166 in → 107 out (105 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 335 in → 605 out (579 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
minimax/minimax-m3:high PASS 100% 319 in → 231 out (212 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m3:high__json_schema PASS 100% 308 in → 31 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
minimax/minimax-m3:low PASS 100% 319 in → 180 out (152 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m3:low__json_schema PASS 100% 308 in → 31 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
minimax/minimax-m3:medium PASS 100% 317 in → 183 out (164 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m3:medium__json_schema PASS 100% 306 in → 31 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
minimax/minimax-m3:xhigh PASS 100% 321 in → 130 out (120 reasoning)
docs-update fix-bug fix-typo
minimax/minimax-m3:xhigh__json_schema PASS 100% 305 in → 31 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
mistralai/devstral-2512 PASS 100% 157 in → 12 out
docs-update fix-bug fix-typo
mistralai/devstral-2512__json_schema PASS 100% 157 in → 23 out
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
mistralai/mistral-medium-3-5:high PASS 100% 175 in → 326 out (327 reasoning)
docs-update fix-bug fix-typo
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 161 in → 263 out (211 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
mistralai/mistral-medium-3-5:low PASS 100% 173 in → 563 out (540 reasoning)
docs-update fix-bug fix-typo
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 161 in → 288 out (249 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
mistralai/mistral-medium-3-5:medium PASS 100% 175 in → 217 out (199 reasoning)
docs-update fix-bug fix-typo
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 161 in → 250 out (203 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
mistralai/mistral-medium-3-5:none PASS 100% 173 in → 12 out (0 reasoning)
docs-update fix-bug fix-typo
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 157 in → 23 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 171 in → 187 out (178 reasoning)
docs-update fix-bug fix-typo
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 163 in → 220 out (205 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
moonshotai/kimi-k2.6:high PASS 100% 147 in → 306 out (293 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k2.6:high__json_schema PASS 100% 146 in → 451 out (434 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
moonshotai/kimi-k2.6:low PASS 100% 149 in → 460 out (449 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k2.6:low__json_schema PASS 100% 149 in → 369 out (338 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
moonshotai/kimi-k2.6:medium PASS 100% 147 in → 348 out (335 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k2.6:xhigh PASS 100% 147 in → 357 out (359 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k2.7-code:high PASS 100% 147 in → 177 out (175 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 150 in → 202 out (135 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
moonshotai/kimi-k2.7-code:low PASS 100% 152 in → 117 out (104 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 147 in → 310 out (134 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
moonshotai/kimi-k2.7-code:medium PASS 100% 152 in → 109 out (97 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 244 in → 78 out (54 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 148 in → 144 out (131 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 244 in → 120 out (96 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
moonshotai/kimi-k3:high PASS 100% 227 in → 218 out (192 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k3:high__json_schema PASS 100% 225 in → 158 out (132 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k3:low PASS 100% 227 in → 151 out (125 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k3:low__json_schema PASS 100% 226 in → 212 out (186 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k3:max PASS 100% 229 in → 134 out (108 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k3:max__json_schema PASS 100% 227 in → 134 out (108 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k3:medium PASS 100% 225 in → 120 out (94 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k3:medium__json_schema PASS 100% 228 in → 152 out (126 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k3:xhigh PASS 100% 222 in → 118 out (92 reasoning)
docs-update fix-bug fix-typo
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 224 in → 207 out (181 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 173 in → 126 out (117 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 170 in → 115 out (87 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 171 in → 102 out (89 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 171 in → 206 out (196 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 176 in → 93 out (81 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 175 in → 199 out (160 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 168 in → 12 out (0 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 169 in → 31 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 174 in → 133 out (110 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 171 in → 158 out (120 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 176 in → 125 out (113 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 173 in → 121 out (85 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 175 in → 82 out (66 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 175 in → 222 out (171 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 173 in → 122 out (115 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 171 in → 134 out (85 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 171 in → 12 out (0 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 171 in → 25 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 172 in → 74 out (57 reasoning)
docs-update fix-bug fix-typo
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 173 in → 110 out (75 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
openai/gpt-5.4-mini:high PASS 100% 152 in → 78 out (61 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4-mini:high__json_schema PASS 100% 191 in → 173 out (145 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4-mini:low PASS 100% 150 in → 50 out (33 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4-mini:low__json_schema PASS 100% 186 in → 57 out (29 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4-mini:medium PASS 100% 144 in → 58 out (41 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4-mini:medium__json_schema PASS 100% 189 in → 102 out (74 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4-mini:none PASS 100% 149 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4-mini:none__json_schema PASS 100% 186 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4-mini:xhigh PASS 100% 148 in → 76 out (59 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 187 in → 1,015 out (987 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4-nano:high PASS 100% 151 in → 52 out (35 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4-nano:high__json_schema PASS 100% 195 in → 71 out (43 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4-nano:low PASS 100% 154 in → 40 out (23 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4-nano:low__json_schema PASS 100% 189 in → 62 out (34 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4-nano:medium PASS 100% 147 in → 51 out (34 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4-nano:medium__json_schema PASS 100% 188 in → 74 out (46 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4-nano:none PASS 100% 150 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4-nano:none__json_schema PASS 100% 185 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4-nano:xhigh PASS 100% 146 in → 112 out (95 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 190 in → 296 out (268 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4:high PASS 100% 144 in → 71 out (54 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4:high__json_schema PASS 100% 192 in → 101 out (73 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4:low PASS 100% 156 in → 38 out (21 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4:low__json_schema PASS 100% 192 in → 61 out (33 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4:medium PASS 100% 147 in → 59 out (42 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4:medium__json_schema PASS 100% 192 in → 80 out (52 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4:none PASS 100% 146 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4:none__json_schema PASS 100% 187 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.4:xhigh PASS 100% 148 in → 91 out (74 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.4:xhigh__json_schema PASS 100% 190 in → 124 out (96 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.5:high PASS 100% 154 in → 92 out (75 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.5:high__json_schema PASS 100% 188 in → 104 out (76 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.5:low PASS 100% 146 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.5:low__json_schema PASS 100% 193 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.5:medium PASS 100% 148 in → 66 out (49 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.5:medium__json_schema PASS 100% 195 in → 244 out (216 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.5:none PASS 100% 149 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.5:none__json_schema PASS 100% 196 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.5:xhigh PASS 100% 152 in → 115 out (98 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.5:xhigh__json_schema PASS 100% 190 in → 118 out (90 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-luna:high PASS 100% 152 in → 45 out (28 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-luna:high__json_schema PASS 100% 188 in → 81 out (53 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-luna:low PASS 100% 151 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-luna:low__json_schema PASS 100% 196 in → 59 out (31 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-luna:max PASS 100% 148 in → 83 out (66 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-luna:max__json_schema PASS 100% 189 in → 538 out (510 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-luna:medium PASS 100% 149 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-luna:medium__json_schema PASS 100% 192 in → 141 out (113 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-luna:none PASS 100% 150 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-luna:none__json_schema PASS 100% 191 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-luna:xhigh PASS 100% 148 in → 61 out (44 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 192 in → 342 out (314 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-sol:high PASS 100% 145 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-sol:high__json_schema PASS 100% 190 in → 118 out (90 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-sol:low PASS 100% 146 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-sol:low__json_schema PASS 100% 189 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-sol:max PASS 100% 146 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-sol:max__json_schema PASS 100% 190 in → 131 out (103 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-sol:medium PASS 100% 149 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-sol:medium__json_schema PASS 100% 192 in → 90 out (62 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-sol:none PASS 100% 142 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-sol:none__json_schema PASS 100% 195 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-sol:xhigh PASS 100% 142 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 188 in → 146 out (118 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-terra:high PASS 100% 148 in → 45 out (28 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-terra:high__json_schema PASS 100% 193 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-terra:low PASS 100% 150 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-terra:low__json_schema PASS 100% 193 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-terra:max PASS 100% 149 in → 69 out (52 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-terra:max__json_schema PASS 100% 194 in → 468 out (440 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-terra:medium PASS 100% 148 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-terra:medium__json_schema PASS 100% 191 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-terra:none PASS 100% 147 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-terra:none__json_schema PASS 100% 191 in → 26 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-5.6-terra:xhigh PASS 100% 147 in → 61 out (44 reasoning)
docs-update fix-bug fix-typo
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 194 in → 52 out (24 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-oss-120b:high PASS 100% 208 in → 265 out (246 reasoning)
docs-update fix-bug fix-typo
openai/gpt-oss-120b:high__json_schema PASS 100% 202 in → 387 out (375 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
openai/gpt-oss-120b:low PASS 100% 210 in → 54 out (32 reasoning)
docs-update fix-bug fix-typo
openai/gpt-oss-120b:low__json_schema PASS 100% 208 in → 71 out (31 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
openai/gpt-oss-120b:medium PASS 100% 194 in → 195 out (190 reasoning)
docs-update fix-bug fix-typo
openai/gpt-oss-120b:medium__json_schema PASS 100% 209 in → 218 out (176 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
openai/gpt-oss-120b:xhigh PASS 100% 207 in → 268 out (248 reasoning)
docs-update fix-bug fix-typo
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 213 in → 354 out (324 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-oss-20b:high PASS 100% 205 in → 571 out (572 reasoning)
docs-update fix-bug fix-typo
openai/gpt-oss-20b:high__json_schema PASS 100% 199 in → 396 out (388 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
openai/gpt-oss-20b:low PASS 100% 205 in → 50 out (31 reasoning)
docs-update fix-bug fix-typo
openai/gpt-oss-20b:low__json_schema PASS 100% 209 in → 64 out (25 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
openai/gpt-oss-20b:medium PASS 100% 209 in → 267 out (229 reasoning)
docs-update fix-bug fix-typo
openai/gpt-oss-20b:medium__json_schema PASS 100% 208 in → 214 out (180 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
openai/gpt-oss-20b:xhigh PASS 100% 209 in → 406 out (367 reasoning)
docs-update fix-bug fix-typo
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 205 in → 505 out (464 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
poolside/laguna-m.1:high PASS 100% 168 in → 265 out (253 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-m.1:high__json_schema PASS 100% 169 in → 382 out (362 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-m.1:low PASS 100% 163 in → 396 out (384 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-m.1:low__json_schema PASS 100% 171 in → 460 out (440 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-m.1:medium PASS 100% 166 in → 195 out (183 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-m.1:medium__json_schema PASS 100% 169 in → 294 out (274 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-m.1:none PASS 100% 170 in → 11 out (0 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-m.1:none__json_schema PASS 100% 167 in → 19 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-m.1:xhigh PASS 100% 167 in → 289 out (277 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-m.1:xhigh__json_schema PASS 100% 169 in → 260 out (225 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
poolside/laguna-xs-2.1:high PASS 100% 164 in → 237 out (227 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-xs-2.1:high__json_schema PASS 100% 171 in → 182 out (162 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-xs-2.1:low PASS 100% 163 in → 374 out (364 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-xs-2.1:low__json_schema PASS 100% 163 in → 292 out (272 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-xs-2.1:medium PASS 100% 165 in → 239 out (229 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 165 in → 222 out (186 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
poolside/laguna-xs-2.1:none PASS 100% 170 in → 9 out (0 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-xs-2.1:none__json_schema PASS 100% 170 in → 19 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-xs-2.1:xhigh PASS 100% 167 in → 198 out (188 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 168 in → 197 out (177 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-xs.2:high PASS 100% 205 in → 208 out (196 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-xs.2:high__json_schema PASS 100% 199 in → 239 out (219 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-xs.2:low PASS 100% 199 in → 369 out (357 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-xs.2:low__json_schema PASS 100% 205 in → 267 out (241 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
poolside/laguna-xs.2:medium PASS 100% 201 in → 200 out (188 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-xs.2:medium__json_schema PASS 100% 208 in → 260 out (240 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-xs.2:none PASS 100% 206 in → 11 out (0 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-xs.2:none__json_schema PASS 100% 199 in → 19 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
poolside/laguna-xs.2:xhigh PASS 100% 206 in → 167 out (155 reasoning)
docs-update fix-bug fix-typo
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 205 in → 207 out (187 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
qwen/qwen3.6-27b:high PASS 100% 168 in → 1,022 out (1,007 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-27b:high__json_schema PASS 100% 158 in → 925 out (865 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
qwen/qwen3.6-27b:low PASS 100% 165 in → 835 out (763 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-27b:low__json_schema PASS 100% 164 in → 524 out (437 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
qwen/qwen3.6-27b:medium PASS 100% 161 in → 739 out (680 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-27b:medium__json_schema PASS 100% 164 in → 1,037 out (1 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
qwen/qwen3.6-27b:none PASS 100% 168 in → 11 out (0 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-27b:none__json_schema PASS 100% 162 in → 35 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 165 in → 1,172 out (1,110 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 168 in → 1,008 out (907 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 166 in → 927 out (912 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 171 in → 997 out (969 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
qwen/qwen3.6-35b-a3b:low PASS 100% 169 in → 747 out (705 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 166 in → 842 out (736 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
qwen/qwen3.6-35b-a3b:medium PASS 100% 162 in → 1,087 out (1,015 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 166 in → 911 out (869 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 167 in → 11 out (0 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 169 in → 35 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 166 in → 1,126 out (1,111 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 164 in → 765 out (722 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
qwen/qwen3.6-flash:high PASS 100% 164 in → 925 out (910 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-flash:low PASS 100% 165 in → 1,051 out (1,036 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-flash:medium PASS 100% 158 in → 779 out (764 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-flash:none PASS 100% 166 in → 10 out (0 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.6-flash:xhigh PASS 100% 164 in → 1,188 out (1,173 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-flash:high PASS 100% 160 in → 460 out (445 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-flash:low PASS 100% 165 in → 749 out (734 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-flash:max PASS 100% 158 in → 715 out (700 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-flash:medium PASS 100% 167 in → 999 out (984 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-flash:none PASS 100% 169 in → 10 out (0 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-flash:xhigh PASS 100% 166 in → 811 out (796 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-max:high PASS 100% 165 in → 204 out (189 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-max:low PASS 100% 161 in → 202 out (187 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-max:medium PASS 100% 161 in → 685 out (670 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-max:none PASS 100% 169 in → 10 out (0 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-max:xhigh PASS 100% 166 in → 202 out (187 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-plus:high PASS 100% 166 in → 494 out (479 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-plus:low PASS 100% 166 in → 734 out (719 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-plus:medium PASS 100% 167 in → 540 out (525 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-plus:none PASS 100% 168 in → 10 out (0 reasoning)
docs-update fix-bug fix-typo
qwen/qwen3.7-plus:xhigh PASS 100% 166 in → 501 out (486 reasoning)
docs-update fix-bug fix-typo
tencent/hy3:high PASS 100% 157 in → 427 out (413 reasoning)
docs-update fix-bug fix-typo
tencent/hy3:high__json_schema PASS 100% 154 in → 605 out (559 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
tencent/hy3:low PASS 100% 152 in → 358 out (324 reasoning)
docs-update fix-bug fix-typo
tencent/hy3:low__json_schema PASS 100% 160 in → 487 out (407 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
tencent/hy3:medium PASS 100% 154 in → 357 out (343 reasoning)
docs-update fix-bug fix-typo
tencent/hy3:medium__json_schema PASS 100% 157 in → 406 out (336 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
tencent/hy3:none PASS 100% 159 in → 13 out (0 reasoning)
docs-update fix-bug fix-typo
tencent/hy3:none__json_schema PASS 100% 162 in → 32 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
tencent/hy3:xhigh PASS 100% 152 in → 385 out (355 reasoning)
docs-update fix-bug fix-typo
tencent/hy3:xhigh__json_schema PASS 100% 157 in → 595 out (545 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
thinkingmachines/inkling-small:high PASS 100% 153 in → 133 out (116 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling-small:high__json_schema PASS 100% 159 in → 151 out (124 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
thinkingmachines/inkling-small:low PASS 100% 155 in → 38 out (19 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling-small:low__json_schema PASS 100% 156 in → 65 out (38 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
thinkingmachines/inkling-small:max PASS 100% 155 in → 249 out (216 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling-small:max__json_schema PASS 100% 152 in → 280 out (245 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
thinkingmachines/inkling-small:medium PASS 100% 155 in → 174 out (126 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling-small:medium__json_schema PASS 100% 154 in → 200 out (166 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 157 in → 193 out (176 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 156 in → 245 out (211 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
thinkingmachines/inkling:high PASS 100% 156 in → 225 out (208 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling:low PASS 100% 158 in → 36 out (19 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling:max PASS 100% 156 in → 202 out (175 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling:max__json_schema PASS 100% 162 in → 256 out (222 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
thinkingmachines/inkling:medium PASS 100% 157 in → 151 out (132 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling:medium__json_schema PASS 100% 159 in → 183 out (154 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
thinkingmachines/inkling:none PASS 100% 150 in → 15 out (0 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling:none__json_schema PASS 100% 153 in → 33 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
thinkingmachines/inkling:xhigh PASS 100% 160 in → 208 out (191 reasoning)
docs-update fix-bug fix-typo
thinkingmachines/inkling:xhigh__json_schema PASS 100% 151 in → 236 out (202 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
x-ai/grok-4.3:high PASS 100% 326 in → 227 out (217 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.3:high__json_schema PASS 100% 390 in → 437 out (415 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
x-ai/grok-4.3:low PASS 100% 325 in → 330 out (320 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.3:low__json_schema PASS 100% 391 in → 319 out (297 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
x-ai/grok-4.3:max PASS 100% 326 in → 360 out (350 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.3:max__json_schema PASS 100% 392 in → 479 out (463 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
x-ai/grok-4.3:medium PASS 100% 323 in → 534 out (524 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.3:medium__json_schema PASS 100% 392 in → 403 out (375 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
x-ai/grok-4.3:none PASS 100% 321 in → 10 out (0 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.3:none__json_schema PASS 100% 386 in → 16 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
x-ai/grok-4.3:xhigh PASS 100% 322 in → 403 out (393 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.3:xhigh__json_schema PASS 100% 395 in → 484 out (468 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
x-ai/grok-4.5:high PASS 100% 358 in → 436 out (424 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.5:high__json_schema PASS 100% 436 in → 377 out (356 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
x-ai/grok-4.5:low PASS 100% 353 in → 249 out (237 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.5:low__json_schema PASS 100% 436 in → 342 out (321 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
x-ai/grok-4.5:max PASS 100% 352 in → 513 out (501 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.5:max__json_schema PASS 100% 435 in → 386 out (365 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
x-ai/grok-4.5:medium PASS 100% 352 in → 392 out (380 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.5:medium__json_schema PASS 100% 435 in → 438 out (417 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
x-ai/grok-4.5:xhigh PASS 100% 351 in → 479 out (467 reasoning)
docs-update fix-bug fix-typo
x-ai/grok-4.5:xhigh__json_schema PASS 100% 439 in → 431 out (410 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete":["docs-update","fix-bug","fix-typo"]}
z-ai/glm-4.7-flash:high PASS 100% 150 in → 629 out (546 reasoning)
docs-update fix-bug fix-typo
z-ai/glm-4.7-flash:low PASS 100% 150 in → 337 out (300 reasoning)
docs-update fix-bug fix-typo
z-ai/glm-4.7-flash:medium PASS 100% 149 in → 647 out (578 reasoning)
docs-update fix-bug fix-typo
z-ai/glm-4.7-flash:none PASS 100% 147 in → 11 out (0 reasoning)
docs-update fix-bug fix-typo
z-ai/glm-4.7-flash:xhigh PASS 100% 149 in → 779 out (720 reasoning)
docs-update fix-bug fix-typo
z-ai/glm-5.2:high PASS 100% 154 in → 288 out (276 reasoning)
docs-update fix-bug fix-typo
z-ai/glm-5.2:high__json_schema PASS 100% 159 in → 384 out (361 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
z-ai/glm-5.2:low PASS 100% 157 in → 219 out (196 reasoning)
docs-update fix-bug fix-typo
z-ai/glm-5.2:low__json_schema PASS 100% 160 in → 391 out (318 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": ["docs-update", "fix-bug", "fix-typo"] }
z-ai/glm-5.2:medium PASS 100% 156 in → 289 out (251 reasoning)
docs-update fix-bug fix-typo
z-ai/glm-5.2:none PASS 100% 152 in → 11 out (0 reasoning)
docs-update fix-bug fix-typo
z-ai/glm-5.2:none__json_schema PASS 100% 151 in → 22 out (0 reasoning)
docs-update fix-bug fix-typo
JSON Schema Structured Output
(raw) {"branches_to_delete": ["docs-update", "fix-bug", "fix-typo"]}
z-ai/glm-5.2:xhigh PASS 100% 157 in → 197 out (185 reasoning)
docs-update fix-bug fix-typo
ibm-granite/granite-4.1-8b__json_schema FAIL 66.7% 165 in → 27 out
docs-update fix-bug fix-typpo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix-bug", "fix-typpo" ] }
Failure: Missing: ['fix-typo'] Extra: ['fix-typpo']
minimax/minimax-m2.5:high__json_schema FAIL 66.7% 195 in → 1,541 out (1,509 reasoning)
docs-update fix- bug fix-typo
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "docs-update", "fix- bug", "fix-typo" ] }
Failure: Missing: ['fix-bug'] Extra: ['fix- bug']
thinkingmachines/inkling-small:none FAIL 66.7% 158 in → 14 out (0 reasoning)
docs-update fix-bug fix-typo<|end_message|>
Failure: Missing: ['fix-typo'] Extra: ['fix-typo<|end_message|>']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
ibm-granite/granite-4.1-8b FAIL 0% 160 in → 8 out
feature-v2 feature-v3
Failure: Missing: ['docs-update', 'fix-bug', 'fix-typo'] Extra: ['feature-v2', 'feature-v3']
liquid/lfm-2-24b-a2b FAIL 0% 165 in → 8 out
feature-v2 feature-v3
Failure: Missing: ['docs-update', 'fix-bug', 'fix-typo'] Extra: ['feature-v2', 'feature-v3']
Invalid JSON. Output: feature-v2 feature-v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: docs-update fix-bug fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: docs-update fix-bug fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fd8456b0-ffd0-9784-a63d-4123d1862698","request_id":"fd8456b0-ffd0-9784-a63d-4123d1862698"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ad54781b-2b8d-9292-8b0e-c216cc899b7f","request_id":"ad54781b-2b8d-9292-8b0e-c216cc899b7f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-321c0fe2-b0fd-9826-97f4-acd722cccb1c","request_id":"321c0fe2-b0fd-9826-97f4-acd722cccb1c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7ca83258-39d6-9d3a-bb31-805c3c7262e7","request_id":"7ca83258-39d6-9d3a-bb31-805c3c7262e7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0559fef3-80b9-963f-b33f-8f45f2a636f5","request_id":"0559fef3-80b9-963f-b33f-8f45f2a636f5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-27f59a4b-63b8-9627-bf27-7a4867c288cf"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8ce8f305-22f6-9790-a8d8-4ad1923a7165"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c476509e-e45e-9def-8647-533631720ebe"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-38f016d9-be4d-9137-84e4-3d59fec406fc"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7c96613e-6c84-92b0-8773-4c42e89d3d70"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-588c68b0-050b-96c6-a267-63a424b354c7"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b4708091-47e7-9814-b395-575cbb6491eb","request_id":"b4708091-47e7-9814-b395-575cbb6491eb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5bf66af9-67e3-9818-b9e0-be54910104e5","request_id":"5bf66af9-67e3-9818-b9e0-be54910104e5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a71da061-fa82-9eb0-a8a7-49789f6e19bd","request_id":"a71da061-fa82-9eb0-a8a7-49789f6e19bd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-82dbbe6a-1081-9c19-8461-b28b985c10f8","request_id":"82dbbe6a-1081-9c19-8461-b28b985c10f8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-60872085-8d45-973b-bf7b-9efa83f01f0c","request_id":"60872085-8d45-973b-bf7b-9efa83f01f0c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-111d8407-5dec-9c1a-a014-08276a783062","request_id":"111d8407-5dec-9c1a-a014-08276a783062"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d04cb07e-7e5e-95e1-a61f-f46b55edf5c9","request_id":"d04cb07e-7e5e-95e1-a61f-f46b55edf5c9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3ca03ebe-4b61-9e4f-bd2a-c86e0c8f9df8","request_id":"3ca03ebe-4b61-9e4f-bd2a-c86e0c8f9df8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-134b0b47-83c1-994f-81a5-a06546aea2b5","request_id":"134b0b47-83c1-994f-81a5-a06546aea2b5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-52d00030-2d4d-9fa9-977e-1153eef4bc82","request_id":"52d00030-2d4d-9fa9-977e-1153eef4bc82"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: docs-update fix-bug fix-typo{ "branches_to_delete": [ "docs-update", "fix-bug", "fix-typo" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: docs-update fix-bug fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: docs-update fix-bug fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0% 150 in → 711 out (606 reasoning)
feature-v2 feature-v3
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-v2", "feature-v3" ] }
Failure: Missing: ['docs-update', 'fix-bug', 'fix-typo'] Extra: ['feature-v2', 'feature-v3']
Invalid JSON. Output: docs-update fix-bug fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 152 in → 24 out (0 reasoning)
feature-v2 feature-v3
JSON Schema Structured Output
(raw) { "branches_to_delete": [ "feature-v2", "feature-v3" ] }
Failure: Missing: ['docs-update', 'fix-bug', 'fix-typo'] Extra: ['feature-v2', 'feature-v3']
Invalid JSON. Output: feature-v2 feature-v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: docs-update fix-bug fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: docs-update fix-bug fix-typo
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)