f002 — merge_conflicts

← merge_conflicts
Single-line version number conflict
Tests ability to resolve a version number merge conflict. Evaluates semantic versioning awareness in conflict resolution.
easy merge-conflict version-number semver resolution
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'VERSION=1.0.0' > version.txt
  5. 05 git add version.txt
  6. 06 git commit -m 'Initial version'
  7. 07 git checkout -b release
  8. 08 echo 'VERSION=2.0.0' > version.txt
  9. 09 git add version.txt
  10. 10 git commit -m 'Bump major version'
  11. 11 git checkout main
  12. 12 echo 'VERSION=1.1.0' > version.txt
  13. 13 git add version.txt
  14. 14 git commit -m 'Bump minor version'
  15. 15 git merge release
Prompt
Resolve the merge conflict in version.txt. The base version was 'VERSION=1.0.0'. Branch A (release) changed it to 'VERSION=2.0.0'. Branch B (main) changed it to 'VERSION=1.1.0'. Provide ONLY the resolved file content.
Expected
VERSION=2.0.0
Fixture Metrics

Scoped model quality, cost, API time, and token usage for merge_conflicts/f002.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (760)
anthropic/claude-fable-5.1:high PASS 100% 168 in → 273 out (261 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5.1:high__json_schema PASS 100% 388 in → 113 out (88 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-fable-5.1:low PASS 100% 168 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5.1:low__json_schema PASS 100% 388 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-fable-5.1:max PASS 100% 168 in → 500 out (488 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5.1:max__json_schema PASS 100% 388 in → 237 out (212 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-fable-5.1:medium PASS 100% 168 in → 149 out (137 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5.1:medium__json_schema PASS 100% 388 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-fable-5.1:xhigh PASS 100% 168 in → 210 out (198 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5.1:xhigh__json_schema PASS 100% 388 in → 148 out (123 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-fable-5:high PASS 100% 166 in → 65 out (32 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:high__json_schema PASS 100% 386 in → 50 out (14 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-fable-5:low PASS 100% 166 in → 28 out (10 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:low__json_schema PASS 100% 386 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-fable-5:max PASS 100% 166 in → 261 out (76 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:max__json_schema PASS 100% 386 in → 215 out (60 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-fable-5:medium PASS 100% 166 in → 37 out (20 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:medium__json_schema PASS 100% 386 in → 38 out (9 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-fable-5:xhigh PASS 100% 166 in → 93 out (67 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 386 in → 69 out (25 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-haiku-4.5:high PASS 100% 145 in → 583 out (498 reasoning)
VERSION=2.0.0
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 313 in → 4,319 out (1,672 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:low PASS 100% 145 in → 385 out (375 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 313 in → 987 out (660 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:medium PASS 100% 145 in → 704 out (595 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 313 in → 1,640 out (805 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:none PASS 100% 116 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 283 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 145 in → 705 out (506 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 313 in → 1,105 out (703 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-opus-4.6:high PASS 100% 116 in → 11 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.6:high__json_schema PASS 100% 284 in → 127 out (78 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.6:low PASS 100% 116 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.6:low__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 284 in → 195 out (141 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.6:medium PASS 100% 116 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.6:none__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.6:xhigh PASS 100% 116 in → 11 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 284 in → 128 out (77 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:high PASS 100% 171 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:high__json_schema PASS 100% 391 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.7:low PASS 100% 171 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:low__json_schema PASS 100% 391 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:max PASS 100% 171 in → 203 out (59 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:max__json_schema PASS 100% 391 in → 235 out (59 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:medium PASS 100% 171 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 391 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:none PASS 100% 171 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:none__json_schema PASS 100% 391 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 171 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 391 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.8:high PASS 100% 166 in → 169 out (54 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:high__json_schema PASS 100% 386 in → 86 out (29 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.8:low PASS 100% 166 in → 53 out (22 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:low__json_schema PASS 100% 386 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.8:max PASS 100% 166 in → 330 out (58 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:max__json_schema PASS 100% 386 in → 379 out (46 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 386 in → 42 out (23 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 386 in → 24 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 166 in → 187 out (60 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 386 in → 151 out (52 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-opus-5.5:high PASS 100% 168 in → 82 out (70 reasoning)
VERSION=2.0.0
anthropic/claude-opus-5.5:high__json_schema PASS 100% 388 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-5.5:low PASS 100% 168 in → 61 out (49 reasoning)
VERSION=2.0.0
anthropic/claude-opus-5.5:low__json_schema PASS 100% 388 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-5.5:max PASS 100% 168 in → 560 out (544 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-opus-5.5:max__json_schema PASS 100% 388 in → 536 out (511 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-opus-5.5:medium PASS 100% 168 in → 65 out (53 reasoning)
VERSION=2.0.0
anthropic/claude-opus-5.5:medium__json_schema PASS 100% 388 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-5.5:xhigh PASS 100% 168 in → 170 out (158 reasoning)
VERSION=2.0.0
anthropic/claude-opus-5.5:xhigh__json_schema PASS 100% 388 in → 85 out (60 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-5:high PASS 100% 166 in → 86 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-5:high__json_schema PASS 100% 386 in → 66 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-5:low PASS 100% 166 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-5:low__json_schema PASS 100% 386 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-5:max PASS 100% 166 in → 186 out (0 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-opus-5:max__json_schema PASS 100% 386 in → 172 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-opus-5:medium PASS 100% 166 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-5:medium__json_schema PASS 100% 386 in → 43 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-5:xhigh PASS 100% 166 in → 119 out (0 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-opus-5:xhigh__json_schema PASS 100% 386 in → 155 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-sonnet-4.6:high PASS 100% 116 in → 231 out (192 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 284 in → 109 out (57 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 116 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-4.6:max PASS 100% 116 in → 233 out (166 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 284 in → 304 out (194 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 116 in → 86 out (58 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-4.6:none PASS 100% 116 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 116 in → 183 out (143 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 284 in → 121 out (89 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-5:high PASS 100% 166 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:high__json_schema PASS 100% 386 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-sonnet-5:low PASS 100% 166 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:low__json_schema PASS 100% 386 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-5:medium PASS 100% 166 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 386 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-5:none PASS 100% 166 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:none__json_schema PASS 100% 386 in → 22 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-5:xhigh PASS 100% 166 in → 133 out (72 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 386 in → 199 out (60 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 105 in → 1,205 out (1,199 reasoning)
VERSION=2.0.0
arcee-ai/trinity-large-thinking:low PASS 100% 105 in → 1,784 out (1,779 reasoning)
VERSION=2.0.0
arcee-ai/trinity-large-thinking:medium PASS 100% 105 in → 1,350 out (1,344 reasoning)
VERSION=2.0.0
arcee-ai/trinity-large-thinking:xhigh PASS 100% 105 in → 1,112 out (1,105 reasoning)
VERSION=2.0.0
arcee-ai/trinity-mini:high__json_schema PASS 100% 105 in → 1,171 out (1,168 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
arcee-ai/trinity-mini:xhigh PASS 100% 105 in → 770 out (766 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:high PASS 100% 109 in → 497 out (487 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 194 in → 413 out (344 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 109 in → 374 out (365 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 255 in → 1,363 out (1,294 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 188 in → 1,259 out (1,250 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 188 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 115 in → 426 out (410 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 109 in → 833 out (776 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 115 in → 9 out (0 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 109 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 109 in → 588 out (558 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 109 in → 465 out (427 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:high PASS 100% 109 in → 630 out (620 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 111 in → 308 out (288 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:low PASS 100% 109 in → 533 out (524 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 111 in → 287 out (267 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:medium PASS 100% 109 in → 313 out (282 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 109 in → 270 out (249 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 109 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 188 in → 1,196 out (1,186 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 190 in → 1,371 out (1,352 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-pro:high PASS 100% 109 in → 679 out (669 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 111 in → 551 out (530 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
deepseek/deepseek-v4-pro:low PASS 100% 109 in → 593 out (557 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:medium PASS 100% 109 in → 462 out (412 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:none PASS 100% 109 in → 8 out (0 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 111 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 188 in → 1,659 out (1,649 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 379 in → 1,622 out (1,606 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
deepseek/deepseek-v4.1-flash:high PASS 100% 135 in → 531 out (521 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4.1-flash:high__json_schema PASS 100% 135 in → 670 out (650 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4.1-flash:low PASS 100% 135 in → 449 out (439 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4.1-flash:low__json_schema PASS 100% 135 in → 319 out (299 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4.1-flash:max PASS 100% 135 in → 256 out (246 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4.1-flash:max__json_schema PASS 100% 135 in → 681 out (661 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4.1-flash:medium PASS 100% 135 in → 284 out (244 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4.1-flash:medium__json_schema PASS 100% 135 in → 503 out (483 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4.1-flash:none PASS 100% 109 in → 9 out (0 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4.1-flash:none__json_schema PASS 100% 109 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4.1-flash:xhigh PASS 100% 135 in → 225 out (198 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4.1-flash:xhigh__json_schema PASS 100% 135 in → 435 out (414 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3-flash-preview:high PASS 100% 100 in → 848 out (841 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:high__json_schema PASS 100% 101 in → 1,145 out (1,131 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3-flash-preview:low PASS 100% 100 in → 319 out (312 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:low__json_schema PASS 100% 160 in → 14 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3-flash-preview:medium PASS 100% 100 in → 775 out (768 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:medium__json_schema PASS 100% 160 in → 727 out (713 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3-flash-preview:xhigh PASS 100% 100 in → 497 out (490 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 160 in → 566 out (548 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.1-flash-lite-preview:high PASS 100% 100 in → 569 out (562 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 101 in → 527 out (513 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 100 in → 170 out (163 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 160 in → 147 out (128 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 100 in → 386 out (379 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 101 in → 346 out (328 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 101 in → 587 out (580 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 160 in → 870 out (856 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-pro-preview:high PASS 100% 100 in → 620 out (613 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 160 in → 810 out (797 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.1-pro-preview:low PASS 100% 100 in → 413 out (406 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 160 in → 318 out (305 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-pro-preview:medium PASS 100% 100 in → 610 out (603 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 160 in → 352 out (339 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 100 in → 840 out (833 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 160 in → 837 out (818 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:high PASS 100% 101 in → 1,664 out (1,657 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 101 in → 904 out (886 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:low PASS 100% 101 in → 7 out (0 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 101 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:max PASS 100% 100 in → 953 out (946 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 101 in → 1,045 out (1,032 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.5-flash-lite:medium PASS 100% 101 in → 699 out (692 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 160 in → 826 out (807 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 100 in → 1,278 out (1,271 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 160 in → 1,070 out (1,051 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash:high PASS 100% 100 in → 776 out (769 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:high__json_schema PASS 100% 160 in → 926 out (905 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
google/gemini-3.5-flash:low PASS 100% 100 in → 246 out (239 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:low__json_schema PASS 100% 160 in → 252 out (233 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash:medium PASS 100% 100 in → 725 out (718 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:medium__json_schema PASS 100% 160 in → 540 out (521 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash:xhigh PASS 100% 100 in → 1,125 out (1,118 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 160 in → 1,348 out (1,327 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
google/gemini-3.6-flash:high PASS 100% 101 in → 494 out (487 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:high__json_schema PASS 100% 101 in → 1,159 out (1,145 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.6-flash:low PASS 100% 101 in → 253 out (246 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:low__json_schema PASS 100% 160 in → 147 out (126 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
google/gemini-3.6-flash:max PASS 100% 100 in → 494 out (487 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:max__json_schema PASS 100% 160 in → 1,023 out (1,007 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
google/gemini-3.6-flash:medium PASS 100% 101 in → 611 out (604 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:medium__json_schema PASS 100% 160 in → 532 out (513 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.6-flash:xhigh PASS 100% 100 in → 530 out (523 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 101 in → 1,167 out (1,153 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.8-flash:high PASS 100% 100 in → 565 out (558 reasoning)
VERSION=2.0.0
google/gemini-3.8-flash:high__json_schema PASS 100% 101 in → 2,432 out (2,419 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.8-flash:low PASS 100% 100 in → 134 out (127 reasoning)
VERSION=2.0.0
google/gemini-3.8-flash:low__json_schema PASS 100% 160 in → 153 out (140 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.8-flash:max PASS 100% 100 in → 588 out (581 reasoning)
VERSION=2.0.0
google/gemini-3.8-flash:max__json_schema PASS 100% 160 in → 1,512 out (1,499 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.8-flash:medium PASS 100% 100 in → 284 out (277 reasoning)
VERSION=2.0.0
google/gemini-3.8-flash:medium__json_schema PASS 100% 101 in → 366 out (351 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
google/gemini-3.8-flash:xhigh PASS 100% 100 in → 1,580 out (1,573 reasoning)
VERSION=2.0.0
google/gemini-3.8-flash:xhigh__json_schema PASS 100% 101 in → 1,574 out (1,561 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemma-4-26b-a4b-it:high PASS 100% 116 in → 444 out (436 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 116 in → 781 out (668 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-26b-a4b-it:low PASS 100% 115 in → 811 out (690 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:medium PASS 100% 116 in → 749 out (705 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 116 in → 691 out (602 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-26b-a4b-it:none PASS 100% 113 in → 8 out (0 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 113 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 116 in → 774 out (704 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 116 in → 1,074 out (905 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-31b-it:high PASS 100% 116 in → 326 out (1 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:high__json_schema PASS 100% 115 in → 264 out (227 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemma-4-31b-it:low PASS 100% 115 in → 282 out (243 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:low__json_schema PASS 100% 115 in → 219 out (170 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-31b-it:medium PASS 100% 116 in → 230 out (194 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:medium__json_schema PASS 100% 119 in → 260 out (245 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemma-4-31b-it:none PASS 100% 113 in → 8 out (0 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:none__json_schema PASS 100% 113 in → 15 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemma-4-31b-it:xhigh PASS 100% 116 in → 248 out (1 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 119 in → 261 out (246 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
ibm-granite/granite-4.1-8b__json_schema PASS 100% 108 in → 18 out
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
inception/mercury-2.5:high PASS 100% 90 in → 1,172 out (1,160 reasoning)
VERSION=2.0.0
inception/mercury-2.5:high__json_schema PASS 100% 293 in → 2,292 out (2,238 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
inception/mercury-2.5:max__json_schema PASS 100% 292 in → 1,540 out (1,453 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
inception/mercury-2.5:medium PASS 100% 92 in → 1,475 out (1,465 reasoning)
VERSION=2.0.0
inception/mercury-2.5:medium__json_schema PASS 100% 287 in → 1,387 out (1,327 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
inception/mercury-2.5:none PASS 100% 95 in → 11 out (0 reasoning)
VERSION=2.0.0
inception/mercury-2.5:xhigh PASS 100% 90 in → 1,709 out (1,699 reasoning)
VERSION=2.0.0
inception/mercury-2.5:xhigh__json_schema PASS 100% 292 in → 1,520 out (1,465 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
inclusionai/ling-2.6-flash PASS 100% 120 in → 8 out
VERSION=2.0.0
inclusionai/ling-2.6-flash__json_schema PASS 100% 120 in → 19 out
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
meta/muse-spark-1.1:high PASS 100% 103 in → 961 out (944 reasoning)
VERSION=2.0.0
meta/muse-spark-1.1:high__json_schema PASS 100% 103 in → 1,474 out (1,448 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
meta/muse-spark-1.1:low PASS 100% 103 in → 634 out (617 reasoning)
VERSION=2.0.0
meta/muse-spark-1.1:low__json_schema PASS 100% 103 in → 776 out (750 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
meta/muse-spark-1.1:max PASS 100% 103 in → 2,356 out (2,339 reasoning)
VERSION=2.0.0
meta/muse-spark-1.1:max__json_schema PASS 100% 103 in → 1,890 out (1,865 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
meta/muse-spark-1.1:medium PASS 100% 103 in → 908 out (891 reasoning)
VERSION=2.0.0
meta/muse-spark-1.1:medium__json_schema PASS 100% 103 in → 1,190 out (1,165 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
meta/muse-spark-1.1:xhigh PASS 100% 103 in → 2,634 out (2,617 reasoning)
VERSION=2.0.0
meta/muse-spark-1.1:xhigh__json_schema PASS 100% 103 in → 2,734 out (2,708 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
meta/muse-spark-1.3-contributor:high PASS 100% 103 in → 1,036 out (981 reasoning)
VERSION=2.0.0
meta/muse-spark-1.3-contributor:high__json_schema PASS 100% 103 in → 2,021 out (1,937 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
meta/muse-spark-1.3-contributor:low PASS 100% 103 in → 495 out (459 reasoning)
VERSION=2.0.0
meta/muse-spark-1.3-contributor:low__json_schema PASS 100% 103 in → 678 out (635 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
meta/muse-spark-1.3-contributor:medium PASS 100% 103 in → 1,482 out (1,428 reasoning)
VERSION=2.0.0
meta/muse-spark-1.3-contributor:medium__json_schema PASS 100% 103 in → 1,451 out (1,394 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
meta/muse-spark-1.3-contributor:xhigh PASS 100% 103 in → 1,147 out (1,109 reasoning)
VERSION=2.0.0
meta/muse-spark-1.3-contributor:xhigh__json_schema PASS 100% 103 in → 1,828 out (1,751 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
minimax/minimax-m2.5:high PASS 100% 142 in → 382 out (373 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:high__json_schema PASS 100% 135 in → 396 out (359 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.5:low PASS 100% 135 in → 508 out (498 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:low__json_schema PASS 100% 135 in → 373 out (357 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.5:medium PASS 100% 135 in → 578 out (548 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:medium__json_schema PASS 100% 135 in → 366 out (388 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.5:xhigh PASS 100% 135 in → 402 out (393 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 142 in → 248 out (232 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.7:high PASS 100% 135 in → 791 out (790 reasoning)
VERSION=2.0.0
minimax/minimax-m2.7:low PASS 100% 135 in → 700 out (707 reasoning)
VERSION=2.0.0
minimax/minimax-m2.7:low__json_schema PASS 100% 268 in → 643 out (629 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
minimax/minimax-m2.7:medium PASS 100% 135 in → 829 out (831 reasoning)
VERSION=2.0.0
minimax/minimax-m2.7:medium__json_schema PASS 100% 268 in → 516 out (502 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
minimax/minimax-m2.7:xhigh PASS 100% 135 in → 392 out (403 reasoning)
VERSION=2.0.0
minimax/minimax-m3:high PASS 100% 273 in → 388 out (417 reasoning)
``` VERSION=2.0.0 ```
minimax/minimax-m3:high__json_schema PASS 100% 260 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m3:low PASS 100% 273 in → 343 out (371 reasoning)
VERSION=2.0.0
minimax/minimax-m3:low__json_schema PASS 100% 260 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m3:medium PASS 100% 273 in → 358 out (370 reasoning)
``` VERSION=2.0.0 ```
minimax/minimax-m3:medium__json_schema PASS 100% 260 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m3:xhigh PASS 100% 273 in → 256 out (276 reasoning)
VERSION=2.0.0
minimax/minimax-m3:xhigh__json_schema PASS 100% 260 in → 14 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/devstral-2512 PASS 100% 109 in → 9 out
VERSION=2.0.0
mistralai/devstral-2512__json_schema PASS 100% 109 in → 16 out
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:high PASS 100% 121 in → 1,323 out (1,252 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 109 in → 696 out (650 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:low PASS 100% 121 in → 573 out (532 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 109 in → 701 out (635 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:medium PASS 100% 121 in → 716 out (650 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 109 in → 1,405 out (1,375 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:none PASS 100% 121 in → 9 out (0 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 109 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 121 in → 620 out (573 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 109 in → 328 out (302 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
moonshotai/kimi-k2.6:high__json_schema PASS 100% 103 in → 1,390 out (1,397 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.6:low__json_schema PASS 100% 104 in → 983 out (961 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 287 in → 1,261 out (1,246 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
moonshotai/kimi-k2.7-code:high PASS 100% 103 in → 326 out (317 reasoning)
VERSION=2.0.0
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 104 in → 346 out (321 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
moonshotai/kimi-k2.7-code:low PASS 100% 104 in → 262 out (253 reasoning)
VERSION=2.0.0
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 104 in → 474 out (446 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.7-code:medium PASS 100% 103 in → 280 out (271 reasoning)
VERSION=2.0.0
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 188 in → 785 out (769 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 104 in → 186 out (168 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k3:high PASS 100% 181 in → 215 out (193 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:high__json_schema PASS 100% 181 in → 217 out (195 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:low PASS 100% 181 in → 213 out (191 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:low__json_schema PASS 100% 181 in → 263 out (241 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:max PASS 100% 181 in → 341 out (319 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:max__json_schema PASS 100% 181 in → 172 out (150 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:medium PASS 100% 181 in → 255 out (233 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:medium__json_schema PASS 100% 181 in → 150 out (128 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:xhigh PASS 100% 181 in → 256 out (234 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 278 in → 138 out (111 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 122 in → 352 out (323 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 122 in → 1,483 out (1,376 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 122 in → 378 out (394 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 122 in → 6,268 out (6,254 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 122 in → 978 out (900 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 122 in → 8,755 out (9,129 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 122 in → 9 out (0 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 122 in → 385 out (370 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 122 in → 442 out (422 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 122 in → 287 out (266 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 122 in → 171 out (146 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 122 in → 295 out (276 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 122 in → 197 out (175 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 122 in → 9 out (0 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 122 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 122 in → 204 out (180 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 122 in → 466 out (438 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-5.4-mini:high PASS 100% 100 in → 891 out (878 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:high__json_schema PASS 100% 136 in → 902 out (880 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:low PASS 100% 100 in → 114 out (101 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:low__json_schema PASS 100% 136 in → 114 out (92 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:medium PASS 100% 100 in → 375 out (362 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:medium__json_schema PASS 100% 136 in → 528 out (506 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:xhigh PASS 100% 100 in → 10,371 out (10,358 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 136 in → 9,517 out (9,495 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-nano:high PASS 100% 100 in → 65 out (52 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:high__json_schema PASS 100% 136 in → 126 out (103 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:low PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:low__json_schema PASS 100% 136 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:medium PASS 100% 100 in → 128 out (115 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:medium__json_schema PASS 100% 136 in → 71 out (48 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:none__json_schema PASS 100% 136 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 100 in → 163 out (150 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 136 in → 298 out (275 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4:high PASS 100% 100 in → 128 out (115 reasoning)
VERSION=2.0.0
openai/gpt-5.4:high__json_schema PASS 100% 136 in → 112 out (90 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:low PASS 100% 100 in → 74 out (61 reasoning)
VERSION=2.0.0
openai/gpt-5.4:low__json_schema PASS 100% 136 in → 31 out (9 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:medium PASS 100% 100 in → 121 out (108 reasoning)
VERSION=2.0.0
openai/gpt-5.4:medium__json_schema PASS 100% 136 in → 128 out (106 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:xhigh PASS 100% 100 in → 365 out (352 reasoning)
VERSION=2.0.0
openai/gpt-5.4:xhigh__json_schema PASS 100% 136 in → 538 out (516 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:high PASS 100% 100 in → 218 out (205 reasoning)
VERSION=2.0.0
openai/gpt-5.5:high__json_schema PASS 100% 136 in → 461 out (439 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:low PASS 100% 100 in → 105 out (92 reasoning)
VERSION=2.0.0
openai/gpt-5.5:low__json_schema PASS 100% 136 in → 211 out (189 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:medium PASS 100% 100 in → 302 out (289 reasoning)
VERSION=2.0.0
openai/gpt-5.5:medium__json_schema PASS 100% 136 in → 230 out (208 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.5:none__json_schema PASS 100% 136 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.5:xhigh PASS 100% 100 in → 267 out (254 reasoning)
VERSION=2.0.0
openai/gpt-5.5:xhigh__json_schema PASS 100% 136 in → 538 out (516 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:high PASS 100% 100 in → 211 out (198 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:high__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:low PASS 100% 100 in → 98 out (85 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:low__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:max PASS 100% 100 in → 323 out (310 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:max__json_schema PASS 100% 136 in → 266 out (244 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:medium PASS 100% 100 in → 120 out (107 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:xhigh PASS 100% 100 in → 232 out (219 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 136 in → 146 out (124 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:high PASS 100% 100 in → 101 out (88 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:high__json_schema PASS 100% 136 in → 90 out (68 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:low PASS 100% 100 in → 44 out (31 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:low__json_schema PASS 100% 136 in → 60 out (38 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:max PASS 100% 100 in → 103 out (90 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:max__json_schema PASS 100% 136 in → 168 out (146 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:medium PASS 100% 100 in → 70 out (57 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:medium__json_schema PASS 100% 136 in → 66 out (44 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:xhigh PASS 100% 100 in → 76 out (63 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 136 in → 112 out (90 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:high PASS 100% 100 in → 45 out (32 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:high__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:low PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:low__json_schema PASS 100% 136 in → 100 out (78 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:max PASS 100% 100 in → 141 out (128 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:max__json_schema PASS 100% 136 in → 428 out (406 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:medium PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 136 in → 95 out (73 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:xhigh PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-astra:high PASS 100% 100 in → 52 out (39 reasoning)
VERSION=2.0.0
openai/gpt-6-astra:high__json_schema PASS 100% 136 in → 78 out (56 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-astra:low PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-6-astra:low__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-astra:max PASS 100% 100 in → 139 out (126 reasoning)
VERSION=2.0.0
openai/gpt-6-astra:max__json_schema PASS 100% 136 in → 341 out (318 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-6-astra:medium PASS 100% 100 in → 28 out (15 reasoning)
VERSION=2.0.0
openai/gpt-6-astra:medium__json_schema PASS 100% 136 in → 65 out (43 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-astra:xhigh PASS 100% 100 in → 106 out (93 reasoning)
VERSION=2.0.0
openai/gpt-6-astra:xhigh__json_schema PASS 100% 136 in → 94 out (72 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-luna:high PASS 100% 100 in → 187 out (174 reasoning)
VERSION=2.0.0
openai/gpt-6-luna:high__json_schema PASS 100% 136 in → 68 out (46 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-luna:low PASS 100% 100 in → 82 out (69 reasoning)
VERSION=2.0.0
openai/gpt-6-luna:low__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-luna:max PASS 100% 100 in → 162 out (149 reasoning)
VERSION=2.0.0
openai/gpt-6-luna:max__json_schema PASS 100% 136 in → 188 out (166 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-luna:medium PASS 100% 100 in → 98 out (85 reasoning)
VERSION=2.0.0
openai/gpt-6-luna:medium__json_schema PASS 100% 136 in → 165 out (143 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-luna:xhigh PASS 100% 100 in → 164 out (151 reasoning)
VERSION=2.0.0
openai/gpt-6-luna:xhigh__json_schema PASS 100% 136 in → 124 out (102 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-sol:high PASS 100% 100 in → 71 out (58 reasoning)
VERSION=2.0.0
openai/gpt-6-sol:high__json_schema PASS 100% 136 in → 110 out (87 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-6-sol:low PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-6-sol:low__json_schema PASS 100% 136 in → 50 out (28 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-sol:max PASS 100% 100 in → 139 out (126 reasoning)
VERSION=2.0.0
openai/gpt-6-sol:max__json_schema PASS 100% 136 in → 539 out (516 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-6-sol:medium PASS 100% 100 in → 65 out (52 reasoning)
VERSION=2.0.0
openai/gpt-6-sol:medium__json_schema PASS 100% 136 in → 62 out (40 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-6-sol:xhigh PASS 100% 100 in → 73 out (60 reasoning)
VERSION=2.0.0
openai/gpt-6-sol:xhigh__json_schema PASS 100% 136 in → 155 out (132 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-oss-120b:high PASS 100% 161 in → 155 out (148 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:high__json_schema PASS 100% 161 in → 559 out (536 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-oss-120b:low PASS 100% 159 in → 82 out (62 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:low__json_schema PASS 100% 163 in → 164 out (133 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
openai/gpt-oss-120b:medium PASS 100% 160 in → 144 out (112 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:medium__json_schema PASS 100% 163 in → 154 out (115 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-oss-120b:xhigh PASS 100% 159 in → 1,130 out (1,103 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 159 in → 680 out (663 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-oss-20b:high PASS 100% 161 in → 300 out (285 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:high__json_schema PASS 100% 159 in → 848 out (860 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-oss-20b:low PASS 100% 165 in → 72 out (56 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:low__json_schema PASS 100% 159 in → 44 out (24 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-oss-20b:medium PASS 100% 159 in → 442 out (369 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:medium__json_schema PASS 100% 161 in → 321 out (297 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-oss-20b:xhigh PASS 100% 159 in → 660 out (635 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 163 in → 1,428 out (1,351 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
poolside/laguna-m.1:high PASS 100% 111 in → 868 out (858 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:high__json_schema PASS 100% 111 in → 756 out (735 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
poolside/laguna-m.1:low PASS 100% 111 in → 449 out (439 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:low__json_schema PASS 100% 111 in → 1,051 out (1,032 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-m.1:medium PASS 100% 111 in → 759 out (749 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:medium__json_schema PASS 100% 111 in → 544 out (529 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-m.1:none PASS 100% 111 in → 9 out (0 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:none__json_schema PASS 100% 111 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-m.1:xhigh PASS 100% 111 in → 818 out (808 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:xhigh__json_schema PASS 100% 111 in → 1,078 out (1,063 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-xs-2.1:high PASS 100% 111 in → 665 out (657 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:high__json_schema PASS 100% 111 in → 474 out (455 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:low PASS 100% 111 in → 487 out (479 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:low__json_schema PASS 100% 111 in → 495 out (476 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:medium PASS 100% 111 in → 521 out (513 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 111 in → 466 out (446 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:none PASS 100% 111 in → 7 out (0 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:none__json_schema PASS 100% 111 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:xhigh PASS 100% 111 in → 1,079 out (1,072 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 111 in → 530 out (511 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs.2:high PASS 100% 148 in → 410 out (400 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:high__json_schema PASS 100% 148 in → 461 out (446 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-xs.2:low PASS 100% 148 in → 519 out (510 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:low__json_schema PASS 100% 148 in → 494 out (474 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs.2:medium PASS 100% 148 in → 660 out (650 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:medium__json_schema PASS 100% 148 in → 300 out (279 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
poolside/laguna-xs.2:xhigh PASS 100% 148 in → 501 out (491 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 148 in → 553 out (533 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
prism-ml/ternary-bonsai-2-27b:high PASS 100% 106 in → 521 out (511 reasoning)
VERSION=2.0.0
prism-ml/ternary-bonsai-2-27b:high__json_schema PASS 100% 202 in → 320 out (305 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
prism-ml/ternary-bonsai-2-27b:low PASS 100% 106 in → 325 out (315 reasoning)
VERSION=2.0.0
prism-ml/ternary-bonsai-2-27b:low__json_schema PASS 100% 202 in → 1,434 out (1,419 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
prism-ml/ternary-bonsai-2-27b:max PASS 100% 148 in → 234 out (224 reasoning)
VERSION=2.0.0
prism-ml/ternary-bonsai-2-27b:max__json_schema PASS 100% 240 in → 405 out (390 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
prism-ml/ternary-bonsai-2-27b:medium PASS 100% 106 in → 347 out (337 reasoning)
VERSION=2.0.0
prism-ml/ternary-bonsai-2-27b:medium__json_schema PASS 100% 202 in → 391 out (376 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
prism-ml/ternary-bonsai-2-27b:none PASS 100% 108 in → 8 out (0 reasoning)
VERSION=2.0.0
prism-ml/ternary-bonsai-2-27b:none__json_schema PASS 100% 204 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
prism-ml/ternary-bonsai-2-27b:xhigh PASS 100% 148 in → 224 out (214 reasoning)
VERSION=2.0.0
prism-ml/ternary-bonsai-2-27b:xhigh__json_schema PASS 100% 240 in → 1,500 out (1,479 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-27b:high PASS 100% 106 in → 998 out (911 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:high__json_schema PASS 100% 106 in → 1,233 out (1,142 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-27b:low PASS 100% 106 in → 814 out (773 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:low__json_schema PASS 100% 106 in → 1,167 out (1,038 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-27b:medium PASS 100% 106 in → 618 out (606 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:medium__json_schema PASS 100% 106 in → 1,301 out (1,198 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-27b:none PASS 100% 108 in → 8 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:xhigh PASS 100% 106 in → 894 out (844 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 106 in → 1,206 out (1,114 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:high PASS 100% 106 in → 861 out (762 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 106 in → 1,550 out (1,450 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:low PASS 100% 106 in → 1,722 out (1,710 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 106 in → 1,951 out (1,755 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 106 in → 1,504 out (1,384 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 106 in → 1,185 out (1,075 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:none PASS 100% 108 in → 8 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 108 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 106 in → 1,205 out (1,168 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 106 in → 1,609 out (1,518 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-flash:high PASS 100% 106 in → 1,216 out (1,204 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:low PASS 100% 106 in → 1,155 out (1,143 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:medium PASS 100% 106 in → 2,049 out (2,037 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:none PASS 100% 108 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:xhigh PASS 100% 106 in → 1,823 out (1,811 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:high PASS 100% 106 in → 1,219 out (1,207 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:low PASS 100% 106 in → 2,050 out (2,038 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:max PASS 100% 106 in → 1,019 out (1,007 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:medium PASS 100% 106 in → 1,593 out (1,581 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:none PASS 100% 108 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:xhigh PASS 100% 106 in → 1,407 out (1,395 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:high PASS 100% 106 in → 429 out (417 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:low PASS 100% 106 in → 350 out (338 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:medium PASS 100% 106 in → 366 out (354 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:none PASS 100% 108 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:xhigh PASS 100% 106 in → 373 out (361 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:high PASS 100% 106 in → 591 out (579 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:low PASS 100% 106 in → 360 out (348 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:medium PASS 100% 106 in → 380 out (368 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:none PASS 100% 108 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:xhigh PASS 100% 106 in → 457 out (445 reasoning)
VERSION=2.0.0
qwen/qwen3.8-flash:high PASS 100% 157 in → 907 out (897 reasoning)
VERSION=2.0.0
qwen/qwen3.8-flash:high__json_schema PASS 100% 159 in → 4,418 out (4,402 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
qwen/qwen3.8-flash:low PASS 100% 157 in → 1,251 out (1,241 reasoning)
VERSION=2.0.0
qwen/qwen3.8-flash:low__json_schema PASS 100% 159 in → 3,175 out (3,158 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
qwen/qwen3.8-flash:max PASS 100% 157 in → 199 out (189 reasoning)
VERSION=2.0.0
qwen/qwen3.8-flash:max__json_schema PASS 100% 159 in → 466 out (450 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
qwen/qwen3.8-flash:medium PASS 100% 157 in → 566 out (556 reasoning)
VERSION=2.0.0
qwen/qwen3.8-flash:medium__json_schema PASS 100% 159 in → 978 out (962 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
qwen/qwen3.8-flash:xhigh PASS 100% 157 in → 307 out (297 reasoning)
VERSION=2.0.0
qwen/qwen3.8-flash:xhigh__json_schema PASS 100% 159 in → 1,641 out (1,618 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.8-max:high PASS 100% 144 in → 232 out (220 reasoning)
VERSION=2.0.0
qwen/qwen3.8-max:low PASS 100% 132 in → 194 out (178 reasoning)
``` VERSION=2.0.0 ```
qwen/qwen3.8-max:low__json_schema PASS 100% 134 in → 203 out (181 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.8-max:max PASS 100% 144 in → 321 out (309 reasoning)
VERSION=2.0.0
qwen/qwen3.8-max:max__json_schema PASS 100% 146 in → 312 out (290 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.8-max:medium PASS 100% 106 in → 169 out (157 reasoning)
VERSION=2.0.0
qwen/qwen3.8-max:medium__json_schema PASS 100% 108 in → 190 out (168 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.8-max:xhigh PASS 100% 144 in → 411 out (399 reasoning)
VERSION=2.0.0
qwen/qwen3.8-max:xhigh__json_schema PASS 100% 146 in → 202 out (180 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
sakana/fugu-max:high PASS 100% 210 in → 605 out (595 reasoning)
VERSION=2.0.0
sakana/fugu-max:high__json_schema PASS 100% 277 in → 416 out (400 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
sakana/fugu-max:low PASS 100% 210 in → 622 out (612 reasoning)
VERSION=2.0.0
sakana/fugu-max:low__json_schema PASS 100% 277 in → 132 out (116 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
sakana/fugu-max:max PASS 100% 210 in → 350 out (340 reasoning)
VERSION=2.0.0
sakana/fugu-max:max__json_schema PASS 100% 277 in → 177 out (160 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
sakana/fugu-max:medium PASS 100% 210 in → 325 out (315 reasoning)
VERSION=2.0.0
sakana/fugu-max:medium__json_schema PASS 100% 277 in → 574 out (557 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
sakana/fugu-max:xhigh PASS 100% 210 in → 627 out (617 reasoning)
VERSION=2.0.0
sakana/fugu-max:xhigh__json_schema PASS 100% 277 in → 403 out (386 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
stealth/ox-alpha:high PASS 100% 181 in → 44 out (0 reasoning)
VERSION=2.0.0
stealth/ox-alpha:low PASS 100% 181 in → 31 out (0 reasoning)
VERSION=2.0.0
stealth/ox-alpha:max PASS 100% 181 in → 331 out (0 reasoning)
VERSION=2.0.0
tencent/hy3:high PASS 100% 110 in → 1,491 out (1,374 reasoning)
VERSION=2.0.0
tencent/hy3:high__json_schema PASS 100% 110 in → 3,303 out (3,198 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n"}
tencent/hy3:low PASS 100% 110 in → 1,718 out (1,489 reasoning)
VERSION=2.0.0
tencent/hy3:low__json_schema PASS 100% 110 in → 1,095 out (1,086 reasoning)
VERSION=2.0.0
tencent/hy3:medium PASS 100% 110 in → 791 out (726 reasoning)
VERSION=2.0.0
tencent/hy3:medium__json_schema PASS 100% 110 in → 1,218 out (1,088 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
tencent/hy3:none PASS 100% 113 in → 8 out (0 reasoning)
VERSION=2.0.0
tencent/hy3:none__json_schema PASS 100% 113 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
tencent/hy3:xhigh PASS 100% 110 in → 3,099 out (3,090 reasoning)
VERSION=2.0.0
tencent/hy3:xhigh__json_schema PASS 100% 110 in → 1,394 out (1,300 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
thinkingmachines/inkling-small:high PASS 100% 109 in → 642 out (616 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:high__json_schema PASS 100% 109 in → 505 out (492 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:low PASS 100% 109 in → 68 out (55 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:low__json_schema PASS 100% 109 in → 82 out (60 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
thinkingmachines/inkling-small:max PASS 100% 109 in → 685 out (672 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:max__json_schema PASS 100% 109 in → 552 out (530 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
thinkingmachines/inkling-small:medium PASS 100% 109 in → 392 out (379 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:medium__json_schema PASS 100% 109 in → 217 out (195 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
thinkingmachines/inkling-small:xhigh PASS 100% 109 in → 394 out (380 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 109 in → 840 out (820 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling:high PASS 100% 109 in → 1,023 out (1,011 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:high__json_schema PASS 100% 109 in → 1,213 out (1,192 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling:low PASS 100% 109 in → 54 out (42 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:max PASS 100% 109 in → 566 out (553 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:max__json_schema PASS 100% 109 in → 3,040 out (3,019 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling:medium PASS 100% 109 in → 615 out (601 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:medium__json_schema PASS 100% 109 in → 620 out (599 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling:xhigh PASS 100% 109 in → 2,092 out (2,078 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:high PASS 100% 279 in → 1,033 out (1,023 reasoning)
``` VERSION=2.0.0 ```
x-ai/grok-4.3:high__json_schema PASS 100% 336 in → 1,475 out (1,459 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:low PASS 100% 285 in → 763 out (756 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:low__json_schema PASS 100% 342 in → 605 out (589 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:max PASS 100% 279 in → 2,329 out (2,322 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:max__json_schema PASS 100% 336 in → 1,165 out (1,149 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:medium PASS 100% 285 in → 1,047 out (1,040 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:medium__json_schema PASS 100% 342 in → 1,269 out (1,253 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:none PASS 100% 277 in → 7 out (0 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:none__json_schema PASS 100% 334 in → 12 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
x-ai/grok-4.3:xhigh PASS 100% 279 in → 2,302 out (2,295 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:xhigh__json_schema PASS 100% 336 in → 1,904 out (1,888 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.5:high PASS 100% 301 in → 1,522 out (1,515 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:high__json_schema PASS 100% 370 in → 610 out (593 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.5:low PASS 100% 301 in → 226 out (219 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:low__json_schema PASS 100% 370 in → 549 out (532 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.5:max PASS 100% 301 in → 624 out (617 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:max__json_schema PASS 100% 370 in → 1,013 out (999 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
x-ai/grok-4.5:medium PASS 100% 301 in → 877 out (870 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:medium__json_schema PASS 100% 370 in → 916 out (902 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
x-ai/grok-4.5:xhigh PASS 100% 301 in → 1,239 out (1,232 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:xhigh__json_schema PASS 100% 370 in → 630 out (613 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.7:high PASS 100% 1,337 in → 587 out (580 reasoning)
VERSION=2.0.0
x-ai/grok-4.7:high__json_schema PASS 100% 1,406 in → 1,242 out (1,226 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
x-ai/grok-4.7:low PASS 100% 1,337 in → 567 out (560 reasoning)
VERSION=2.0.0
x-ai/grok-4.7:low__json_schema PASS 100% 1,406 in → 309 out (293 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
x-ai/grok-4.7:max PASS 100% 1,338 in → 579 out (572 reasoning)
VERSION=2.0.0
x-ai/grok-4.7:max__json_schema PASS 100% 1,407 in → 1,422 out (1,406 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
x-ai/grok-4.7:medium PASS 100% 1,337 in → 868 out (861 reasoning)
VERSION=2.0.0
x-ai/grok-4.7:medium__json_schema PASS 100% 1,406 in → 914 out (898 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
x-ai/grok-4.7:xhigh PASS 100% 1,338 in → 445 out (438 reasoning)
VERSION=2.0.0
x-ai/grok-4.7:xhigh__json_schema PASS 100% 1,407 in → 950 out (934 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
xiaomi/mimo-v2.6-flash:high PASS 100% 101 in → 233 out (224 reasoning)
VERSION=2.0.0
xiaomi/mimo-v2.6-flash:high__json_schema PASS 100% 101 in → 102 out (86 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
xiaomi/mimo-v2.6-flash:low PASS 100% 101 in → 67 out (57 reasoning)
VERSION=2.0.0
xiaomi/mimo-v2.6-flash:low__json_schema PASS 100% 101 in → 76 out (57 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
xiaomi/mimo-v2.6-flash:max PASS 100% 101 in → 133 out (123 reasoning)
VERSION=2.0.0
xiaomi/mimo-v2.6-flash:max__json_schema PASS 100% 101 in → 109 out (94 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
xiaomi/mimo-v2.6-flash:medium PASS 100% 101 in → 104 out (94 reasoning)
VERSION=2.0.0
xiaomi/mimo-v2.6-flash:medium__json_schema PASS 100% 101 in → 101 out (86 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
xiaomi/mimo-v2.6-flash:none PASS 100% 103 in → 8 out (0 reasoning)
VERSION=2.0.0
xiaomi/mimo-v2.6-flash:none__json_schema PASS 100% 103 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
xiaomi/mimo-v2.6-flash:xhigh PASS 100% 101 in → 64 out (55 reasoning)
VERSION=2.0.0
xiaomi/mimo-v2.6-flash:xhigh__json_schema PASS 100% 101 in → 69 out (53 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
xiaomi/mimo-v2.6-pro:high PASS 100% 101 in → 330 out (317 reasoning)
``` VERSION=2.0.0 ```
xiaomi/mimo-v2.6-pro:high__json_schema PASS 100% 101 in → 109 out (90 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
xiaomi/mimo-v2.6-pro:low PASS 100% 101 in → 250 out (241 reasoning)
VERSION=2.0.0
xiaomi/mimo-v2.6-pro:low__json_schema PASS 100% 101 in → 110 out (94 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
xiaomi/mimo-v2.6-pro:max PASS 100% 101 in → 86 out (73 reasoning)
``` VERSION=2.0.0 ```
xiaomi/mimo-v2.6-pro:max__json_schema PASS 100% 101 in → 143 out (124 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
xiaomi/mimo-v2.6-pro:medium PASS 100% 101 in → 95 out (82 reasoning)
``` VERSION=2.0.0 ```
xiaomi/mimo-v2.6-pro:medium__json_schema PASS 100% 101 in → 139 out (122 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
xiaomi/mimo-v2.6-pro:xhigh__json_schema PASS 100% 101 in → 196 out (176 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
z-ai/glm-4.7-flash:high PASS 100% 99 in → 1,773 out (1,682 reasoning)
VERSION=2.0.0
z-ai/glm-4.7-flash:high__json_schema PASS 100% 99 in → 1,749 out (1,570 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-4.7-flash:low PASS 100% 99 in → 2,807 out (2,730 reasoning)
VERSION=2.0.0
z-ai/glm-4.7-flash:medium PASS 100% 99 in → 374 out (361 reasoning)
VERSION=2.0.0
z-ai/glm-4.7-flash:none__json_schema PASS 100% 99 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-4.7-flash:xhigh PASS 100% 99 in → 1,323 out (1,314 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:high PASS 100% 106 in → 449 out (370 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:high__json_schema PASS 100% 106 in → 585 out (568 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.2:low PASS 100% 106 in → 853 out (844 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:low__json_schema PASS 100% 106 in → 215 out (183 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.2:medium PASS 100% 106 in → 583 out (574 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:none PASS 100% 100 in → 8 out (0 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:none__json_schema PASS 100% 100 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.2:xhigh PASS 100% 106 in → 1,034 out (859 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:xhigh__json_schema PASS 100% 106 in → 827 out (809 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.3-flash:high PASS 100% 106 in → 56 out (47 reasoning)
VERSION=2.0.0
z-ai/glm-5.3-flash:high__json_schema PASS 100% 106 in → 142 out (125 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.3-flash:low__json_schema PASS 100% 106 in → 77 out (67 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.3-flash:max PASS 100% 106 in → 479 out (466 reasoning)
``` VERSION=2.0.0 ```
z-ai/glm-5.3-flash:max__json_schema PASS 100% 106 in → 627 out (609 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.3-flash:medium PASS 100% 106 in → 9 out (0 reasoning)
VERSION=2.0.0
z-ai/glm-5.3-flash:medium__json_schema PASS 100% 106 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.3-flash:xhigh PASS 100% 106 in → 108 out (86 reasoning)
VERSION=2.0.0
z-ai/glm-5.3-flash:xhigh__json_schema PASS 100% 106 in → 225 out (205 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
anthropic/claude-opus-4.6:max FAIL 0% 116 in → 244 out (150 reasoning)
``` VERSION=2.0.0 ``` **Reasoning:** The `release` branch has the higher version (`2.0.0`), which represents a deliberate major release that supersedes the minor bump (`1.1.0`) on `main`. Taking the higher, more intentional release version is the correct resolution.
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.6:none FAIL 0% 116 in → 122 out (0 reasoning)
**Resolved file content:** ``` VERSION=2.0.0 ``` **Reasoning:** The base version was `1.0.0`. Branch A (release) bumped it to `2.0.0` (a major version increase), while Branch B (main) bumped it to `1.1.0` (a minor version increase). Since `2.0.0` is the higher version and a major release supersedes a minor release, the correct resolution is to accept the release branch's version `2.0.0`.
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.8:medium FAIL 0% 166 in → 220 out (60 reasoning)
``` VERSION=2.0.0 ``` The resolution takes `VERSION=2.0.0` because, under semantic versioning, the release branch's major version bump (1.0.0 → 2.0.0) supersedes main's minor bump (1.0.0 → 1.1.0). The higher version number always wins to avoid version regression.
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.8:none FAIL 0% 166 in → 221 out (0 reasoning)
VERSION=2.0.0 **Reasoning:** Using semantic versioning, `2.0.0` is the higher version and supersedes `1.1.0`. The `2.0.0` release (major version bump) indicates a more significant/newer set of changes than the `1.1.0` minor update, so the resolution keeps the release branch's version. If your workflow instead requires the main branch's version to win (e.g., release was a hotfix branch and 1.1.0 is your latest development target), the resolved content would be: ``` VERSION=1.1.0 ``` But based on standard version-precedence, the answer above (`VERSION=2.0.0`) is the recommended resolution.
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 105 in → 1,286 out (1,357 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:low FAIL 0% 105 in → 1,207 out (1,237 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 105 in → 1,083 out (1,160 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:medium FAIL 0% 105 in → 789 out (792 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 105 in → 912 out (892 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content":"VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 105 in → 735 out (744 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
deepseek/deepseek-v4-flash:none FAIL 0% 109 in → 8 out (0 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
deepseek/deepseek-v4-pro:low__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 111 in → 626 out (606 reasoning)
VERSION=2.0.1
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.1" }
Failure: Content mismatch: ['version.txt']
google/gemma-4-26b-a4b-it:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
ibm-granite/granite-4.1-8b FAIL 0% 108 in → 8 out
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:high FAIL 0% 115 in → 1,063 out (961 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:high__json_schema FAIL 0% 115 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:low FAIL 0% 123 in → 95 out (50 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:low__json_schema FAIL 0% 123 in → 19 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:max FAIL 0% 115 in → 2,032 out (1,772 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:max__json_schema FAIL 0% 115 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:medium FAIL 0% 123 in → 58 out (38 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:medium__json_schema FAIL 0% 123 in → 14 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:none FAIL 0% 115 in → 8 out (0 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:none__json_schema FAIL 0% 115 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:xhigh FAIL 0% 115 in → 2,250 out (2,088 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
ibm-granite/granite-4.2-8b:xhigh__json_schema FAIL 0% 115 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
inception/mercury-2.5:low FAIL 0% 93 in → 259 out (250 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
inception/mercury-2.5:low__json_schema FAIL 0% 292 in → 483 out (251 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
inception/mercury-2.5:max FAIL 0% 92 in → 1,530 out (1,521 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
inception/mercury-2.5:none__json_schema FAIL 0% 294 in → 153 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
liquid/lfm-2-24b-a2b FAIL 0% 120 in → 120 out
Here’s a resolved version of `version.txt` after merging: ``` VERSION=2.0.0 ``` **Explanation:** Since Branch A (release) is typically the target branch for merges, its changes (`VERSION=2.0.0`) are kept. If you want to keep both versions, you could also choose `VERSION=1.1.0`, but usually, the branch being merged into (here, main) takes precedence unless specified otherwise. Here, I chose the release branch's change as the resolution.
Failure: Content mismatch: ['version.txt']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: Here’s a resolved version of `version.txt` that merges both changes logically: ``` VERSION=2.0.0 ``` **Explanation:** - Branch A (release) set `VERSION=2.0.0`, which is typically the final stable release. - Branch B (main) set `VERSION=1.1.0`, but since Branch A supersedes it, we keep the higher version (`2.0.0`). If you want to preserve both versions for reference, you could instead use: ``` VERSION=2.0.0 # Release branch # or VERSION=1.1.0 # Main branch ``` But by default, the merged result should reflect the latest stable release.
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:high FAIL 0% 104 in → 1,106 out (1,162 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:low FAIL 0% 104 in → 1,223 out (1,234 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:medium FAIL 0% 104 in → 1,069 out (1,117 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:xhigh FAIL 0% 104 in → 1,124 out (1,211 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0% 104 in → 919 out (912 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"resolved_content":""}
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.7-code:xhigh FAIL 0% 104 in → 303 out (309 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 122 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
Failure: Content mismatch: ['version.txt']
nvidia/nemotron-3-super-120b-a12b:high__json_schema FAIL 0% 122 in → 7,560 out (7,385 reasoning)
VERSION=2.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.1.0" }
Failure: Content mismatch: ['version.txt']
nvidia/nemotron-3-super-120b-a12b:low FAIL 0% 122 in → 525 out (512 reasoning)
VERSION=2.1.0
Failure: Content mismatch: ['version.txt']
poolside/laguna-xs.2:none FAIL 0% 148 in → 13 out (0 reasoning)
``` VERSION=1.1.0 ```
Failure: Content mismatch: ['version.txt']
poolside/laguna-xs.2:none__json_schema FAIL 0% 148 in → 14 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
Failure: Content mismatch: ['version.txt']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 108 in → 19 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3cbcaed3-eebf-9392-9377-87470a91d6af","request_id":"3cbcaed3-eebf-9392-9377-87470a91d6af"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7e3053da-6baa-98ef-8ad4-e191e982f7bd","request_id":"7e3053da-6baa-98ef-8ad4-e191e982f7bd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d6c6d4ac-4788-9054-98f4-199dd549d02a","request_id":"d6c6d4ac-4788-9054-98f4-199dd549d02a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-899da8a4-3845-985e-99be-f1b00020fb25","request_id":"899da8a4-3845-985e-99be-f1b00020fb25"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1176e09e-eeb9-98b4-89e3-98039a7a052a","request_id":"1176e09e-eeb9-98b4-89e3-98039a7a052a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a7b35d4b-ce0a-9175-adbb-45d8b8126501"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8ebe7dbc-56c4-9572-9887-f78877c71ee7"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-420ecc8b-7273-97eb-941d-2e814187829e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-107b92c9-31ad-9bf2-9441-f0042c1dbbe3"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-eac505c0-c30f-9627-8b6b-6ee68e3e462e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8b3914dc-a689-9c9d-a33a-4e86770364e2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e2f7fc7e-5508-97cd-b3f9-9683bf10c5d2","request_id":"e2f7fc7e-5508-97cd-b3f9-9683bf10c5d2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-88b278db-ade0-9f39-aa80-d878fa4498ac","request_id":"88b278db-ade0-9f39-aa80-d878fa4498ac"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1b8984d0-d1b1-97bc-b787-fa79a6498505","request_id":"1b8984d0-d1b1-97bc-b787-fa79a6498505"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-71bf44fe-7809-956c-9182-25a4497bc7bc","request_id":"71bf44fe-7809-956c-9182-25a4497bc7bc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-919c521c-adb7-9d1c-b9b2-e82bf4506e33","request_id":"919c521c-adb7-9d1c-b9b2-e82bf4506e33"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dda458b8-9bd6-9b0f-aded-f38a4bf7ba47","request_id":"dda458b8-9bd6-9b0f-aded-f38a4bf7ba47"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e8028d81-80e6-913b-a0cc-22433c43ff13","request_id":"e8028d81-80e6-913b-a0cc-22433c43ff13"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0617f206-1be8-9787-b1fc-4638b59df830","request_id":"0617f206-1be8-9787-b1fc-4638b59df830"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3414f683-30c1-9b84-bdcd-fd5fc9523d6d","request_id":"3414f683-30c1-9b84-bdcd-fd5fc9523d6d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-97b6df33-5dad-9616-ac2c-b51e78afc5d6","request_id":"97b6df33-5dad-9616-ac2c-b51e78afc5d6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.8-max:high__json_schema FAIL 0% 146 in → 190 out (168 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
stealth/ox-alpha:high__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
stealth/ox-alpha:low__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
stealth/ox-alpha:max__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling-small:none FAIL 0% 109 in → 10 out (0 reasoning)
VERSION=2.0.0<|end_message|>
Failure: Content mismatch: ['version.txt']
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0{ "resolved_content": "VERSION=2.0.0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 109 in → 10 out (0 reasoning)
VERSION=2.0.0<|end_message|>
Failure: Content mismatch: ['version.txt']
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0{ "resolved_content": "VERSION=2.0.0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:xhigh__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
xiaomi/mimo-v2.6-pro:xhigh FAIL 0% 101 in → 310 out (200 reasoning)
VERSION=2.0.0 **Reasoning:** The base was `1.0.0`. Branch B (main) made a minor bump to `1.1.0`, while Branch A (release) made a deliberate major version bump to `2.0.0`. The `release` branch represents an intentional major release, and semantic versioning dictates that the major version change is the more significant and deliberate action. Taking the highest/most intentional version bump avoids downgrading a major release back to a minor one.
Failure: Content mismatch: ['version.txt']
z-ai/glm-4.7-flash:low__json_schema FAIL 0% 99 in → 2,614 out (2,340 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 99 in → 7 out (0 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:medium__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.3-flash:low FAIL 0% 106 in → 85 out (18 reasoning)
``` VERSION=2.0.0 ``` Rationale: For version files, the release branch's version (2.0.0) represents the most recent release version, which supersedes the 1.1.0 development version on main. Taking the higher version ensures the release doesn't regress.
Failure: Content mismatch: ['version.txt']