Single-line version number conflict
Tests ability to resolve a version number merge conflict. Evaluates semantic versioning awareness in conflict resolution.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'VERSION=1.0.0' > version.txt
  5. 05 git add version.txt
  6. 06 git commit -m 'Initial version'
  7. 07 git checkout -b release
  8. 08 echo 'VERSION=2.0.0' > version.txt
  9. 09 git add version.txt
  10. 10 git commit -m 'Bump major version'
  11. 11 git checkout main
  12. 12 echo 'VERSION=1.1.0' > version.txt
  13. 13 git add version.txt
  14. 14 git commit -m 'Bump minor version'
  15. 15 git merge release
Prompt
Resolve the merge conflict in version.txt. The base version was 'VERSION=1.0.0'. Branch A (release) changed it to 'VERSION=2.0.0'. Branch B (main) changed it to 'VERSION=1.1.0'. Provide ONLY the resolved file content.
Expected
VERSION=2.0.0

Scoped model quality, cost, API time, and token usage for merge_conflicts/f002.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 166 in → 65 out (32 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:high__json_schema PASS 100% 386 in → 50 out (14 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-fable-5:low PASS 100% 166 in → 28 out (10 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:low__json_schema PASS 100% 386 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-fable-5:max PASS 100% 166 in → 261 out (76 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:max__json_schema PASS 100% 386 in → 215 out (60 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-fable-5:medium PASS 100% 166 in → 37 out (20 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:medium__json_schema PASS 100% 386 in → 38 out (9 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-fable-5:xhigh PASS 100% 166 in → 93 out (67 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 386 in → 69 out (25 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-haiku-4.5:high PASS 100% 145 in → 583 out (498 reasoning)
VERSION=2.0.0
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 313 in → 4,319 out (1,672 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:low PASS 100% 145 in → 385 out (375 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 313 in → 987 out (660 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:medium PASS 100% 145 in → 704 out (595 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 313 in → 1,640 out (805 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:none PASS 100% 116 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 283 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 145 in → 705 out (506 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 313 in → 1,105 out (703 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-opus-4.6:high PASS 100% 116 in → 11 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.6:high__json_schema PASS 100% 284 in → 127 out (78 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.6:low PASS 100% 116 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.6:low__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 284 in → 195 out (141 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.6:medium PASS 100% 116 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.6:none__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.6:xhigh PASS 100% 116 in → 11 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 284 in → 128 out (77 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:high PASS 100% 171 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:high__json_schema PASS 100% 391 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.7:low PASS 100% 171 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:low__json_schema PASS 100% 391 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:max PASS 100% 171 in → 203 out (59 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:max__json_schema PASS 100% 391 in → 235 out (59 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:medium PASS 100% 171 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 391 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:none PASS 100% 171 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:none__json_schema PASS 100% 391 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 171 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 391 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.8:high PASS 100% 166 in → 169 out (54 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:high__json_schema PASS 100% 386 in → 86 out (29 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.8:low PASS 100% 166 in → 53 out (22 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:low__json_schema PASS 100% 386 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.8:max PASS 100% 166 in → 330 out (58 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:max__json_schema PASS 100% 386 in → 379 out (46 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 386 in → 42 out (23 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 386 in → 24 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 166 in → 187 out (60 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 386 in → 151 out (52 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:high PASS 100% 116 in → 231 out (192 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 284 in → 109 out (57 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 116 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-4.6:max PASS 100% 116 in → 233 out (166 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 284 in → 304 out (194 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 116 in → 86 out (58 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-4.6:none PASS 100% 116 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 284 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 116 in → 183 out (143 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 284 in → 121 out (89 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-5:high PASS 100% 166 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:high__json_schema PASS 100% 386 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-sonnet-5:low PASS 100% 166 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:low__json_schema PASS 100% 386 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-5:medium PASS 100% 166 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 386 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-5:none PASS 100% 166 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:none__json_schema PASS 100% 386 in → 22 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-5:xhigh PASS 100% 166 in → 133 out (72 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 386 in → 199 out (60 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 105 in → 1,205 out (1,199 reasoning)
VERSION=2.0.0
arcee-ai/trinity-large-thinking:low PASS 100% 105 in → 1,784 out (1,779 reasoning)
VERSION=2.0.0
arcee-ai/trinity-large-thinking:medium PASS 100% 105 in → 1,350 out (1,344 reasoning)
VERSION=2.0.0
arcee-ai/trinity-large-thinking:xhigh PASS 100% 105 in → 1,112 out (1,105 reasoning)
VERSION=2.0.0
arcee-ai/trinity-mini:high__json_schema PASS 100% 105 in → 1,171 out (1,168 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
arcee-ai/trinity-mini:xhigh PASS 100% 105 in → 770 out (766 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:high PASS 100% 109 in → 497 out (487 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 194 in → 413 out (344 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 109 in → 374 out (365 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 255 in → 1,363 out (1,294 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 188 in → 1,259 out (1,250 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 188 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 115 in → 426 out (410 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 109 in → 833 out (776 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 115 in → 9 out (0 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 109 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 109 in → 588 out (558 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 109 in → 465 out (427 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:high PASS 100% 109 in → 630 out (620 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 111 in → 308 out (288 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:low PASS 100% 109 in → 533 out (524 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 111 in → 287 out (267 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:medium PASS 100% 109 in → 313 out (282 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 109 in → 270 out (249 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 109 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 188 in → 1,196 out (1,186 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 190 in → 1,371 out (1,352 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-pro:high PASS 100% 109 in → 679 out (669 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 111 in → 551 out (530 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
deepseek/deepseek-v4-pro:low PASS 100% 109 in → 593 out (557 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:medium PASS 100% 109 in → 462 out (412 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:none PASS 100% 109 in → 8 out (0 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 111 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 188 in → 1,659 out (1,649 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 379 in → 1,622 out (1,606 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3-flash-preview:high PASS 100% 100 in → 848 out (841 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:high__json_schema PASS 100% 101 in → 1,145 out (1,131 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3-flash-preview:low PASS 100% 100 in → 319 out (312 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:low__json_schema PASS 100% 160 in → 14 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3-flash-preview:medium PASS 100% 100 in → 775 out (768 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:medium__json_schema PASS 100% 160 in → 727 out (713 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3-flash-preview:xhigh PASS 100% 100 in → 497 out (490 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 160 in → 566 out (548 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.1-flash-lite-preview:high PASS 100% 100 in → 569 out (562 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 101 in → 527 out (513 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 100 in → 170 out (163 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 160 in → 147 out (128 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 100 in → 386 out (379 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 101 in → 346 out (328 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 101 in → 587 out (580 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 160 in → 870 out (856 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-pro-preview:high PASS 100% 100 in → 620 out (613 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 160 in → 810 out (797 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.1-pro-preview:low PASS 100% 100 in → 413 out (406 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 160 in → 318 out (305 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-pro-preview:medium PASS 100% 100 in → 610 out (603 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 160 in → 352 out (339 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 100 in → 840 out (833 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 160 in → 837 out (818 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:high PASS 100% 101 in → 1,664 out (1,657 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 101 in → 904 out (886 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:low PASS 100% 101 in → 7 out (0 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 101 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:max PASS 100% 100 in → 953 out (946 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 101 in → 1,045 out (1,032 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.5-flash-lite:medium PASS 100% 101 in → 699 out (692 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 160 in → 826 out (807 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 100 in → 1,278 out (1,271 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 160 in → 1,070 out (1,051 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash:high PASS 100% 100 in → 776 out (769 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:high__json_schema PASS 100% 160 in → 926 out (905 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
google/gemini-3.5-flash:low PASS 100% 100 in → 246 out (239 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:low__json_schema PASS 100% 160 in → 252 out (233 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash:medium PASS 100% 100 in → 725 out (718 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:medium__json_schema PASS 100% 160 in → 540 out (521 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash:xhigh PASS 100% 100 in → 1,125 out (1,118 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 160 in → 1,348 out (1,327 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
google/gemini-3.6-flash:high PASS 100% 101 in → 494 out (487 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:high__json_schema PASS 100% 101 in → 1,159 out (1,145 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.6-flash:low PASS 100% 101 in → 253 out (246 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:low__json_schema PASS 100% 160 in → 147 out (126 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
google/gemini-3.6-flash:max PASS 100% 100 in → 494 out (487 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:max__json_schema PASS 100% 160 in → 1,023 out (1,007 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
google/gemini-3.6-flash:medium PASS 100% 101 in → 611 out (604 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:medium__json_schema PASS 100% 160 in → 532 out (513 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.6-flash:xhigh PASS 100% 100 in → 530 out (523 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 101 in → 1,167 out (1,153 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemma-4-26b-a4b-it:high PASS 100% 116 in → 444 out (436 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 116 in → 781 out (668 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-26b-a4b-it:low PASS 100% 115 in → 811 out (690 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:medium PASS 100% 116 in → 749 out (705 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 116 in → 691 out (602 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-26b-a4b-it:none PASS 100% 113 in → 8 out (0 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 113 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 116 in → 774 out (704 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 116 in → 1,074 out (905 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-31b-it:high PASS 100% 116 in → 326 out (1 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:high__json_schema PASS 100% 115 in → 264 out (227 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemma-4-31b-it:low PASS 100% 115 in → 282 out (243 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:low__json_schema PASS 100% 115 in → 219 out (170 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-31b-it:medium PASS 100% 116 in → 230 out (194 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:medium__json_schema PASS 100% 119 in → 260 out (245 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemma-4-31b-it:none PASS 100% 113 in → 8 out (0 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:none__json_schema PASS 100% 113 in → 15 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemma-4-31b-it:xhigh PASS 100% 116 in → 248 out (1 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 119 in → 261 out (246 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
ibm-granite/granite-4.1-8b__json_schema PASS 100% 108 in → 18 out
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
inclusionai/ling-2.6-flash PASS 100% 120 in → 8 out
VERSION=2.0.0
inclusionai/ling-2.6-flash__json_schema PASS 100% 120 in → 19 out
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.5:high PASS 100% 142 in → 382 out (373 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:high__json_schema PASS 100% 135 in → 396 out (359 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.5:low PASS 100% 135 in → 508 out (498 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:low__json_schema PASS 100% 135 in → 373 out (357 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.5:medium PASS 100% 135 in → 578 out (548 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:medium__json_schema PASS 100% 135 in → 366 out (388 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.5:xhigh PASS 100% 135 in → 402 out (393 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 142 in → 248 out (232 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.7:high PASS 100% 135 in → 791 out (790 reasoning)
VERSION=2.0.0
minimax/minimax-m2.7:low PASS 100% 135 in → 700 out (707 reasoning)
VERSION=2.0.0
minimax/minimax-m2.7:low__json_schema PASS 100% 268 in → 643 out (629 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
minimax/minimax-m2.7:medium PASS 100% 135 in → 829 out (831 reasoning)
VERSION=2.0.0
minimax/minimax-m2.7:medium__json_schema PASS 100% 268 in → 516 out (502 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
minimax/minimax-m2.7:xhigh PASS 100% 135 in → 392 out (403 reasoning)
VERSION=2.0.0
minimax/minimax-m3:high PASS 100% 273 in → 388 out (417 reasoning)
``` VERSION=2.0.0 ```
minimax/minimax-m3:high__json_schema PASS 100% 260 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m3:low PASS 100% 273 in → 343 out (371 reasoning)
VERSION=2.0.0
minimax/minimax-m3:low__json_schema PASS 100% 260 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m3:medium PASS 100% 273 in → 358 out (370 reasoning)
``` VERSION=2.0.0 ```
minimax/minimax-m3:medium__json_schema PASS 100% 260 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m3:xhigh PASS 100% 273 in → 256 out (276 reasoning)
VERSION=2.0.0
minimax/minimax-m3:xhigh__json_schema PASS 100% 260 in → 14 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/devstral-2512 PASS 100% 109 in → 9 out
VERSION=2.0.0
mistralai/devstral-2512__json_schema PASS 100% 109 in → 16 out
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:high PASS 100% 121 in → 1,323 out (1,252 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 109 in → 696 out (650 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:low PASS 100% 121 in → 573 out (532 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 109 in → 701 out (635 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:medium PASS 100% 121 in → 716 out (650 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 109 in → 1,405 out (1,375 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:none PASS 100% 121 in → 9 out (0 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 109 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 121 in → 620 out (573 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 109 in → 328 out (302 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
moonshotai/kimi-k2.6:high__json_schema PASS 100% 103 in → 1,390 out (1,397 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.6:low__json_schema PASS 100% 104 in → 983 out (961 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 287 in → 1,261 out (1,246 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
moonshotai/kimi-k2.7-code:high PASS 100% 103 in → 326 out (317 reasoning)
VERSION=2.0.0
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 104 in → 346 out (321 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
moonshotai/kimi-k2.7-code:low PASS 100% 104 in → 262 out (253 reasoning)
VERSION=2.0.0
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 104 in → 474 out (446 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.7-code:medium PASS 100% 103 in → 280 out (271 reasoning)
VERSION=2.0.0
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 188 in → 785 out (769 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 104 in → 186 out (168 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k3:high PASS 100% 181 in → 215 out (193 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:high__json_schema PASS 100% 181 in → 217 out (195 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:low PASS 100% 181 in → 213 out (191 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:low__json_schema PASS 100% 181 in → 263 out (241 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:max PASS 100% 181 in → 341 out (319 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:max__json_schema PASS 100% 181 in → 172 out (150 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:medium PASS 100% 181 in → 255 out (233 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:medium__json_schema PASS 100% 181 in → 150 out (128 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:xhigh PASS 100% 181 in → 256 out (234 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 278 in → 138 out (111 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 122 in → 352 out (323 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 122 in → 1,483 out (1,376 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 122 in → 378 out (394 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 122 in → 6,268 out (6,254 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 122 in → 978 out (900 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 122 in → 8,755 out (9,129 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 122 in → 9 out (0 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 122 in → 385 out (370 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 122 in → 442 out (422 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 122 in → 287 out (266 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 122 in → 171 out (146 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 122 in → 295 out (276 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 122 in → 197 out (175 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 122 in → 9 out (0 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 122 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 122 in → 204 out (180 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 122 in → 466 out (438 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-5.4-mini:high PASS 100% 100 in → 891 out (878 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:high__json_schema PASS 100% 136 in → 902 out (880 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:low PASS 100% 100 in → 114 out (101 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:low__json_schema PASS 100% 136 in → 114 out (92 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:medium PASS 100% 100 in → 375 out (362 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:medium__json_schema PASS 100% 136 in → 528 out (506 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:xhigh PASS 100% 100 in → 10,371 out (10,358 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 136 in → 9,517 out (9,495 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-nano:high PASS 100% 100 in → 65 out (52 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:high__json_schema PASS 100% 136 in → 126 out (103 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:low PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:low__json_schema PASS 100% 136 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:medium PASS 100% 100 in → 128 out (115 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:medium__json_schema PASS 100% 136 in → 71 out (48 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:none__json_schema PASS 100% 136 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 100 in → 163 out (150 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 136 in → 298 out (275 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4:high PASS 100% 100 in → 128 out (115 reasoning)
VERSION=2.0.0
openai/gpt-5.4:high__json_schema PASS 100% 136 in → 112 out (90 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:low PASS 100% 100 in → 74 out (61 reasoning)
VERSION=2.0.0
openai/gpt-5.4:low__json_schema PASS 100% 136 in → 31 out (9 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:medium PASS 100% 100 in → 121 out (108 reasoning)
VERSION=2.0.0
openai/gpt-5.4:medium__json_schema PASS 100% 136 in → 128 out (106 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:xhigh PASS 100% 100 in → 365 out (352 reasoning)
VERSION=2.0.0
openai/gpt-5.4:xhigh__json_schema PASS 100% 136 in → 538 out (516 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:high PASS 100% 100 in → 218 out (205 reasoning)
VERSION=2.0.0
openai/gpt-5.5:high__json_schema PASS 100% 136 in → 461 out (439 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:low PASS 100% 100 in → 105 out (92 reasoning)
VERSION=2.0.0
openai/gpt-5.5:low__json_schema PASS 100% 136 in → 211 out (189 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:medium PASS 100% 100 in → 302 out (289 reasoning)
VERSION=2.0.0
openai/gpt-5.5:medium__json_schema PASS 100% 136 in → 230 out (208 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.5:none__json_schema PASS 100% 136 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.5:xhigh PASS 100% 100 in → 267 out (254 reasoning)
VERSION=2.0.0
openai/gpt-5.5:xhigh__json_schema PASS 100% 136 in → 538 out (516 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:high PASS 100% 100 in → 211 out (198 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:high__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:low PASS 100% 100 in → 98 out (85 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:low__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:max PASS 100% 100 in → 323 out (310 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:max__json_schema PASS 100% 136 in → 266 out (244 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:medium PASS 100% 100 in → 120 out (107 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:xhigh PASS 100% 100 in → 232 out (219 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 136 in → 146 out (124 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:high PASS 100% 100 in → 101 out (88 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:high__json_schema PASS 100% 136 in → 90 out (68 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:low PASS 100% 100 in → 44 out (31 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:low__json_schema PASS 100% 136 in → 60 out (38 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:max PASS 100% 100 in → 103 out (90 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:max__json_schema PASS 100% 136 in → 168 out (146 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:medium PASS 100% 100 in → 70 out (57 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:medium__json_schema PASS 100% 136 in → 66 out (44 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:xhigh PASS 100% 100 in → 76 out (63 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 136 in → 112 out (90 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:high PASS 100% 100 in → 45 out (32 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:high__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:low PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:low__json_schema PASS 100% 136 in → 100 out (78 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:max PASS 100% 100 in → 141 out (128 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:max__json_schema PASS 100% 136 in → 428 out (406 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:medium PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 136 in → 95 out (73 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:none PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:none__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:xhigh PASS 100% 100 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 136 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-oss-120b:high PASS 100% 161 in → 155 out (148 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:high__json_schema PASS 100% 161 in → 559 out (536 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-oss-120b:low PASS 100% 159 in → 82 out (62 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:low__json_schema PASS 100% 163 in → 164 out (133 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
openai/gpt-oss-120b:medium PASS 100% 160 in → 144 out (112 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:medium__json_schema PASS 100% 163 in → 154 out (115 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-oss-120b:xhigh PASS 100% 159 in → 1,130 out (1,103 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 159 in → 680 out (663 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-oss-20b:high PASS 100% 161 in → 300 out (285 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:high__json_schema PASS 100% 159 in → 848 out (860 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-oss-20b:low PASS 100% 165 in → 72 out (56 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:low__json_schema PASS 100% 159 in → 44 out (24 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-oss-20b:medium PASS 100% 159 in → 442 out (369 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:medium__json_schema PASS 100% 161 in → 321 out (297 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-oss-20b:xhigh PASS 100% 159 in → 660 out (635 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 163 in → 1,428 out (1,351 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
poolside/laguna-m.1:high PASS 100% 111 in → 868 out (858 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:high__json_schema PASS 100% 111 in → 756 out (735 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
poolside/laguna-m.1:low PASS 100% 111 in → 449 out (439 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:low__json_schema PASS 100% 111 in → 1,051 out (1,032 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-m.1:medium PASS 100% 111 in → 759 out (749 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:medium__json_schema PASS 100% 111 in → 544 out (529 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-m.1:none PASS 100% 111 in → 9 out (0 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:none__json_schema PASS 100% 111 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-m.1:xhigh PASS 100% 111 in → 818 out (808 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:xhigh__json_schema PASS 100% 111 in → 1,078 out (1,063 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-xs-2.1:high PASS 100% 111 in → 665 out (657 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:high__json_schema PASS 100% 111 in → 474 out (455 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:low PASS 100% 111 in → 487 out (479 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:low__json_schema PASS 100% 111 in → 495 out (476 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:medium PASS 100% 111 in → 521 out (513 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 111 in → 466 out (446 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:none PASS 100% 111 in → 7 out (0 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:none__json_schema PASS 100% 111 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:xhigh PASS 100% 111 in → 1,079 out (1,072 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 111 in → 530 out (511 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs.2:high PASS 100% 148 in → 410 out (400 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:high__json_schema PASS 100% 148 in → 461 out (446 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-xs.2:low PASS 100% 148 in → 519 out (510 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:low__json_schema PASS 100% 148 in → 494 out (474 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs.2:medium PASS 100% 148 in → 660 out (650 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:medium__json_schema PASS 100% 148 in → 300 out (279 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
poolside/laguna-xs.2:xhigh PASS 100% 148 in → 501 out (491 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 148 in → 553 out (533 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
qwen/qwen3.6-27b:high PASS 100% 106 in → 998 out (911 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:high__json_schema PASS 100% 106 in → 1,233 out (1,142 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-27b:low PASS 100% 106 in → 814 out (773 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:low__json_schema PASS 100% 106 in → 1,167 out (1,038 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-27b:medium PASS 100% 106 in → 618 out (606 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:medium__json_schema PASS 100% 106 in → 1,301 out (1,198 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-27b:none PASS 100% 108 in → 8 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:xhigh PASS 100% 106 in → 894 out (844 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 106 in → 1,206 out (1,114 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:high PASS 100% 106 in → 861 out (762 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 106 in → 1,550 out (1,450 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:low PASS 100% 106 in → 1,722 out (1,710 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 106 in → 1,951 out (1,755 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 106 in → 1,504 out (1,384 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 106 in → 1,185 out (1,075 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:none PASS 100% 108 in → 8 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 108 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 106 in → 1,205 out (1,168 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 106 in → 1,609 out (1,518 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-flash:high PASS 100% 106 in → 1,216 out (1,204 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:low PASS 100% 106 in → 1,155 out (1,143 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:medium PASS 100% 106 in → 2,049 out (2,037 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:none PASS 100% 108 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:xhigh PASS 100% 106 in → 1,823 out (1,811 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:high PASS 100% 106 in → 1,219 out (1,207 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:low PASS 100% 106 in → 2,050 out (2,038 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:max PASS 100% 106 in → 1,019 out (1,007 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:medium PASS 100% 106 in → 1,593 out (1,581 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:none PASS 100% 108 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:xhigh PASS 100% 106 in → 1,407 out (1,395 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:high PASS 100% 106 in → 429 out (417 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:low PASS 100% 106 in → 350 out (338 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:medium PASS 100% 106 in → 366 out (354 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:none PASS 100% 108 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:xhigh PASS 100% 106 in → 373 out (361 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:high PASS 100% 106 in → 591 out (579 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:low PASS 100% 106 in → 360 out (348 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:medium PASS 100% 106 in → 380 out (368 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:none PASS 100% 108 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:xhigh PASS 100% 106 in → 457 out (445 reasoning)
VERSION=2.0.0
tencent/hy3:high PASS 100% 110 in → 1,491 out (1,374 reasoning)
VERSION=2.0.0
tencent/hy3:high__json_schema PASS 100% 110 in → 3,303 out (3,198 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n"}
tencent/hy3:low PASS 100% 110 in → 1,718 out (1,489 reasoning)
VERSION=2.0.0
tencent/hy3:low__json_schema PASS 100% 110 in → 1,095 out (1,086 reasoning)
VERSION=2.0.0
tencent/hy3:medium PASS 100% 110 in → 791 out (726 reasoning)
VERSION=2.0.0
tencent/hy3:medium__json_schema PASS 100% 110 in → 1,218 out (1,088 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
tencent/hy3:none PASS 100% 113 in → 8 out (0 reasoning)
VERSION=2.0.0
tencent/hy3:none__json_schema PASS 100% 113 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
tencent/hy3:xhigh PASS 100% 110 in → 3,099 out (3,090 reasoning)
VERSION=2.0.0
tencent/hy3:xhigh__json_schema PASS 100% 110 in → 1,394 out (1,300 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
thinkingmachines/inkling-small:high PASS 100% 109 in → 642 out (616 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:high__json_schema PASS 100% 109 in → 505 out (492 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:low PASS 100% 109 in → 68 out (55 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:low__json_schema PASS 100% 109 in → 82 out (60 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
thinkingmachines/inkling-small:max PASS 100% 109 in → 685 out (672 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:max__json_schema PASS 100% 109 in → 552 out (530 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
thinkingmachines/inkling-small:medium PASS 100% 109 in → 392 out (379 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:medium__json_schema PASS 100% 109 in → 217 out (195 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
thinkingmachines/inkling-small:xhigh PASS 100% 109 in → 394 out (380 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 109 in → 840 out (820 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling:high PASS 100% 109 in → 1,023 out (1,011 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:high__json_schema PASS 100% 109 in → 1,213 out (1,192 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling:low PASS 100% 109 in → 54 out (42 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:max PASS 100% 109 in → 566 out (553 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:max__json_schema PASS 100% 109 in → 3,040 out (3,019 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling:medium PASS 100% 109 in → 615 out (601 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:medium__json_schema PASS 100% 109 in → 620 out (599 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling:xhigh PASS 100% 109 in → 2,092 out (2,078 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:high PASS 100% 279 in → 1,033 out (1,023 reasoning)
``` VERSION=2.0.0 ```
x-ai/grok-4.3:high__json_schema PASS 100% 336 in → 1,475 out (1,459 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:low PASS 100% 285 in → 763 out (756 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:low__json_schema PASS 100% 342 in → 605 out (589 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:max PASS 100% 279 in → 2,329 out (2,322 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:max__json_schema PASS 100% 336 in → 1,165 out (1,149 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:medium PASS 100% 285 in → 1,047 out (1,040 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:medium__json_schema PASS 100% 342 in → 1,269 out (1,253 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:none PASS 100% 277 in → 7 out (0 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:none__json_schema PASS 100% 334 in → 12 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
x-ai/grok-4.3:xhigh PASS 100% 279 in → 2,302 out (2,295 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:xhigh__json_schema PASS 100% 336 in → 1,904 out (1,888 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.5:high PASS 100% 301 in → 1,522 out (1,515 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:high__json_schema PASS 100% 370 in → 610 out (593 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.5:low PASS 100% 301 in → 226 out (219 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:low__json_schema PASS 100% 370 in → 549 out (532 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.5:max PASS 100% 301 in → 624 out (617 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:max__json_schema PASS 100% 370 in → 1,013 out (999 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
x-ai/grok-4.5:medium PASS 100% 301 in → 877 out (870 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:medium__json_schema PASS 100% 370 in → 916 out (902 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
x-ai/grok-4.5:xhigh PASS 100% 301 in → 1,239 out (1,232 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:xhigh__json_schema PASS 100% 370 in → 630 out (613 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-4.7-flash:high PASS 100% 99 in → 1,773 out (1,682 reasoning)
VERSION=2.0.0
z-ai/glm-4.7-flash:high__json_schema PASS 100% 99 in → 1,749 out (1,570 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-4.7-flash:low PASS 100% 99 in → 2,807 out (2,730 reasoning)
VERSION=2.0.0
z-ai/glm-4.7-flash:medium PASS 100% 99 in → 374 out (361 reasoning)
VERSION=2.0.0
z-ai/glm-4.7-flash:none__json_schema PASS 100% 99 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-4.7-flash:xhigh PASS 100% 99 in → 1,323 out (1,314 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:high PASS 100% 106 in → 449 out (370 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:high__json_schema PASS 100% 106 in → 585 out (568 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.2:low PASS 100% 106 in → 853 out (844 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:low__json_schema PASS 100% 106 in → 215 out (183 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.2:medium PASS 100% 106 in → 583 out (574 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:none PASS 100% 100 in → 8 out (0 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:none__json_schema PASS 100% 100 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.2:xhigh PASS 100% 106 in → 1,034 out (859 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:xhigh__json_schema PASS 100% 106 in → 827 out (809 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
anthropic/claude-opus-4.6:max FAIL 0% 116 in → 244 out (150 reasoning)
``` VERSION=2.0.0 ``` **Reasoning:** The `release` branch has the higher version (`2.0.0`), which represents a deliberate major release that supersedes the minor bump (`1.1.0`) on `main`. Taking the higher, more intentional release version is the correct resolution.
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.6:none FAIL 0% 116 in → 122 out (0 reasoning)
**Resolved file content:** ``` VERSION=2.0.0 ``` **Reasoning:** The base version was `1.0.0`. Branch A (release) bumped it to `2.0.0` (a major version increase), while Branch B (main) bumped it to `1.1.0` (a minor version increase). Since `2.0.0` is the higher version and a major release supersedes a minor release, the correct resolution is to accept the release branch's version `2.0.0`.
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.8:medium FAIL 0% 166 in → 220 out (60 reasoning)
``` VERSION=2.0.0 ``` The resolution takes `VERSION=2.0.0` because, under semantic versioning, the release branch's major version bump (1.0.0 → 2.0.0) supersedes main's minor bump (1.0.0 → 1.1.0). The higher version number always wins to avoid version regression.
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.8:none FAIL 0% 166 in → 221 out (0 reasoning)
VERSION=2.0.0 **Reasoning:** Using semantic versioning, `2.0.0` is the higher version and supersedes `1.1.0`. The `2.0.0` release (major version bump) indicates a more significant/newer set of changes than the `1.1.0` minor update, so the resolution keeps the release branch's version. If your workflow instead requires the main branch's version to win (e.g., release was a hotfix branch and 1.1.0 is your latest development target), the resolved content would be: ``` VERSION=1.1.0 ``` But based on standard version-precedence, the answer above (`VERSION=2.0.0`) is the recommended resolution.
Failure: Content mismatch: ['version.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 105 in → 1,286 out (1,357 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:low FAIL 0% 105 in → 1,207 out (1,237 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 105 in → 1,083 out (1,160 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:medium FAIL 0% 105 in → 789 out (792 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 105 in → 912 out (892 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content":"VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 105 in → 735 out (744 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
deepseek/deepseek-v4-flash:none FAIL 0% 109 in → 8 out (0 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 111 in → 626 out (606 reasoning)
VERSION=2.0.1
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.1" }
Failure: Content mismatch: ['version.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
ibm-granite/granite-4.1-8b FAIL 0% 108 in → 8 out
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
liquid/lfm-2-24b-a2b FAIL 0% 120 in → 120 out
Here’s a resolved version of `version.txt` after merging: ``` VERSION=2.0.0 ``` **Explanation:** Since Branch A (release) is typically the target branch for merges, its changes (`VERSION=2.0.0`) are kept. If you want to keep both versions, you could also choose `VERSION=1.1.0`, but usually, the branch being merged into (here, main) takes precedence unless specified otherwise. Here, I chose the release branch's change as the resolution.
Failure: Content mismatch: ['version.txt']
Invalid JSON. Output: Here’s a resolved version of `version.txt` that merges both changes logically: ``` VERSION=2.0.0 ``` **Explanation:** - Branch A (release) set `VERSION=2.0.0`, which is typically the final stable release. - Branch B (main) set `VERSION=1.1.0`, but since Branch A supersedes it, we keep the higher version (`2.0.0`). If you want to preserve both versions for reference, you could instead use: ``` VERSION=2.0.0 # Release branch # or VERSION=1.1.0 # Main branch ``` But by default, the merged result should reflect the latest stable release.
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:high FAIL 0% 104 in → 1,106 out (1,162 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:low FAIL 0% 104 in → 1,223 out (1,234 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:medium FAIL 0% 104 in → 1,069 out (1,117 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:xhigh FAIL 0% 104 in → 1,124 out (1,211 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0% 104 in → 919 out (912 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"resolved_content":""}
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.7-code:xhigh FAIL 0% 104 in → 303 out (309 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 122 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
Failure: Content mismatch: ['version.txt']
nvidia/nemotron-3-super-120b-a12b:high__json_schema FAIL 0% 122 in → 7,560 out (7,385 reasoning)
VERSION=2.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.1.0" }
Failure: Content mismatch: ['version.txt']
nvidia/nemotron-3-super-120b-a12b:low FAIL 0% 122 in → 525 out (512 reasoning)
VERSION=2.1.0
Failure: Content mismatch: ['version.txt']
poolside/laguna-xs.2:none FAIL 0% 148 in → 13 out (0 reasoning)
``` VERSION=1.1.0 ```
Failure: Content mismatch: ['version.txt']
poolside/laguna-xs.2:none__json_schema FAIL 0% 148 in → 14 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
Failure: Content mismatch: ['version.txt']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 108 in → 19 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3cbcaed3-eebf-9392-9377-87470a91d6af","request_id":"3cbcaed3-eebf-9392-9377-87470a91d6af"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7e3053da-6baa-98ef-8ad4-e191e982f7bd","request_id":"7e3053da-6baa-98ef-8ad4-e191e982f7bd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d6c6d4ac-4788-9054-98f4-199dd549d02a","request_id":"d6c6d4ac-4788-9054-98f4-199dd549d02a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-899da8a4-3845-985e-99be-f1b00020fb25","request_id":"899da8a4-3845-985e-99be-f1b00020fb25"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1176e09e-eeb9-98b4-89e3-98039a7a052a","request_id":"1176e09e-eeb9-98b4-89e3-98039a7a052a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a7b35d4b-ce0a-9175-adbb-45d8b8126501"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8ebe7dbc-56c4-9572-9887-f78877c71ee7"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-420ecc8b-7273-97eb-941d-2e814187829e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-107b92c9-31ad-9bf2-9441-f0042c1dbbe3"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-eac505c0-c30f-9627-8b6b-6ee68e3e462e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8b3914dc-a689-9c9d-a33a-4e86770364e2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e2f7fc7e-5508-97cd-b3f9-9683bf10c5d2","request_id":"e2f7fc7e-5508-97cd-b3f9-9683bf10c5d2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-88b278db-ade0-9f39-aa80-d878fa4498ac","request_id":"88b278db-ade0-9f39-aa80-d878fa4498ac"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1b8984d0-d1b1-97bc-b787-fa79a6498505","request_id":"1b8984d0-d1b1-97bc-b787-fa79a6498505"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-71bf44fe-7809-956c-9182-25a4497bc7bc","request_id":"71bf44fe-7809-956c-9182-25a4497bc7bc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-919c521c-adb7-9d1c-b9b2-e82bf4506e33","request_id":"919c521c-adb7-9d1c-b9b2-e82bf4506e33"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dda458b8-9bd6-9b0f-aded-f38a4bf7ba47","request_id":"dda458b8-9bd6-9b0f-aded-f38a4bf7ba47"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e8028d81-80e6-913b-a0cc-22433c43ff13","request_id":"e8028d81-80e6-913b-a0cc-22433c43ff13"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0617f206-1be8-9787-b1fc-4638b59df830","request_id":"0617f206-1be8-9787-b1fc-4638b59df830"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3414f683-30c1-9b84-bdcd-fd5fc9523d6d","request_id":"3414f683-30c1-9b84-bdcd-fd5fc9523d6d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-97b6df33-5dad-9616-ac2c-b51e78afc5d6","request_id":"97b6df33-5dad-9616-ac2c-b51e78afc5d6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 109 in → 10 out (0 reasoning)
VERSION=2.0.0<|end_message|>
Failure: Content mismatch: ['version.txt']
Invalid JSON. Output: VERSION=2.0.0{ "resolved_content": "VERSION=2.0.0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 109 in → 10 out (0 reasoning)
VERSION=2.0.0<|end_message|>
Failure: Content mismatch: ['version.txt']
Invalid JSON. Output: VERSION=2.0.0{ "resolved_content": "VERSION=2.0.0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0% 99 in → 2,614 out (2,340 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 99 in → 7 out (0 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)