f002 — cherry_pick

← cherry_pick
Single-line version number conflict
Tests ability to resolve a version number conflict during cherry-pick. Evaluates handling of semantic versioning clashes between branches.
easy cherry-pick conflict-resolution version-number semver
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'VERSION=1.0.0' > version.txt
  5. 05 git add version.txt
  6. 06 git commit -m 'Initial version'
  7. 07 git checkout -b release
  8. 08 echo 'VERSION=2.0.0' > version.txt
  9. 09 git add version.txt
  10. 10 git commit -m 'Bump major version'
  11. 11 git checkout main
  12. 12 echo 'VERSION=1.1.0' > version.txt
  13. 13 git add version.txt
  14. 14 git commit -m 'Bump minor version'
  15. 15 git cherry-pick $(git rev-parse release)
Prompt
Resolve the cherry-pick conflict in version.txt. The base version was 'VERSION=1.0.0'. The incoming commit (release) changed it to 'VERSION=2.0.0'. The current branch (main) has 'VERSION=1.1.0'. Provide ONLY the resolved file content.
Expected
VERSION=2.0.0
Fixture Metrics

Scoped model quality, cost, API time, and token usage for cherry_pick/f002.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 180 in → 62 out (33 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:high__json_schema PASS 100% 401 in → 53 out (18 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-fable-5:low PASS 100% 180 in → 41 out (20 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:low__json_schema PASS 100% 402 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-fable-5:max PASS 100% 180 in → 407 out (92 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:max__json_schema PASS 100% 402 in → 273 out (96 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-fable-5:medium PASS 100% 181 in → 41 out (27 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:medium__json_schema PASS 100% 400 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-fable-5:xhigh PASS 100% 181 in → 95 out (33 reasoning)
VERSION=2.0.0
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 399 in → 63 out (25 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 326 in → 4,265 out (1,511 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:low PASS 100% 158 in → 705 out (616 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 325 in → 588 out (469 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:medium PASS 100% 157 in → 515 out (450 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 324 in → 810 out (613 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 295 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 326 in → 6,004 out (2,528 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-opus-4.6:high PASS 100% 129 in → 15 out (0 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 298 in → 226 out (162 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.6:low PASS 100% 128 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.6:low__json_schema PASS 100% 297 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 296 in → 295 out (215 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.6:medium PASS 100% 128 in → 11 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 296 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.6:none PASS 100% 128 in → 14 out (0 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 296 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 130 in → 15 out (0 reasoning)
``` VERSION=2.0.0 ```
anthropic/claude-opus-4.7:high PASS 100% 186 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:high__json_schema PASS 100% 405 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:low PASS 100% 187 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:low__json_schema PASS 100% 405 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:max PASS 100% 186 in → 385 out (141 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:max__json_schema PASS 100% 406 in → 401 out (80 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:medium PASS 100% 186 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 406 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:none PASS 100% 184 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:none__json_schema PASS 100% 407 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 186 in → 14 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 404 in → 27 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 399 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-opus-4.8:low PASS 100% 179 in → 97 out (51 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:low__json_schema PASS 100% 401 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-opus-4.8:max__json_schema PASS 100% 401 in → 304 out (65 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-opus-4.8:medium PASS 100% 181 in → 107 out (52 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 401 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 401 in → 24 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 180 in → 365 out (56 reasoning)
VERSION=2.0.0
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 401 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:high PASS 100% 129 in → 323 out (194 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 298 in → 261 out (166 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 129 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 296 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:max PASS 100% 130 in → 277 out (186 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 296 in → 284 out (164 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 128 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 296 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 130 in → 10 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 297 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 130 in → 564 out (263 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 296 in → 300 out (176 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-5:high PASS 100% 181 in → 205 out (69 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:high__json_schema PASS 100% 401 in → 173 out (79 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-5:low PASS 100% 180 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:low__json_schema PASS 100% 400 in → 25 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
anthropic/claude-sonnet-5:medium PASS 100% 183 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 401 in → 23 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
anthropic/claude-sonnet-5:none PASS 100% 181 in → 12 out (0 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:none__json_schema PASS 100% 401 in → 24 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 180 in → 184 out (68 reasoning)
VERSION=2.0.0
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 401 in → 119 out (56 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 114 in → 1,348 out (1,360 reasoning)
VERSION=2.0.0
arcee-ai/trinity-large-thinking:low PASS 100% 114 in → 2,227 out (2,251 reasoning)
VERSION=2.0.0
arcee-ai/trinity-large-thinking:medium PASS 100% 113 in → 2,208 out (2,240 reasoning)
VERSION=2.0.0
arcee-ai/trinity-large-thinking:xhigh PASS 100% 113 in → 2,075 out (2,106 reasoning)
VERSION=2.0.0
arcee-ai/trinity-mini:medium PASS 100% 113 in → 1,069 out (980 reasoning)
VERSION=2.0.0
arcee-ai/trinity-mini:xhigh PASS 100% 116 in → 1,170 out (1,159 reasoning)
VERSION=2.0.0
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 113 in → 1,397 out (1,486 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 118 in → 904 out (881 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 200 in → 1,375 out (1,223 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 120 in → 326 out (278 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 120 in → 566 out (486 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 212 in → 4,351 out (3,995 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 199 in → 1,005 out (835 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 126 in → 923 out (877 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 120 in → 1,001 out (927 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 122 in → 8 out (0 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 122 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 200 in → 7,066 out (6,453 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 119 in → 1,092 out (1,057 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:high PASS 100% 120 in → 574 out (566 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 121 in → 303 out (283 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:low PASS 100% 121 in → 470 out (439 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 120 in → 275 out (243 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:medium PASS 100% 119 in → 405 out (395 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 121 in → 303 out (283 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:none PASS 100% 120 in → 9 out (0 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 124 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 199 in → 1,079 out (1,069 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:high PASS 100% 120 in → 865 out (855 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 312 in → 298 out (280 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
deepseek/deepseek-v4-pro:low PASS 100% 120 in → 563 out (536 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 312 in → 201 out (185 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
deepseek/deepseek-v4-pro:medium PASS 100% 121 in → 251 out (241 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 312 in → 450 out (433 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
deepseek/deepseek-v4-pro:none PASS 100% 120 in → 8 out (0 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 312 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 199 in → 1,504 out (1,495 reasoning)
VERSION=2.0.0
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 390 in → 6,307 out (6,289 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
google/gemini-3-flash-preview:high PASS 100% 114 in → 382 out (375 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:high__json_schema PASS 100% 174 in → 367 out (349 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3-flash-preview:low PASS 100% 114 in → 285 out (278 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:low__json_schema PASS 100% 115 in → 13 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3-flash-preview:medium PASS 100% 114 in → 442 out (435 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:medium__json_schema PASS 100% 114 in → 637 out (624 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3-flash-preview:xhigh PASS 100% 113 in → 219 out (212 reasoning)
VERSION=2.0.0
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 173 in → 1,268 out (1,255 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 114 in → 386 out (379 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 173 in → 707 out (694 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 115 in → 153 out (146 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 171 in → 553 out (535 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 114 in → 307 out (300 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 115 in → 374 out (360 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 114 in → 256 out (249 reasoning)
VERSION=2.0.0
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 173 in → 452 out (434 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.1-pro-preview:high PASS 100% 114 in → 784 out (777 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 172 in → 506 out (492 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-pro-preview:low PASS 100% 113 in → 605 out (598 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 174 in → 575 out (561 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-pro-preview:medium PASS 100% 114 in → 531 out (524 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 174 in → 496 out (482 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 114 in → 770 out (763 reasoning)
VERSION=2.0.0
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 173 in → 1,782 out (1,769 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.5-flash-lite:high PASS 100% 115 in → 634 out (627 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 115 in → 1,045 out (1,031 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemini-3.5-flash-lite:low PASS 100% 112 in → 7 out (0 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 174 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:max PASS 100% 113 in → 828 out (821 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 114 in → 743 out (724 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:medium PASS 100% 115 in → 510 out (503 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 114 in → 548 out (529 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 114 in → 611 out (604 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 171 in → 1,050 out (1,031 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemini-3.5-flash:high PASS 100% 111 in → 923 out (916 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:high__json_schema PASS 100% 173 in → 728 out (712 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
google/gemini-3.5-flash:low PASS 100% 114 in → 482 out (475 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:low__json_schema PASS 100% 174 in → 339 out (323 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
google/gemini-3.5-flash:medium PASS 100% 113 in → 549 out (542 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:medium__json_schema PASS 100% 174 in → 806 out (785 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
google/gemini-3.5-flash:xhigh PASS 100% 110 in → 1,121 out (1,114 reasoning)
VERSION=2.0.0
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 174 in → 976 out (960 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
google/gemini-3.6-flash:high PASS 100% 112 in → 534 out (527 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:high__json_schema PASS 100% 174 in → 821 out (808 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
google/gemini-3.6-flash:low PASS 100% 112 in → 314 out (307 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:low__json_schema PASS 100% 115 in → 172 out (157 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
google/gemini-3.6-flash:max PASS 100% 114 in → 270 out (263 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:max__json_schema PASS 100% 112 in → 989 out (974 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
google/gemini-3.6-flash:medium PASS 100% 112 in → 553 out (546 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:medium__json_schema PASS 100% 173 in → 1,028 out (1,013 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
google/gemini-3.6-flash:xhigh PASS 100% 113 in → 779 out (772 reasoning)
VERSION=2.0.0
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 173 in → 961 out (945 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0\n"}
google/gemma-4-26b-a4b-it:high PASS 100% 128 in → 369 out (303 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 127 in → 378 out (314 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-26b-a4b-it:low PASS 100% 128 in → 318 out (274 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 128 in → 353 out (278 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-26b-a4b-it:medium PASS 100% 129 in → 278 out (225 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 130 in → 331 out (287 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-26b-a4b-it:none PASS 100% 126 in → 8 out (0 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 122 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 130 in → 397 out (389 reasoning)
VERSION=2.0.0
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 128 in → 495 out (396 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-31b-it:high PASS 100% 127 in → 203 out (159 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:high__json_schema PASS 100% 128 in → 225 out (160 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemma-4-31b-it:low PASS 100% 128 in → 179 out (134 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:low__json_schema PASS 100% 128 in → 232 out (170 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-31b-it:medium PASS 100% 130 in → 228 out (181 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:medium__json_schema PASS 100% 130 in → 252 out (172 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
google/gemma-4-31b-it:none PASS 100% 127 in → 8 out (0 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:none__json_schema PASS 100% 125 in → 15 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
google/gemma-4-31b-it:xhigh PASS 100% 129 in → 207 out (162 reasoning)
VERSION=2.0.0
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 127 in → 213 out (153 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
ibm-granite/granite-4.1-8b PASS 100% 120 in → 8 out
VERSION=2.0.0
ibm-granite/granite-4.1-8b__json_schema PASS 100% 120 in → 18 out
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
inclusionai/ling-2.6-flash PASS 100% 133 in → 8 out
VERSION=2.0.0
inclusionai/ling-2.6-flash__json_schema PASS 100% 134 in → 19 out
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
liquid/lfm-2-24b-a2b PASS 100% 132 in → 12 out
version.txt VERSION=2.0.0
minimax/minimax-m2.5:high PASS 100% 145 in → 212 out (203 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:high__json_schema PASS 100% 148 in → 481 out (471 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.5:low PASS 100% 147 in → 332 out (309 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:low__json_schema PASS 100% 146 in → 457 out (430 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.5:medium PASS 100% 145 in → 895 out (887 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:medium__json_schema PASS 100% 145 in → 277 out (257 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.5:xhigh PASS 100% 145 in → 432 out (410 reasoning)
VERSION=2.0.0
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 147 in → 727 out (711 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.7:high PASS 100% 149 in → 656 out (648 reasoning)
VERSION=2.0.0
minimax/minimax-m2.7:high__json_schema PASS 100% 145 in → 448 out (417 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.7:low PASS 100% 150 in → 290 out (282 reasoning)
VERSION=2.0.0
minimax/minimax-m2.7:low__json_schema PASS 100% 278 in → 540 out (526 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
minimax/minimax-m2.7:medium PASS 100% 146 in → 665 out (656 reasoning)
VERSION=2.0.0
minimax/minimax-m2.7:medium__json_schema PASS 100% 279 in → 724 out (706 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m2.7:xhigh PASS 100% 149 in → 502 out (494 reasoning)
VERSION=2.0.0
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 278 in → 680 out (666 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
minimax/minimax-m3:high PASS 100% 284 in → 136 out (126 reasoning)
VERSION=2.0.0
minimax/minimax-m3:high__json_schema PASS 100% 271 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
minimax/minimax-m3:low PASS 100% 283 in → 212 out (211 reasoning)
VERSION=2.0.0
minimax/minimax-m3:low__json_schema PASS 100% 272 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
minimax/minimax-m3:medium PASS 100% 283 in → 633 out (644 reasoning)
VERSION=2.0.0
minimax/minimax-m3:medium__json_schema PASS 100% 270 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
minimax/minimax-m3:xhigh PASS 100% 284 in → 235 out (229 reasoning)
VERSION=2.0.0
minimax/minimax-m3:xhigh__json_schema PASS 100% 231 in → 20 out (16 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
mistralai/devstral-2512 PASS 100% 122 in → 9 out
VERSION=2.0.0
mistralai/devstral-2512__json_schema PASS 100% 122 in → 16 out
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:high PASS 100% 132 in → 997 out (887 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 123 in → 241 out (223 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:low PASS 100% 134 in → 1,144 out (1,030 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 122 in → 645 out (626 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:medium PASS 100% 132 in → 1,229 out (1,092 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 123 in → 248 out (218 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:none PASS 100% 134 in → 9 out (0 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 122 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 133 in → 1,810 out (1,546 reasoning)
VERSION=2.0.0
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 122 in → 1,732 out (1,668 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
moonshotai/kimi-k2.6:high PASS 100% 112 in → 2,417 out (2,408 reasoning)
VERSION=2.0.0
moonshotai/kimi-k2.6:high__json_schema PASS 100% 114 in → 6,339 out (6,321 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.6:low__json_schema PASS 100% 113 in → 2,485 out (2,524 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 114 in → 4,085 out (4,068 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 113 in → 1,986 out (1,355 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.7-code:high PASS 100% 113 in → 508 out (499 reasoning)
VERSION=2.0.0
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 114 in → 279 out (268 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
moonshotai/kimi-k2.7-code:low PASS 100% 114 in → 311 out (302 reasoning)
VERSION=2.0.0
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 113 in → 243 out (225 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 115 in → 591 out (573 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 112 in → 982 out (973 reasoning)
VERSION=2.0.0
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 114 in → 841 out (828 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
moonshotai/kimi-k3:high PASS 100% 191 in → 463 out (441 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:low PASS 100% 191 in → 291 out (269 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:low__json_schema PASS 100% 191 in → 512 out (487 reasoning)
``` VERSION=2.0.0 ```
moonshotai/kimi-k3:max PASS 100% 190 in → 294 out (272 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:max__json_schema PASS 100% 190 in → 378 out (356 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:medium__json_schema PASS 100% 191 in → 850 out (828 reasoning)
VERSION=2.0.0
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 189 in → 582 out (560 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 135 in → 399 out (384 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 136 in → 230 out (200 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 135 in → 275 out (266 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 134 in → 4,314 out (4,595 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 136 in → 317 out (299 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 136 in → 369 out (330 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 135 in → 9 out (0 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 136 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 135 in → 1,009 out (992 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 136 in → 417 out (379 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 135 in → 134 out (120 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 135 in → 346 out (315 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 135 in → 408 out (353 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 134 in → 191 out (168 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 136 in → 216 out (200 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 136 in → 242 out (220 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 135 in → 9 out (0 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 136 in → 19 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 135 in → 166 out (147 reasoning)
VERSION=2.0.0
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 134 in → 250 out (228 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-5.4-mini:high PASS 100% 111 in → 496 out (483 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:high__json_schema PASS 100% 146 in → 1,234 out (1,212 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:low PASS 100% 112 in → 138 out (125 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:low__json_schema PASS 100% 149 in → 156 out (134 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:medium PASS 100% 110 in → 361 out (348 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:medium__json_schema PASS 100% 145 in → 374 out (352 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:none PASS 100% 110 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:none__json_schema PASS 100% 146 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-mini:xhigh PASS 100% 111 in → 529 out (516 reasoning)
VERSION=2.0.0
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 146 in → 8,929 out (8,907 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4-nano:high PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:high__json_schema PASS 100% 146 in → 91 out (68 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:low PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:low__json_schema PASS 100% 147 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:medium PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:medium__json_schema PASS 100% 147 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:none PASS 100% 110 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:none__json_schema PASS 100% 148 in → 21 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 110 in → 118 out (105 reasoning)
VERSION=2.0.0
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 147 in → 160 out (137 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.4:high PASS 100% 112 in → 134 out (121 reasoning)
VERSION=2.0.0
openai/gpt-5.4:high__json_schema PASS 100% 146 in → 267 out (245 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:low PASS 100% 113 in → 105 out (92 reasoning)
VERSION=2.0.0
openai/gpt-5.4:low__json_schema PASS 100% 148 in → 82 out (60 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:medium PASS 100% 110 in → 113 out (100 reasoning)
VERSION=2.0.0
openai/gpt-5.4:medium__json_schema PASS 100% 147 in → 143 out (121 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:none PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.4:none__json_schema PASS 100% 146 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.4:xhigh PASS 100% 112 in → 665 out (652 reasoning)
VERSION=2.0.0
openai/gpt-5.4:xhigh__json_schema PASS 100% 148 in → 538 out (516 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:high PASS 100% 112 in → 173 out (160 reasoning)
VERSION=2.0.0
openai/gpt-5.5:high__json_schema PASS 100% 146 in → 538 out (516 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:low PASS 100% 111 in → 71 out (58 reasoning)
VERSION=2.0.0
openai/gpt-5.5:low__json_schema PASS 100% 146 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:medium PASS 100% 111 in → 149 out (136 reasoning)
VERSION=2.0.0
openai/gpt-5.5:medium__json_schema PASS 100% 146 in → 375 out (353 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:none PASS 100% 110 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.5:none__json_schema PASS 100% 147 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.5:xhigh PASS 100% 109 in → 183 out (170 reasoning)
VERSION=2.0.0
openai/gpt-5.5:xhigh__json_schema PASS 100% 146 in → 539 out (516 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-5.6-luna:high PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:high__json_schema PASS 100% 147 in → 125 out (103 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:low PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:low__json_schema PASS 100% 146 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:max PASS 100% 112 in → 177 out (164 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:max__json_schema PASS 100% 148 in → 325 out (303 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:medium PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 147 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:none PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:none__json_schema PASS 100% 146 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-luna:xhigh PASS 100% 114 in → 90 out (77 reasoning)
VERSION=2.0.0
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 147 in → 287 out (265 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:high PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:high__json_schema PASS 100% 148 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:low PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:low__json_schema PASS 100% 147 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:max PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:max__json_schema PASS 100% 146 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:medium PASS 100% 110 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:medium__json_schema PASS 100% 148 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:none PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:none__json_schema PASS 100% 146 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-sol:xhigh PASS 100% 110 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 147 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:high PASS 100% 111 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:high__json_schema PASS 100% 147 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:low PASS 100% 112 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:low__json_schema PASS 100% 147 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:max PASS 100% 110 in → 72 out (59 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:max__json_schema PASS 100% 146 in → 100 out (78 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:medium PASS 100% 112 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 148 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:none PASS 100% 110 in → 11 out (0 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:none__json_schema PASS 100% 147 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-5.6-terra:xhigh PASS 100% 114 in → 52 out (39 reasoning)
VERSION=2.0.0
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 145 in → 20 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-oss-120b:high PASS 100% 173 in → 173 out (157 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:high__json_schema PASS 100% 171 in → 747 out (713 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
openai/gpt-oss-120b:low PASS 100% 160 in → 72 out (45 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:low__json_schema PASS 100% 158 in → 96 out (52 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
openai/gpt-oss-120b:medium PASS 100% 171 in → 113 out (106 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:medium__json_schema PASS 100% 175 in → 244 out (208 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
openai/gpt-oss-120b:xhigh PASS 100% 173 in → 1,684 out (1,780 reasoning)
VERSION=2.0.0
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 170 in → 969 out (932 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0\n"}
openai/gpt-oss-20b:high PASS 100% 172 in → 821 out (768 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:high__json_schema PASS 100% 223 in → 1,462 out (1,431 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-oss-20b:low PASS 100% 169 in → 48 out (25 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:low__json_schema PASS 100% 171 in → 61 out (31 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
openai/gpt-oss-20b:medium PASS 100% 174 in → 417 out (382 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:xhigh PASS 100% 174 in → 633 out (588 reasoning)
VERSION=2.0.0
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 173 in → 728 out (672 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.0.0"}
poolside/laguna-m.1:high PASS 100% 120 in → 753 out (743 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:high__json_schema PASS 100% 123 in → 682 out (662 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
poolside/laguna-m.1:low PASS 100% 123 in → 459 out (449 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:low__json_schema PASS 100% 122 in → 1,005 out (985 reasoning)
--- version.txt VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "--- version.txt\nVERSION=2.0.0\n"}
poolside/laguna-m.1:medium PASS 100% 122 in → 888 out (878 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:medium__json_schema PASS 100% 122 in → 822 out (803 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-m.1:none PASS 100% 120 in → 9 out (0 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:none__json_schema PASS 100% 122 in → 14 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-m.1:xhigh PASS 100% 123 in → 1,077 out (1,067 reasoning)
VERSION=2.0.0
poolside/laguna-m.1:xhigh__json_schema PASS 100% 121 in → 688 out (673 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-xs-2.1:high PASS 100% 122 in → 629 out (621 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:high__json_schema PASS 100% 123 in → 573 out (553 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:low PASS 100% 123 in → 1,258 out (1,250 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:low__json_schema PASS 100% 122 in → 868 out (849 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:medium PASS 100% 123 in → 532 out (524 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 122 in → 970 out (950 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:none PASS 100% 122 in → 7 out (0 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:none__json_schema PASS 100% 122 in → 18 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs-2.1:xhigh PASS 100% 122 in → 533 out (525 reasoning)
VERSION=2.0.0
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 124 in → 492 out (473 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
poolside/laguna-xs.2:high PASS 100% 160 in → 309 out (299 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:high__json_schema PASS 100% 159 in → 239 out (224 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-xs.2:low PASS 100% 157 in → 375 out (365 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:low__json_schema PASS 100% 160 in → 419 out (404 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-xs.2:medium PASS 100% 159 in → 446 out (436 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:medium__json_schema PASS 100% 159 in → 304 out (289 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-xs.2:none PASS 100% 161 in → 9 out (0 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:none__json_schema PASS 100% 159 in → 14 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
poolside/laguna-xs.2:xhigh PASS 100% 161 in → 188 out (178 reasoning)
VERSION=2.0.0
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 161 in → 279 out (258 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
qwen/qwen3.6-27b:high PASS 100% 119 in → 1,138 out (1,102 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:high__json_schema PASS 100% 120 in → 909 out (821 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-27b:low PASS 100% 120 in → 934 out (862 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:low__json_schema PASS 100% 121 in → 1,475 out (1,322 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-27b:medium PASS 100% 121 in → 1,258 out (1,131 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:none PASS 100% 122 in → 8 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:xhigh PASS 100% 120 in → 1,417 out (1,308 reasoning)
VERSION=2.0.0
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 120 in → 1,243 out (1,176 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:high PASS 100% 120 in → 1,403 out (1,370 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 121 in → 957 out (862 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:low PASS 100% 119 in → 956 out (874 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 120 in → 1,153 out (1,055 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 120 in → 1,083 out (1,071 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 121 in → 1,756 out (1,687 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-35b-a3b:none PASS 100% 122 in → 8 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 121 in → 1,379 out (1,229 reasoning)
VERSION=2.0.0
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 119 in → 1,606 out (1,548 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
qwen/qwen3.6-flash:high PASS 100% 121 in → 1,452 out (1,440 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:low PASS 100% 121 in → 1,202 out (1,190 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:medium PASS 100% 120 in → 1,432 out (1,420 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:none PASS 100% 122 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.6-flash:xhigh PASS 100% 120 in → 1,228 out (1,216 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:high PASS 100% 120 in → 1,491 out (1,479 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:low PASS 100% 121 in → 868 out (856 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:max PASS 100% 119 in → 1,206 out (1,194 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:medium PASS 100% 121 in → 1,084 out (1,072 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:none PASS 100% 123 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.7-flash:xhigh PASS 100% 121 in → 1,217 out (1,205 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:high PASS 100% 121 in → 371 out (359 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:low PASS 100% 120 in → 846 out (834 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:medium PASS 100% 116 in → 494 out (482 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:none PASS 100% 123 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.7-max:xhigh PASS 100% 121 in → 456 out (444 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:high PASS 100% 121 in → 954 out (942 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:low PASS 100% 121 in → 476 out (464 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:medium PASS 100% 121 in → 767 out (755 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:none PASS 100% 121 in → 7 out (0 reasoning)
VERSION=2.0.0
qwen/qwen3.7-plus:xhigh PASS 100% 118 in → 1,511 out (1,499 reasoning)
VERSION=2.0.0
tencent/hy3:high PASS 100% 121 in → 5,094 out (5,085 reasoning)
VERSION=2.0.0
tencent/hy3:high__json_schema PASS 100% 121 in → 2,116 out (2,023 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
tencent/hy3:low PASS 100% 122 in → 1,760 out (1,751 reasoning)
VERSION=2.0.0
tencent/hy3:low__json_schema PASS 100% 121 in → 1,159 out (962 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
tencent/hy3:medium PASS 100% 122 in → 1,549 out (1,540 reasoning)
VERSION=2.0.0
tencent/hy3:medium__json_schema PASS 100% 120 in → 1,537 out (1,305 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
tencent/hy3:none PASS 100% 124 in → 8 out (0 reasoning)
VERSION=2.0.0
tencent/hy3:none__json_schema PASS 100% 123 in → 16 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
tencent/hy3:xhigh PASS 100% 121 in → 2,339 out (2,330 reasoning)
VERSION=2.0.0
tencent/hy3:xhigh__json_schema PASS 100% 120 in → 3,217 out (3,029 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling-small:high PASS 100% 119 in → 644 out (594 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:high__json_schema PASS 100% 120 in → 609 out (591 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
thinkingmachines/inkling-small:low PASS 100% 119 in → 49 out (41 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:low__json_schema PASS 100% 119 in → 81 out (63 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
thinkingmachines/inkling-small:max PASS 100% 120 in → 817 out (742 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:max__json_schema PASS 100% 120 in → 813 out (793 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling-small:medium PASS 100% 121 in → 366 out (341 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:medium__json_schema PASS 100% 118 in → 229 out (208 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling-small:none PASS 100% 118 in → 10 out (0 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:xhigh PASS 100% 119 in → 359 out (328 reasoning)
VERSION=2.0.0
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 120 in → 598 out (578 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
thinkingmachines/inkling:high PASS 100% 120 in → 1,044 out (1,032 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:low PASS 100% 121 in → 89 out (76 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:max PASS 100% 121 in → 3,003 out (2,991 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:medium PASS 100% 122 in → 400 out (387 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:none PASS 100% 117 in → 10 out (0 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:xhigh PASS 100% 120 in → 2,102 out (2,088 reasoning)
VERSION=2.0.0
thinkingmachines/inkling:xhigh__json_schema PASS 100% 120 in → 1,762 out (1,742 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:high PASS 100% 288 in → 2,492 out (2,485 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:high__json_schema PASS 100% 345 in → 2,303 out (2,287 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:low PASS 100% 296 in → 876 out (869 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:low__json_schema PASS 100% 352 in → 730 out (717 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
x-ai/grok-4.3:max PASS 100% 289 in → 2,474 out (2,467 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:max__json_schema PASS 100% 347 in → 1,800 out (1,787 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
x-ai/grok-4.3:medium__json_schema PASS 100% 354 in → 1,421 out (1,405 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.3:none PASS 100% 286 in → 10 out (0 reasoning)
``` VERSION=2.0.0 ```
x-ai/grok-4.3:xhigh PASS 100% 289 in → 1,503 out (1,496 reasoning)
VERSION=2.0.0
x-ai/grok-4.3:xhigh__json_schema PASS 100% 347 in → 1,637 out (1,621 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.5:high PASS 100% 311 in → 1,458 out (1,451 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:high__json_schema PASS 100% 379 in → 1,131 out (1,114 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
x-ai/grok-4.5:low PASS 100% 311 in → 652 out (645 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:low__json_schema PASS 100% 380 in → 358 out (339 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
x-ai/grok-4.5:max PASS 100% 312 in → 878 out (871 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:max__json_schema PASS 100% 379 in → 790 out (771 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
x-ai/grok-4.5:medium__json_schema PASS 100% 381 in → 840 out (821 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
x-ai/grok-4.5:xhigh PASS 100% 312 in → 1,413 out (1,406 reasoning)
VERSION=2.0.0
x-ai/grok-4.5:xhigh__json_schema PASS 100% 378 in → 512 out (493 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0\n" }
z-ai/glm-4.7-flash:high PASS 100% 110 in → 755 out (727 reasoning)
VERSION=2.0.0
z-ai/glm-4.7-flash:medium PASS 100% 111 in → 1,083 out (1,105 reasoning)
VERSION=2.0.0
z-ai/glm-4.7-flash:none PASS 100% 111 in → 7 out (0 reasoning)
VERSION=2.0.0
z-ai/glm-4.7-flash:none__json_schema PASS 100% 111 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-4.7-flash:xhigh PASS 100% 110 in → 578 out (548 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:high PASS 100% 118 in → 396 out (359 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:high__json_schema PASS 100% 118 in → 476 out (386 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.2:low PASS 100% 118 in → 191 out (157 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:low__json_schema PASS 100% 112 in → 920 out (810 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.2:medium PASS 100% 119 in → 539 out (530 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:medium__json_schema PASS 100% 110 in → 886 out (725 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.2:none PASS 100% 112 in → 8 out (0 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:none__json_schema PASS 100% 110 in → 17 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.0.0" }
z-ai/glm-5.2:xhigh PASS 100% 118 in → 1,719 out (1,660 reasoning)
VERSION=2.0.0
z-ai/glm-5.2:xhigh__json_schema PASS 100% 112 in → 14 out (0 reasoning)
VERSION=2.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.0.0"}
anthropic/claude-haiku-4.5:high FAIL 0% 158 in → 1,789 out (822 reasoning)
``` VERSION=2.0.0 ``` **Reasoning**: The incoming commit from the release branch intentionally bumps to a major version (2.0.0). Since this is an explicit release commit being cherry-picked with the message "Bump major version," accepting the incoming change (2.0.0) respects the intent of the cherry-pick operation.
Failure: Content mismatch: ['version.txt']
anthropic/claude-haiku-4.5:none FAIL 0% 130 in → 44 out (0 reasoning)
``` VERSION=2.0.0 ``` The incoming commit from the release branch represents an intentional major version bump that should take precedence over the incremental minor version change in main.
Failure: Content mismatch: ['version.txt']
anthropic/claude-haiku-4.5:xhigh FAIL 0% 159 in → 1,200 out (685 reasoning)
``` VERSION=2.0.0 ``` **Reasoning:** When cherry-picking a commit explicitly titled "Bump major version" from the release branch, the intended change (incrementing to 2.0.0) should be accepted. The incoming commit represents the deliberate versioning decision being cherry-picked, so accepting it (resolving to the incoming version) is the appropriate resolution.
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.6:max FAIL 0% 129 in → 374 out (199 reasoning)
``` VERSION=2.0.0 ``` **Reasoning:** The cherry-picked commit's intent was to **bump the major version** (as noted in the commit message `Bump major version`). The change `1.0.0 → 2.0.0` represents a deliberate major release. The current branch's `1.1.0` is a minor increment that is superseded by the major version bump to `2.0.0`. Accepting the incoming cherry-picked version is the correct resolution.
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 299 in → 367 out (184 reasoning)
VERSION=2.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=2.1.0\n"}
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.8:high FAIL 0% 180 in → 190 out (46 reasoning)
``` VERSION=2.0.0 ``` The incoming release commit's intent is to bump the major version, which supersedes the minor bump on `main`. Resolving to `2.0.0` preserves the purpose of the cherry-picked commit (a major version bump takes precedence over the earlier `1.1.0` minor increment).
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.8:max FAIL 0% 182 in → 1,098 out (162 reasoning)
``` VERSION=2.0.0 ``` The "Bump major version" commit's intent is a major version increment, which by semantic versioning resets minor and patch to `0`. Applying that intent to `main` (`1.1.0`) yields `2.0.0` — matching the incoming value, so keep it.
Failure: Content mismatch: ['version.txt']
anthropic/claude-opus-4.8:none FAIL 0% 182 in → 257 out (0 reasoning)
VERSION=2.0.0 Explanation of the resolution: - **Base:** `1.0.0` - **Incoming (release commit being cherry-picked):** `2.0.0` — a major version bump - **Current (main):** `1.1.0` — a minor bump Since the cherry-picked commit's *intent* is to "Bump major version," and `2.0.0` supersedes `1.1.0` in semantic versioning, keeping `2.0.0` is the correct resolution. The minor bump on main (`1.1.0`) is subsumed by the major release. If you instead intended to combine both changes (e.g., the release should reflect main's work too), you might want `2.0.0` still — a major bump inherently resets minor/patch, so no version-number merging is needed here.
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 113 in → 515 out (487 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:high__json_schema FAIL 0% 113 in → 930 out (843 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:low FAIL 0% 114 in → 395 out (366 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 114 in → 667 out (652 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 114 in → 619 out (582 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
Failure: Content mismatch: ['version.txt']
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: version.txt VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:low FAIL 0% 114 in → 1,177 out (1,153 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:medium FAIL 0% 114 in → 1,373 out (1,402 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:xhigh FAIL 0% 114 in → 1,338 out (1,394 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.7-code:medium FAIL 0% 113 in → 486 out (471 reasoning)
VERSION=2.0.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k3:high__json_schema FAIL 0% 287 in → 387 out (359 reasoning)
VERSION=2.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=2.1.0"}
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k3:medium FAIL 0% 190 in → 605 out (583 reasoning)
VERSION=2.1.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k3:xhigh FAIL 0%
(empty output)
Failure: OpenAI-compatible response contained provider error: Provider returned error
openai/gpt-oss-20b:medium__json_schema FAIL 0% 173 in → 220 out (0 reasoning)
>>?? This is …???…? etc…..?.....… ………..uri?……...... ……..……...…??..?????…?……  
??—??…,...…..…-…????…???..…?? 보…??...
JSON Schema Structured Output
(raw) { "resolved_content" : ">>?? This is …???…? etc…..?.....… ………..uri?……...... ……..……...…??..?????…?……  
??—??…,...…..…-…????…???..…?? 보…??..." }
Failure: Content mismatch: ['version.txt']
qwen/qwen3.6-27b:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:none__json_schema FAIL 0% 121 in → 19 out (0 reasoning)
VERSION=2.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=2.1.0" }
Failure: Content mismatch: ['version.txt']
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 123 in → 27 out (0 reasoning)
VERSION=1.1.0 VERSION=2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\nVERSION=2.0.0" }
Failure: Content mismatch: ['version.txt']
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9ade3a9d-bb62-95ae-ab96-34ec751f574f","request_id":"9ade3a9d-bb62-95ae-ab96-34ec751f574f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-60986180-7f8a-948c-86f6-60d6198ff48d","request_id":"60986180-7f8a-948c-86f6-60d6198ff48d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7e8beecd-7ca8-92b3-8860-1f1ecc5a3ca4","request_id":"7e8beecd-7ca8-92b3-8860-1f1ecc5a3ca4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c31fcb96-f865-928e-86e8-aa95d89e41ed","request_id":"c31fcb96-f865-928e-86e8-aa95d89e41ed"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-27b1dc12-4157-9abc-ae19-33a4d41e50a4","request_id":"27b1dc12-4157-9abc-ae19-33a4d41e50a4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-70dfa960-dec1-92e2-a4c1-867046c72ab0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-64dbce53-a6dc-931d-ae78-7e00604925fd"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6663827e-49b7-99fc-aa5e-f0afb789d443"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d786661c-340f-95f8-9600-f5956b0d46d9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-12dea670-e371-9140-bb38-6c27360786a4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-00172ec7-f5cf-9a8a-8d94-339dd07db736"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-54a51cca-1ad9-92a9-88ff-88e66bb9feaa","request_id":"54a51cca-1ad9-92a9-88ff-88e66bb9feaa"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7d9271d7-f19f-9aa4-94f7-455e2a13b665","request_id":"7d9271d7-f19f-9aa4-94f7-455e2a13b665"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ee59cccd-4d17-90f3-8772-e1585499a32c","request_id":"ee59cccd-4d17-90f3-8772-e1585499a32c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3fa3feb5-823e-936b-b3c9-94940acf27cb","request_id":"3fa3feb5-823e-936b-b3c9-94940acf27cb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a44296bc-2c1e-98b3-8d40-abd471501059","request_id":"a44296bc-2c1e-98b3-8d40-abd471501059"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f6ad447e-f08a-9a1e-b248-ac5212d2a220","request_id":"f6ad447e-f08a-9a1e-b248-ac5212d2a220"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-02288b3a-4f91-9724-878e-b800bd3a77ba","request_id":"02288b3a-4f91-9724-878e-b800bd3a77ba"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4600760a-1660-902d-b92e-b2f2bab77d33","request_id":"4600760a-1660-902d-b92e-b2f2bab77d33"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-61fe5577-6e79-9326-b111-ce1b7ef1c090","request_id":"61fe5577-6e79-9326-b111-ce1b7ef1c090"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-832d9a96-4582-9082-b4cd-14d44a999766","request_id":"832d9a96-4582-9082-b4cd-14d44a999766"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0{ "resolved_content": "VERSION=2.0.0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:max__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0{ "resolved_content": "VERSION=2.0.0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:medium FAIL 0% 295 in → 1,932 out (1,834 reasoning)
VERSION=2.0.0 The conflict occurs while cherry-picking the release commit (which intentionally replaces the original 1.0.0 line with 2.0.0) onto main. Because the purpose of the cherry-pick is to bring the release commit’s change across, the resolution accepts the incoming side, leaving only the line VERSION=2.0.0. No other content exists in the file, so that single line is the complete resolved result.
Failure: Content mismatch: ['version.txt']
x-ai/grok-4.3:none__json_schema FAIL 0% 345 in → 12 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
Failure: Content mismatch: ['version.txt']
x-ai/grok-4.5:medium FAIL 0% 312 in → 2,715 out (2,708 reasoning)
VERSION=2.1.0
Failure: Content mismatch: ['version.txt']
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low FAIL 0% 111 in → 4,059 out (4,139 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output: VERSION=2.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)