f002 — rebase

← rebase
Single-line version number conflict
Tests ability to resolve a version number conflict during rebase. Evaluates handling of semantic version clashes with rebase polarity.
easy rebase version-number conflict-resolution
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'VERSION=1.0.0' > version.txt
  5. 05 git add version.txt
  6. 06 git commit -m 'Initial version'
  7. 07 git checkout -b release
  8. 08 echo 'VERSION=2.0.0' > version.txt
  9. 09 git add version.txt
  10. 10 git commit -m 'Bump major version'
  11. 11 git checkout main
  12. 12 echo 'VERSION=1.1.0' > version.txt
  13. 13 git add version.txt
  14. 14 git commit -m 'Bump minor version'
  15. 15 git rebase release
Prompt
Resolve the rebase conflict in version.txt. The base version was 'VERSION=1.0.0'. The upstream release branch set it to 'VERSION=2.0.0'. The current branch being rebased (main) set it to 'VERSION=1.1.0'. Keep the current branch value. Provide ONLY the resolved file content.
Expected
VERSION=1.1.0
Fixture Metrics

Scoped model quality, cost, API time, and token usage for rebase/f002.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 190 in → 69 out (34 reasoning)
VERSION=1.1.0
anthropic/claude-fable-5:high__json_schema PASS 100% 410 in → 49 out (19 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0\n"}
anthropic/claude-fable-5:low PASS 100% 191 in → 12 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-fable-5:low__json_schema PASS 100% 411 in → 23 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
anthropic/claude-fable-5:max PASS 100% 190 in → 440 out (81 reasoning)
VERSION=1.1.0
anthropic/claude-fable-5:max__json_schema PASS 100% 411 in → 163 out (51 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
anthropic/claude-fable-5:medium PASS 100% 192 in → 46 out (33 reasoning)
VERSION=1.1.0
anthropic/claude-fable-5:medium__json_schema PASS 100% 412 in → 23 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
anthropic/claude-fable-5:xhigh PASS 100% 193 in → 87 out (60 reasoning)
VERSION=1.1.0
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 410 in → 54 out (25 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0\n"}
anthropic/claude-haiku-4.5:high PASS 100% 166 in → 328 out (280 reasoning)
``` VERSION=1.1.0 ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 335 in → 240 out (214 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
anthropic/claude-haiku-4.5:low PASS 100% 165 in → 266 out (223 reasoning)
``` VERSION=1.1.0 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 333 in → 262 out (207 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0\n"}
anthropic/claude-haiku-4.5:medium PASS 100% 166 in → 235 out (209 reasoning)
VERSION=1.1.0
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 335 in → 233 out (196 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
anthropic/claude-haiku-4.5:none PASS 100% 138 in → 14 out (0 reasoning)
``` VERSION=1.1.0 ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 303 in → 19 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0\n"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 167 in → 218 out (175 reasoning)
``` VERSION=1.1.0 ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 332 in → 760 out (499 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0\n"}
anthropic/claude-opus-4.6:high PASS 100% 135 in → 11 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.6:high__json_schema PASS 100% 304 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.6:low PASS 100% 136 in → 10 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.6:low__json_schema PASS 100% 306 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.6:max PASS 100% 137 in → 137 out (110 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.6:max__json_schema PASS 100% 303 in → 168 out (125 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.6:medium PASS 100% 136 in → 10 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 306 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.6:none PASS 100% 136 in → 10 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.6:none__json_schema PASS 100% 304 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 137 in → 11 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 304 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.7:high PASS 100% 197 in → 14 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.7:high__json_schema PASS 100% 414 in → 27 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.7:low PASS 100% 196 in → 14 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.7:low__json_schema PASS 100% 417 in → 27 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.7:max PASS 100% 198 in → 288 out (63 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.7:max__json_schema PASS 100% 418 in → 186 out (49 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.7:medium PASS 100% 196 in → 14 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 418 in → 27 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.7:none PASS 100% 195 in → 14 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.7:none__json_schema PASS 100% 417 in → 27 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 198 in → 14 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 416 in → 27 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.8:high PASS 100% 191 in → 164 out (75 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.8:high__json_schema PASS 100% 411 in → 23 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
anthropic/claude-opus-4.8:low PASS 100% 191 in → 66 out (47 reasoning)
``` VERSION=1.1.0 ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 412 in → 25 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.8:max PASS 100% 190 in → 496 out (128 reasoning)
``` VERSION=1.1.0 ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 410 in → 675 out (120 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0\n"}
anthropic/claude-opus-4.8:medium PASS 100% 190 in → 80 out (68 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 410 in → 25 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-opus-4.8:none PASS 100% 191 in → 12 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.8:none__json_schema PASS 100% 412 in → 22 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
anthropic/claude-opus-4.8:xhigh PASS 100% 192 in → 288 out (95 reasoning)
VERSION=1.1.0
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 410 in → 122 out (55 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-sonnet-4.6:high PASS 100% 137 in → 118 out (89 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 306 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 139 in → 10 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 305 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-sonnet-4.6:max PASS 100% 136 in → 147 out (106 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 304 in → 138 out (97 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 135 in → 10 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 304 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 137 in → 10 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 304 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 137 in → 112 out (84 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 306 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-sonnet-5:high PASS 100% 191 in → 87 out (63 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-5:high__json_schema PASS 100% 410 in → 105 out (47 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-sonnet-5:low PASS 100% 192 in → 12 out (0 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-5:low__json_schema PASS 100% 411 in → 25 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-sonnet-5:medium PASS 100% 191 in → 61 out (48 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 412 in → 136 out (59 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
anthropic/claude-sonnet-5:none__json_schema PASS 100% 410 in → 24 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 191 in → 242 out (67 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 410 in → 90 out (58 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 121 in → 753 out (755 reasoning)
VERSION=1.1.0
arcee-ai/trinity-large-thinking:low PASS 100% 122 in → 630 out (627 reasoning)
VERSION=1.1.0
arcee-ai/trinity-large-thinking:medium PASS 100% 121 in → 585 out (582 reasoning)
VERSION=1.1.0
arcee-ai/trinity-large-thinking:xhigh PASS 100% 122 in → 1,776 out (1,777 reasoning)
VERSION=1.1.0
arcee-ai/trinity-mini:high PASS 100% 120 in → 447 out (413 reasoning)
VERSION=1.1.0
arcee-ai/trinity-mini:high__json_schema PASS 100% 124 in → 317 out (294 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
arcee-ai/trinity-mini:low PASS 100% 123 in → 961 out (988 reasoning)
VERSION=1.1.0
arcee-ai/trinity-mini:low__json_schema PASS 100% 121 in → 505 out (442 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
arcee-ai/trinity-mini:medium PASS 100% 121 in → 620 out (607 reasoning)
VERSION=1.1.0
arcee-ai/trinity-mini:medium__json_schema PASS 100% 121 in → 396 out (369 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
arcee-ai/trinity-mini:xhigh PASS 100% 122 in → 273 out (262 reasoning)
VERSION=1.1.0
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 122 in → 283 out (256 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 206 in → 49 out (40 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 130 in → 186 out (143 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 127 in → 150 out (139 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 128 in → 186 out (172 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 220 in → 168 out (144 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash-0731:medium PASS 100% 128 in → 137 out (128 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 129 in → 109 out (77 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 127 in → 9 out (0 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 127 in → 19 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 208 in → 147 out (138 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 205 in → 456 out (413 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
deepseek/deepseek-v4-flash:high PASS 100% 127 in → 95 out (86 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 130 in → 137 out (114 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
deepseek/deepseek-v4-flash:low PASS 100% 130 in → 105 out (95 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 224 in → 19 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
deepseek/deepseek-v4-flash:medium PASS 100% 127 in → 100 out (71 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 127 in → 128 out (108 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
deepseek/deepseek-v4-flash:none PASS 100% 128 in → 9 out (0 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 129 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 206 in → 437 out (412 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 207 in → 228 out (185 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
deepseek/deepseek-v4-pro:high PASS 100% 129 in → 216 out (206 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 126 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
deepseek/deepseek-v4-pro:low PASS 100% 127 in → 175 out (165 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 319 in → 129 out (112 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
deepseek/deepseek-v4-pro:medium PASS 100% 128 in → 9 out (0 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 128 in → 222 out (202 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
deepseek/deepseek-v4-pro:none PASS 100% 129 in → 9 out (0 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-pro:xhigh PASS 100% 127 in → 194 out (173 reasoning)
VERSION=1.1.0
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 209 in → 405 out (384 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
google/gemini-3-flash-preview:high PASS 100% 121 in → 244 out (237 reasoning)
VERSION=1.1.0
google/gemini-3-flash-preview:high__json_schema PASS 100% 181 in → 1,303 out (1,290 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
google/gemini-3-flash-preview:low PASS 100% 122 in → 269 out (262 reasoning)
VERSION=1.1.0
google/gemini-3-flash-preview:low__json_schema PASS 100% 182 in → 270 out (257 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
google/gemini-3-flash-preview:medium PASS 100% 120 in → 248 out (241 reasoning)
VERSION=1.1.0
google/gemini-3-flash-preview:medium__json_schema PASS 100% 180 in → 1,811 out (1,798 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
google/gemini-3-flash-preview:xhigh PASS 100% 121 in → 200 out (193 reasoning)
VERSION=1.1.0
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 181 in → 510 out (496 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 122 in → 358 out (351 reasoning)
VERSION=1.1.0
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 179 in → 292 out (278 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 121 in → 125 out (118 reasoning)
VERSION=1.1.0
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 123 in → 124 out (105 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 120 in → 326 out (319 reasoning)
VERSION=1.1.0
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 182 in → 310 out (296 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 123 in → 705 out (698 reasoning)
VERSION=1.1.0
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 180 in → 512 out (498 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
google/gemini-3.1-pro-preview:high PASS 100% 121 in → 355 out (348 reasoning)
VERSION=1.1.0
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 179 in → 998 out (983 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
google/gemini-3.1-pro-preview:low PASS 100% 121 in → 291 out (284 reasoning)
VERSION=1.1.0
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 180 in → 579 out (564 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
google/gemini-3.1-pro-preview:medium PASS 100% 121 in → 297 out (290 reasoning)
VERSION=1.1.0
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 179 in → 237 out (224 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 122 in → 376 out (369 reasoning)
VERSION=1.1.0
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 180 in → 498 out (485 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
google/gemini-3.5-flash-lite:high PASS 100% 121 in → 360 out (353 reasoning)
VERSION=1.1.0
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 181 in → 361 out (342 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemini-3.5-flash-lite:low PASS 100% 121 in → 7 out (0 reasoning)
VERSION=1.1.0
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 122 in → 19 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemini-3.5-flash-lite:max PASS 100% 123 in → 551 out (544 reasoning)
VERSION=1.1.0
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 181 in → 566 out (547 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemini-3.5-flash-lite:medium PASS 100% 123 in → 340 out (333 reasoning)
VERSION=1.1.0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 180 in → 547 out (528 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 119 in → 560 out (553 reasoning)
VERSION=1.1.0
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 121 in → 527 out (513 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
google/gemini-3.5-flash:high PASS 100% 120 in → 478 out (471 reasoning)
VERSION=1.1.0
google/gemini-3.5-flash:high__json_schema PASS 100% 181 in → 718 out (704 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
google/gemini-3.5-flash:low PASS 100% 118 in → 233 out (226 reasoning)
VERSION=1.1.0
google/gemini-3.5-flash:low__json_schema PASS 100% 182 in → 334 out (315 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemini-3.5-flash:medium PASS 100% 119 in → 396 out (389 reasoning)
VERSION=1.1.0
google/gemini-3.5-flash:medium__json_schema PASS 100% 182 in → 628 out (609 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemini-3.5-flash:xhigh PASS 100% 120 in → 486 out (479 reasoning)
VERSION=1.1.0
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 180 in → 1,002 out (988 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
google/gemini-3.6-flash:high PASS 100% 123 in → 280 out (273 reasoning)
VERSION=1.1.0
google/gemini-3.6-flash:high__json_schema PASS 100% 180 in → 870 out (855 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
google/gemini-3.6-flash:low PASS 100% 121 in → 265 out (258 reasoning)
VERSION=1.1.0
google/gemini-3.6-flash:low__json_schema PASS 100% 182 in → 21 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
google/gemini-3.6-flash:max PASS 100% 121 in → 368 out (361 reasoning)
VERSION=1.1.0
google/gemini-3.6-flash:max__json_schema PASS 100% 123 in → 702 out (687 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
google/gemini-3.6-flash:medium PASS 100% 121 in → 337 out (330 reasoning)
VERSION=1.1.0
google/gemini-3.6-flash:medium__json_schema PASS 100% 182 in → 538 out (517 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
google/gemini-3.6-flash:xhigh PASS 100% 119 in → 377 out (370 reasoning)
VERSION=1.1.0
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 181 in → 855 out (840 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
google/gemma-4-26b-a4b-it:high PASS 100% 138 in → 236 out (188 reasoning)
VERSION=1.1.0
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 138 in → 285 out (215 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemma-4-26b-a4b-it:low PASS 100% 137 in → 288 out (228 reasoning)
VERSION=1.1.0
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 138 in → 261 out (189 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemma-4-26b-a4b-it:medium PASS 100% 137 in → 271 out (241 reasoning)
VERSION=1.1.0
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 137 in → 256 out (196 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemma-4-26b-a4b-it:none PASS 100% 135 in → 8 out (0 reasoning)
VERSION=1.1.0
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 135 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 138 in → 210 out (183 reasoning)
VERSION=1.1.0
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 136 in → 313 out (253 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemma-4-31b-it:high PASS 100% 136 in → 268 out (213 reasoning)
VERSION=1.1.0
google/gemma-4-31b-it:high__json_schema PASS 100% 137 in → 250 out (180 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemma-4-31b-it:low PASS 100% 137 in → 241 out (192 reasoning)
VERSION=1.1.0
google/gemma-4-31b-it:low__json_schema PASS 100% 135 in → 214 out (158 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
google/gemma-4-31b-it:medium PASS 100% 134 in → 249 out (1 reasoning)
VERSION=1.1.0
google/gemma-4-31b-it:medium__json_schema PASS 100% 141 in → 239 out (224 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
google/gemma-4-31b-it:none PASS 100% 134 in → 8 out (0 reasoning)
VERSION=1.1.0
google/gemma-4-31b-it:none__json_schema PASS 100% 133 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
google/gemma-4-31b-it:xhigh PASS 100% 138 in → 245 out (174 reasoning)
VERSION=1.1.0
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 137 in → 261 out (203 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
ibm-granite/granite-4.1-8b PASS 100% 127 in → 8 out
VERSION=1.1.0
ibm-granite/granite-4.1-8b__json_schema PASS 100% 128 in → 18 out
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
inclusionai/ling-2.6-flash PASS 100% 143 in → 8 out
VERSION=1.1.0
inclusionai/ling-2.6-flash__json_schema PASS 100% 143 in → 13 out
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
liquid/lfm-2-24b-a2b PASS 100% 139 in → 12 out
version.txt VERSION=1.1.0
minimax/minimax-m2.5:high PASS 100% 153 in → 109 out (104 reasoning)
VERSION=1.1.0
minimax/minimax-m2.5:high__json_schema PASS 100% 155 in → 159 out (144 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
minimax/minimax-m2.5:low PASS 100% 154 in → 190 out (171 reasoning)
VERSION=1.1.0
minimax/minimax-m2.5:low__json_schema PASS 100% 156 in → 322 out (284 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
minimax/minimax-m2.5:medium PASS 100% 155 in → 268 out (256 reasoning)
VERSION=1.1.0
minimax/minimax-m2.5:medium__json_schema PASS 100% 154 in → 157 out (141 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
minimax/minimax-m2.5:xhigh PASS 100% 153 in → 183 out (173 reasoning)
VERSION=1.1.0
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 155 in → 134 out (117 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
minimax/minimax-m2.7:high PASS 100% 153 in → 202 out (193 reasoning)
VERSION=1.1.0
minimax/minimax-m2.7:high__json_schema PASS 100% 155 in → 16 out (14 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
minimax/minimax-m2.7:low PASS 100% 153 in → 147 out (131 reasoning)
VERSION=1.1.0
minimax/minimax-m2.7:medium PASS 100% 155 in → 311 out (310 reasoning)
VERSION=1.1.0
minimax/minimax-m2.7:medium__json_schema PASS 100% 286 in → 350 out (336 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
minimax/minimax-m2.7:xhigh PASS 100% 154 in → 139 out (132 reasoning)
VERSION=1.1.0
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 152 in → 196 out (192 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
minimax/minimax-m3:high PASS 100% 293 in → 187 out (162 reasoning)
``` VERSION=1.1.0 ```
minimax/minimax-m3:high__json_schema PASS 100% 239 in → 17 out (16 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
minimax/minimax-m3:low PASS 100% 293 in → 161 out (137 reasoning)
VERSION=1.1.0
minimax/minimax-m3:medium PASS 100% 293 in → 185 out (172 reasoning)
VERSION=1.1.0
minimax/minimax-m3:medium__json_schema PASS 100% 281 in → 21 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
minimax/minimax-m3:xhigh PASS 100% 291 in → 253 out (240 reasoning)
VERSION=1.1.0
minimax/minimax-m3:xhigh__json_schema PASS 100% 279 in → 17 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
mistralai/devstral-2512 PASS 100% 131 in → 9 out
VERSION=1.1.0
mistralai/devstral-2512__json_schema PASS 100% 130 in → 16 out
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
mistralai/mistral-medium-3-5:high PASS 100% 142 in → 250 out (212 reasoning)
VERSION=1.1.0
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 130 in → 207 out (188 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
mistralai/mistral-medium-3-5:low PASS 100% 142 in → 353 out (318 reasoning)
VERSION=1.1.0
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 131 in → 207 out (197 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
mistralai/mistral-medium-3-5:medium PASS 100% 143 in → 237 out (215 reasoning)
VERSION=1.1.0
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 130 in → 224 out (190 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
mistralai/mistral-medium-3-5:none PASS 100% 140 in → 9 out (0 reasoning)
VERSION=1.1.0
mistralai/mistral-medium-3-5:xhigh PASS 100% 142 in → 448 out (383 reasoning)
VERSION=1.1.0
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 130 in → 196 out (164 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
moonshotai/kimi-k2.6:high__json_schema PASS 100% 122 in → 426 out (408 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
moonshotai/kimi-k2.6:low PASS 100% 120 in → 318 out (309 reasoning)
VERSION=1.1.0
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 306 in → 1,791 out (1,776 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
moonshotai/kimi-k2.6:xhigh PASS 100% 121 in → 352 out (343 reasoning)
VERSION=1.1.0
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 121 in → 436 out (334 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 122 in → 158 out (140 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
moonshotai/kimi-k2.7-code:low PASS 100% 125 in → 143 out (136 reasoning)
VERSION=1.1.0
moonshotai/kimi-k2.7-code:medium PASS 100% 120 in → 147 out (138 reasoning)
VERSION=1.1.0
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 123 in → 169 out (159 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 122 in → 238 out (229 reasoning)
VERSION=1.1.0
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 121 in → 197 out (179 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
moonshotai/kimi-k3:high PASS 100% 200 in → 276 out (254 reasoning)
VERSION=1.1.0
moonshotai/kimi-k3:high__json_schema PASS 100% 198 in → 121 out (96 reasoning)
``` VERSION=1.1.0 ```
moonshotai/kimi-k3:low PASS 100% 200 in → 249 out (227 reasoning)
VERSION=1.1.0
moonshotai/kimi-k3:low__json_schema PASS 100% 198 in → 267 out (245 reasoning)
VERSION=1.1.0
moonshotai/kimi-k3:max PASS 100% 198 in → 121 out (99 reasoning)
VERSION=1.1.0
moonshotai/kimi-k3:max__json_schema PASS 100% 198 in → 576 out (554 reasoning)
VERSION=1.1.0
moonshotai/kimi-k3:medium PASS 100% 198 in → 109 out (87 reasoning)
VERSION=1.1.0
moonshotai/kimi-k3:medium__json_schema PASS 100% 198 in → 142 out (120 reasoning)
VERSION=1.1.0
moonshotai/kimi-k3:xhigh PASS 100% 201 in → 95 out (73 reasoning)
VERSION=1.1.0
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 199 in → 250 out (228 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 144 in → 77 out (63 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 142 in → 63 out (42 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 144 in → 142 out (130 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 144 in → 131 out (101 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 143 in → 184 out (166 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 144 in → 282 out (236 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 144 in → 9 out (0 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 142 in → 19 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 141 in → 85 out (70 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 144 in → 83 out (69 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 142 in → 62 out (52 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 143 in → 73 out (53 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 142 in → 128 out (113 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 144 in → 60 out (40 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 143 in → 104 out (87 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 142 in → 197 out (164 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 144 in → 9 out (0 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 143 in → 19 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 143 in → 59 out (48 reasoning)
VERSION=1.1.0
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 143 in → 85 out (64 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
openai/gpt-5.4-mini:high PASS 100% 118 in → 126 out (113 reasoning)
VERSION=1.1.0
openai/gpt-5.4-mini:high__json_schema PASS 100% 154 in → 163 out (141 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.4-mini:low PASS 100% 119 in → 24 out (11 reasoning)
VERSION=1.1.0
openai/gpt-5.4-mini:low__json_schema PASS 100% 155 in → 43 out (21 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.4-mini:medium PASS 100% 120 in → 56 out (43 reasoning)
VERSION=1.1.0
openai/gpt-5.4-mini:medium__json_schema PASS 100% 154 in → 102 out (80 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.4-mini:none PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.4-mini:none__json_schema PASS 100% 155 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.4-mini:xhigh PASS 100% 119 in → 159 out (146 reasoning)
VERSION=1.1.0
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 157 in → 340 out (317 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
openai/gpt-5.4-nano:high PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.4-nano:high__json_schema PASS 100% 157 in → 21 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
openai/gpt-5.4-nano:low PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.4-nano:low__json_schema PASS 100% 156 in → 21 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
openai/gpt-5.4-nano:medium PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.4-nano:medium__json_schema PASS 100% 157 in → 21 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
openai/gpt-5.4-nano:none PASS 100% 118 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.4-nano:none__json_schema PASS 100% 154 in → 21 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 117 in → 102 out (89 reasoning)
VERSION=1.1.0
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 154 in → 104 out (81 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
openai/gpt-5.4:high PASS 100% 119 in → 88 out (75 reasoning)
VERSION=1.1.0
openai/gpt-5.4:high__json_schema PASS 100% 155 in → 95 out (73 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.4:low PASS 100% 120 in → 32 out (19 reasoning)
VERSION=1.1.0
openai/gpt-5.4:low__json_schema PASS 100% 156 in → 31 out (9 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.4:medium PASS 100% 118 in → 120 out (107 reasoning)
VERSION=1.1.0
openai/gpt-5.4:medium__json_schema PASS 100% 154 in → 63 out (41 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.4:none PASS 100% 118 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.4:none__json_schema PASS 100% 154 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.4:xhigh PASS 100% 120 in → 95 out (82 reasoning)
VERSION=1.1.0
openai/gpt-5.4:xhigh__json_schema PASS 100% 155 in → 210 out (188 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.5:high PASS 100% 118 in → 98 out (85 reasoning)
VERSION=1.1.0
openai/gpt-5.5:high__json_schema PASS 100% 154 in → 267 out (245 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.5:low PASS 100% 118 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.5:low__json_schema PASS 100% 153 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.5:medium PASS 100% 118 in → 74 out (61 reasoning)
VERSION=1.1.0
openai/gpt-5.5:medium__json_schema PASS 100% 155 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.5:none PASS 100% 118 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.5:none__json_schema PASS 100% 155 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.5:xhigh PASS 100% 120 in → 101 out (88 reasoning)
VERSION=1.1.0
openai/gpt-5.5:xhigh__json_schema PASS 100% 155 in → 539 out (516 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0\n"}
openai/gpt-5.6-luna:high PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-luna:high__json_schema PASS 100% 155 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-luna:low PASS 100% 118 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-luna:low__json_schema PASS 100% 154 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-luna:max PASS 100% 118 in → 47 out (34 reasoning)
VERSION=1.1.0
openai/gpt-5.6-luna:max__json_schema PASS 100% 156 in → 218 out (196 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-luna:medium PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 156 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-luna:none PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-luna:none__json_schema PASS 100% 156 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-luna:xhigh PASS 100% 119 in → 44 out (31 reasoning)
VERSION=1.1.0
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 158 in → 190 out (168 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-sol:high PASS 100% 118 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-sol:high__json_schema PASS 100% 155 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-sol:low PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-sol:low__json_schema PASS 100% 156 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-sol:max PASS 100% 118 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-sol:max__json_schema PASS 100% 157 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-sol:medium PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-sol:medium__json_schema PASS 100% 155 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-sol:none PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-sol:none__json_schema PASS 100% 154 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-sol:xhigh PASS 100% 118 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 153 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-terra:high PASS 100% 118 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-terra:high__json_schema PASS 100% 156 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-terra:low PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-terra:low__json_schema PASS 100% 153 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-terra:max PASS 100% 119 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-terra:max__json_schema PASS 100% 156 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-terra:medium PASS 100% 117 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 154 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-terra:none PASS 100% 120 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-terra:none__json_schema PASS 100% 156 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-5.6-terra:xhigh PASS 100% 117 in → 11 out (0 reasoning)
VERSION=1.1.0
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 156 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-oss-120b:high PASS 100% 178 in → 118 out (90 reasoning)
VERSION=1.1.0
openai/gpt-oss-120b:high__json_schema PASS 100% 179 in → 183 out (165 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
openai/gpt-oss-120b:low PASS 100% 178 in → 82 out (75 reasoning)
VERSION=1.1.0
openai/gpt-oss-120b:low__json_schema PASS 100% 168 in → 44 out (17 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
openai/gpt-oss-120b:medium PASS 100% 178 in → 134 out (118 reasoning)
VERSION=1.1.0
openai/gpt-oss-120b:medium__json_schema PASS 100% 165 in → 130 out (89 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
openai/gpt-oss-120b:xhigh PASS 100% 180 in → 166 out (149 reasoning)
VERSION=1.1.0
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 179 in → 245 out (218 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
openai/gpt-oss-20b:high PASS 100% 183 in → 747 out (712 reasoning)
VERSION=1.1.0
openai/gpt-oss-20b:high__json_schema PASS 100% 179 in → 635 out (629 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-oss-20b:low PASS 100% 166 in → 44 out (21 reasoning)
VERSION=1.1.0
openai/gpt-oss-20b:low__json_schema PASS 100% 168 in → 54 out (28 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
openai/gpt-oss-20b:medium PASS 100% 167 in → 208 out (193 reasoning)
VERSION=1.1.0
openai/gpt-oss-20b:medium__json_schema PASS 100% 180 in → 129 out (100 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
openai/gpt-oss-20b:xhigh PASS 100% 168 in → 914 out (909 reasoning)
VERSION=1.1.0
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 231 in → 419 out (456 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
poolside/laguna-m.1:high PASS 100% 130 in → 320 out (311 reasoning)
VERSION=1.1.0
poolside/laguna-m.1:high__json_schema PASS 100% 131 in → 265 out (246 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
poolside/laguna-m.1:low PASS 100% 132 in → 241 out (231 reasoning)
VERSION=1.1.0
poolside/laguna-m.1:low__json_schema PASS 100% 130 in → 329 out (311 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
poolside/laguna-m.1:medium PASS 100% 130 in → 281 out (272 reasoning)
VERSION=1.1.0
poolside/laguna-m.1:medium__json_schema PASS 100% 131 in → 498 out (479 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
poolside/laguna-m.1:none PASS 100% 130 in → 9 out (0 reasoning)
VERSION=1.1.0
poolside/laguna-m.1:xhigh PASS 100% 132 in → 428 out (418 reasoning)
VERSION=1.1.0
poolside/laguna-m.1:xhigh__json_schema PASS 100% 130 in → 208 out (193 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
poolside/laguna-xs-2.1:high PASS 100% 132 in → 195 out (188 reasoning)
VERSION=1.1.0
poolside/laguna-xs-2.1:high__json_schema PASS 100% 131 in → 248 out (229 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
poolside/laguna-xs-2.1:low PASS 100% 130 in → 250 out (242 reasoning)
VERSION=1.1.0
poolside/laguna-xs-2.1:low__json_schema PASS 100% 132 in → 262 out (244 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
poolside/laguna-xs-2.1:medium PASS 100% 131 in → 286 out (278 reasoning)
VERSION=1.1.0
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 132 in → 228 out (207 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
poolside/laguna-xs-2.1:none PASS 100% 132 in → 7 out (0 reasoning)
VERSION=1.1.0
poolside/laguna-xs-2.1:none__json_schema PASS 100% 132 in → 19 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
poolside/laguna-xs-2.1:xhigh PASS 100% 130 in → 2,027 out (2,024 reasoning)
VERSION=1.1.0
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 132 in → 269 out (250 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
poolside/laguna-xs.2:high PASS 100% 168 in → 177 out (167 reasoning)
VERSION=1.1.0
poolside/laguna-xs.2:high__json_schema PASS 100% 168 in → 197 out (182 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
poolside/laguna-xs.2:low PASS 100% 169 in → 210 out (200 reasoning)
VERSION=1.1.0
poolside/laguna-xs.2:low__json_schema PASS 100% 169 in → 202 out (187 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
poolside/laguna-xs.2:medium PASS 100% 167 in → 417 out (409 reasoning)
VERSION=1.1.0
poolside/laguna-xs.2:medium__json_schema PASS 100% 169 in → 216 out (201 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
poolside/laguna-xs.2:none PASS 100% 169 in → 9 out (0 reasoning)
VERSION=1.1.0
poolside/laguna-xs.2:none__json_schema PASS 100% 167 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
poolside/laguna-xs.2:xhigh PASS 100% 168 in → 200 out (190 reasoning)
VERSION=1.1.0
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 169 in → 197 out (182 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
qwen/qwen3.6-27b:high PASS 100% 129 in → 902 out (841 reasoning)
VERSION=1.1.0
qwen/qwen3.6-27b:low PASS 100% 129 in → 981 out (969 reasoning)
VERSION=1.1.0
qwen/qwen3.6-27b:low__json_schema PASS 100% 128 in → 1,071 out (969 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
qwen/qwen3.6-27b:medium PASS 100% 128 in → 332 out (323 reasoning)
VERSION=1.1.0
qwen/qwen3.6-27b:medium__json_schema PASS 100% 126 in → 1,239 out (1 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
qwen/qwen3.6-27b:none PASS 100% 130 in → 8 out (0 reasoning)
VERSION=1.1.0
qwen/qwen3.6-27b:xhigh PASS 100% 128 in → 1,054 out (948 reasoning)
VERSION=1.1.0
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 130 in → 20 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
qwen/qwen3.6-35b-a3b:high PASS 100% 129 in → 1,143 out (1,054 reasoning)
VERSION=1.1.0
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 128 in → 1,049 out (963 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
qwen/qwen3.6-35b-a3b:low PASS 100% 129 in → 981 out (934 reasoning)
VERSION=1.1.0
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 127 in → 1,038 out (948 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 127 in → 608 out (558 reasoning)
VERSION=1.1.0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 129 in → 1,170 out (1,085 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
qwen/qwen3.6-35b-a3b:none PASS 100% 130 in → 8 out (0 reasoning)
VERSION=1.1.0
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 127 in → 1,190 out (1,080 reasoning)
VERSION=1.1.0
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 126 in → 885 out (822 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
qwen/qwen3.6-flash:high PASS 100% 129 in → 1,387 out (1,375 reasoning)
VERSION=1.1.0
qwen/qwen3.6-flash:low PASS 100% 129 in → 598 out (586 reasoning)
VERSION=1.1.0
qwen/qwen3.6-flash:medium PASS 100% 129 in → 1,090 out (1,078 reasoning)
VERSION=1.1.0
qwen/qwen3.6-flash:none PASS 100% 131 in → 7 out (0 reasoning)
VERSION=1.1.0
qwen/qwen3.6-flash:xhigh PASS 100% 128 in → 985 out (973 reasoning)
VERSION=1.1.0
qwen/qwen3.7-flash:high PASS 100% 129 in → 875 out (863 reasoning)
VERSION=1.1.0
qwen/qwen3.7-flash:low PASS 100% 129 in → 1,246 out (1,234 reasoning)
VERSION=1.1.0
qwen/qwen3.7-flash:max PASS 100% 128 in → 927 out (915 reasoning)
VERSION=1.1.0
qwen/qwen3.7-flash:medium PASS 100% 128 in → 909 out (897 reasoning)
VERSION=1.1.0
qwen/qwen3.7-flash:none PASS 100% 129 in → 7 out (0 reasoning)
VERSION=1.1.0
qwen/qwen3.7-flash:xhigh PASS 100% 127 in → 1,086 out (1,074 reasoning)
VERSION=1.1.0
qwen/qwen3.7-max:high PASS 100% 127 in → 214 out (202 reasoning)
VERSION=1.1.0
qwen/qwen3.7-max:low PASS 100% 129 in → 320 out (308 reasoning)
VERSION=1.1.0
qwen/qwen3.7-max:medium PASS 100% 129 in → 126 out (114 reasoning)
VERSION=1.1.0
qwen/qwen3.7-max:none PASS 100% 129 in → 7 out (0 reasoning)
VERSION=1.1.0
qwen/qwen3.7-max:xhigh PASS 100% 128 in → 220 out (208 reasoning)
VERSION=1.1.0
qwen/qwen3.7-plus:high PASS 100% 128 in → 495 out (483 reasoning)
VERSION=1.1.0
qwen/qwen3.7-plus:low PASS 100% 129 in → 652 out (640 reasoning)
VERSION=1.1.0
qwen/qwen3.7-plus:medium PASS 100% 127 in → 528 out (516 reasoning)
VERSION=1.1.0
qwen/qwen3.7-plus:none PASS 100% 130 in → 7 out (0 reasoning)
VERSION=1.1.0
qwen/qwen3.7-plus:xhigh PASS 100% 129 in → 448 out (436 reasoning)
VERSION=1.1.0
tencent/hy3:high PASS 100% 129 in → 789 out (780 reasoning)
VERSION=1.1.0
tencent/hy3:high__json_schema PASS 100% 130 in → 817 out (760 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
tencent/hy3:low PASS 100% 129 in → 275 out (266 reasoning)
VERSION=1.1.0
tencent/hy3:low__json_schema PASS 100% 128 in → 611 out (553 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
tencent/hy3:medium PASS 100% 129 in → 674 out (665 reasoning)
VERSION=1.1.0
tencent/hy3:medium__json_schema PASS 100% 129 in → 477 out (413 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
tencent/hy3:none PASS 100% 133 in → 8 out (0 reasoning)
VERSION=1.1.0
tencent/hy3:none__json_schema PASS 100% 131 in → 16 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
tencent/hy3:xhigh PASS 100% 129 in → 462 out (453 reasoning)
VERSION=1.1.0
tencent/hy3:xhigh__json_schema PASS 100% 132 in → 539 out (512 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
thinkingmachines/inkling-small:high PASS 100% 129 in → 288 out (275 reasoning)
VERSION=1.1.0
thinkingmachines/inkling-small:high__json_schema PASS 100% 129 in → 302 out (284 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
thinkingmachines/inkling-small:low PASS 100% 127 in → 24 out (8 reasoning)
VERSION=1.1.0
thinkingmachines/inkling-small:low__json_schema PASS 100% 129 in → 32 out (12 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
thinkingmachines/inkling-small:max PASS 100% 128 in → 223 out (211 reasoning)
VERSION=1.1.0
thinkingmachines/inkling-small:max__json_schema PASS 100% 129 in → 334 out (313 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
thinkingmachines/inkling-small:medium PASS 100% 127 in → 204 out (191 reasoning)
VERSION=1.1.0
thinkingmachines/inkling-small:medium__json_schema PASS 100% 128 in → 276 out (255 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
thinkingmachines/inkling-small:none PASS 100% 126 in → 10 out (0 reasoning)
VERSION=1.1.0
thinkingmachines/inkling-small:xhigh PASS 100% 129 in → 271 out (258 reasoning)
VERSION=1.1.0
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 128 in → 183 out (162 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
thinkingmachines/inkling:high PASS 100% 128 in → 118 out (104 reasoning)
VERSION=1.1.0
thinkingmachines/inkling:low PASS 100% 129 in → 28 out (14 reasoning)
VERSION=1.1.0
thinkingmachines/inkling:low__json_schema PASS 100% 127 in → 39 out (20 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0\n"}
thinkingmachines/inkling:max PASS 100% 128 in → 349 out (338 reasoning)
VERSION=1.1.0
thinkingmachines/inkling:medium PASS 100% 130 in → 33 out (19 reasoning)
VERSION=1.1.0
thinkingmachines/inkling:none PASS 100% 126 in → 10 out (0 reasoning)
VERSION=1.1.0
thinkingmachines/inkling:none__json_schema PASS 100% 125 in → 18 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
thinkingmachines/inkling:xhigh PASS 100% 128 in → 107 out (93 reasoning)
VERSION=1.1.0
thinkingmachines/inkling:xhigh__json_schema PASS 100% 127 in → 114 out (93 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
x-ai/grok-4.3:high PASS 100% 297 in → 318 out (308 reasoning)
``` VERSION=1.1.0 ```
x-ai/grok-4.3:high__json_schema PASS 100% 356 in → 532 out (520 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
x-ai/grok-4.3:low PASS 100% 303 in → 383 out (376 reasoning)
VERSION=1.1.0
x-ai/grok-4.3:low__json_schema PASS 100% 360 in → 308 out (295 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content": "VERSION=1.1.0"}
x-ai/grok-4.3:max PASS 100% 298 in → 355 out (348 reasoning)
VERSION=1.1.0
x-ai/grok-4.3:max__json_schema PASS 100% 354 in → 318 out (302 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
x-ai/grok-4.3:medium PASS 100% 303 in → 280 out (273 reasoning)
VERSION=1.1.0
x-ai/grok-4.3:medium__json_schema PASS 100% 360 in → 370 out (354 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
x-ai/grok-4.3:none PASS 100% 295 in → 7 out (0 reasoning)
VERSION=1.1.0
x-ai/grok-4.3:none__json_schema PASS 100% 352 in → 12 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) {"resolved_content":"VERSION=1.1.0"}
x-ai/grok-4.3:xhigh PASS 100% 299 in → 270 out (263 reasoning)
VERSION=1.1.0
x-ai/grok-4.3:xhigh__json_schema PASS 100% 354 in → 425 out (409 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
x-ai/grok-4.5:high PASS 100% 317 in → 258 out (251 reasoning)
VERSION=1.1.0
x-ai/grok-4.5:high__json_schema PASS 100% 388 in → 546 out (527 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
x-ai/grok-4.5:low PASS 100% 318 in → 153 out (146 reasoning)
VERSION=1.1.0
x-ai/grok-4.5:low__json_schema PASS 100% 388 in → 243 out (224 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
x-ai/grok-4.5:max PASS 100% 319 in → 250 out (243 reasoning)
VERSION=1.1.0
x-ai/grok-4.5:max__json_schema PASS 100% 388 in → 371 out (352 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
x-ai/grok-4.5:medium PASS 100% 318 in → 216 out (209 reasoning)
VERSION=1.1.0
x-ai/grok-4.5:medium__json_schema PASS 100% 388 in → 398 out (379 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
x-ai/grok-4.5:xhigh PASS 100% 321 in → 175 out (168 reasoning)
VERSION=1.1.0
x-ai/grok-4.5:xhigh__json_schema PASS 100% 386 in → 261 out (242 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
z-ai/glm-4.7-flash:high PASS 100% 119 in → 392 out (359 reasoning)
VERSION=1.1.0
z-ai/glm-4.7-flash:low PASS 100% 118 in → 343 out (315 reasoning)
VERSION=1.1.0
z-ai/glm-4.7-flash:medium PASS 100% 118 in → 2,166 out (2,018 reasoning)
VERSION=1.1.0
z-ai/glm-4.7-flash:none PASS 100% 118 in → 8 out (0 reasoning)
VERSION=1.1.0
z-ai/glm-4.7-flash:none__json_schema PASS 100% 120 in → 17 out (0 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
z-ai/glm-4.7-flash:xhigh PASS 100% 119 in → 392 out (364 reasoning)
VERSION=1.1.0
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 119 in → 344 out (307 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
z-ai/glm-5.2:high PASS 100% 127 in → 352 out (303 reasoning)
VERSION=1.1.0
z-ai/glm-5.2:high__json_schema PASS 100% 303 in → 190 out (171 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0\n" }
z-ai/glm-5.2:low PASS 100% 125 in → 230 out (192 reasoning)
VERSION=1.1.0
z-ai/glm-5.2:low__json_schema PASS 100% 118 in → 276 out (213 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
z-ai/glm-5.2:medium PASS 100% 125 in → 310 out (301 reasoning)
VERSION=1.1.0
z-ai/glm-5.2:medium__json_schema PASS 100% 126 in → 147 out (116 reasoning)
VERSION=1.1.0
JSON Schema Structured Output
(raw) { "resolved_content": "VERSION=1.1.0" }
z-ai/glm-5.2:none PASS 100% 121 in → 8 out (0 reasoning)
VERSION=1.1.0
z-ai/glm-5.2:xhigh PASS 100% 124 in → 371 out (362 reasoning)
VERSION=1.1.0
anthropic/claude-sonnet-5:none FAIL 0% 192 in → 110 out (0 reasoning)
During a rebase, `HEAD` refers to the branch you're rebasing **onto** (upstream, `2.0.0`), while the incoming commit content (below the separator) is from the branch being replayed (main, `1.1.0`). Since you want to keep the current branch's (main's) value: ``` VERSION=1.1.0 ```
Failure: Content mismatch: ['version.txt']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash-0731:max__json_schema FAIL 0% 209 in → 122 out (98 reasoning)
...
JSON Schema Structured Output
(raw) { "resolved_content": "..." }
Failure: Content mismatch: ['version.txt']
deepseek/deepseek-v4-pro:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: version.txt VERSION=1.1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
minimax/minimax-m3:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
mistralai/mistral-medium-3-5:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:high FAIL 0% 123 in → 262 out (254 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.6:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:medium FAIL 0% 123 in → 488 out (457 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.7-code:high FAIL 0% 122 in → 222 out (202 reasoning)
VERSION=1.1.0
Failure: Content mismatch: ['version.txt']
moonshotai/kimi-k2.7-code:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
poolside/laguna-m.1:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-27b:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-04e9e186-8fef-9f8c-b3a7-f139007613f5","request_id":"04e9e186-8fef-9f8c-b3a7-f139007613f5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ab315432-ab6d-9c66-a203-fa8f9e73375b","request_id":"ab315432-ab6d-9c66-a203-fa8f9e73375b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9ae2dd31-5a0c-9949-abf1-00e05df71da2","request_id":"9ae2dd31-5a0c-9949-abf1-00e05df71da2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c25f13ee-72cb-9f47-9229-bc6123fae5cf","request_id":"c25f13ee-72cb-9f47-9229-bc6123fae5cf"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d0a7e622-8985-9535-a953-9f92ca39a7e3","request_id":"d0a7e622-8985-9535-a953-9f92ca39a7e3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7549ee98-17ac-94d1-9383-1ec1ef11c77a"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-590ce80c-8c23-91ee-9a44-8a502d52c718"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a3c47f48-5d9b-91df-93c0-ff31148afd0d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2b77d7e3-d176-9455-b3bc-b4179794b297"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-83ad06dd-6749-96a0-8bab-6e51565904d1"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d1d3a001-f0f8-931b-b68d-b4d9508f8b60"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fe61c0a8-926b-9dde-b998-4f914b6e66e6","request_id":"fe61c0a8-926b-9dde-b998-4f914b6e66e6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-be447fc9-a153-94a1-9185-9eab92ae7d76","request_id":"be447fc9-a153-94a1-9185-9eab92ae7d76"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1b1878f6-ef71-9824-99d7-c89af93b01b6","request_id":"1b1878f6-ef71-9824-99d7-c89af93b01b6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2d277978-01cf-9b1f-abb6-095c97fc0957","request_id":"2d277978-01cf-9b1f-abb6-095c97fc0957"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5b352920-e20e-92fc-b9af-b823d713808b","request_id":"5b352920-e20e-92fc-b9af-b823d713808b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-309c4427-b930-93f2-9a78-b25553c6e55c","request_id":"309c4427-b930-93f2-9a78-b25553c6e55c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-99f18aaa-ca3c-973b-9d07-049dda00f6c3","request_id":"99f18aaa-ca3c-973b-9d07-049dda00f6c3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-995360f2-2074-900b-9d46-348248d7a1fc","request_id":"995360f2-2074-900b-9d46-348248d7a1fc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: VERSION=1.1.0{ "resolved_content": "VERSION=1.1.0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: VERSION=1.1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:max__json_schema FAIL 0%
Invalid JSON. Output: VERSION=1.1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: VERSION=1.1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output: VERSION=1.1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
z-ai/glm-5.2:xhigh__json_schema FAIL 0%
Invalid JSON. Output: VERSION=1.1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)