f001 — merge_conflicts

← merge_conflicts
Single-line greeting conflict
Tests ability to resolve a simple single-line merge conflict in a greeting file. Evaluates basic conflict marker comprehension.
easy merge-conflict single-line basic resolution
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Hello, World!' > greeting.txt
  5. 05 git add greeting.txt
  6. 06 git commit -m 'Initial greeting'
  7. 07 git checkout -b feature
  8. 08 echo 'Hello, World!!!' > greeting.txt
  9. 09 git add greeting.txt
  10. 10 git commit -m 'Add excitement'
  11. 11 git checkout main
  12. 12 echo 'Hello, Planet!' > greeting.txt
  13. 13 git add greeting.txt
  14. 14 git commit -m 'Update greeting'
  15. 15 git merge feature
Prompt
Resolve the merge conflict in greeting.txt. The base version was 'Hello, World!'. Branch A changed it to 'Hello, World!!!'. Branch B changed it to 'Hello, Planet!'. Provide ONLY the resolved file content, nothing else.
Expected
Hello, Planet!!!
Fixture Metrics

Scoped model quality, cost, API time, and token usage for merge_conflicts/f001.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 158 in → 74 out (53 reasoning)
Hello, Planet!!!
anthropic/claude-fable-5:high__json_schema PASS 100% 378 in → 80 out (59 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-fable-5:low PASS 100% 158 in → 32 out (36 reasoning)
Hello, Planet!!!
anthropic/claude-fable-5:low__json_schema PASS 100% 378 in → 42 out (14 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-fable-5:max PASS 100% 158 in → 218 out (53 reasoning)
Hello, Planet!!!
anthropic/claude-fable-5:max__json_schema PASS 100% 378 in → 199 out (50 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-fable-5:medium PASS 100% 158 in → 72 out (44 reasoning)
Hello, Planet!!!
anthropic/claude-fable-5:medium__json_schema PASS 100% 378 in → 44 out (21 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-fable-5:xhigh PASS 100% 158 in → 86 out (54 reasoning)
Hello, Planet!!!
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 378 in → 100 out (53 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!\n"}
anthropic/claude-haiku-4.5:high PASS 100% 127 in → 419 out (438 reasoning)
Hello, Planet!!!
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 295 in → 369 out (360 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-haiku-4.5:low PASS 100% 127 in → 600 out (592 reasoning)
Hello, Planet!!!
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 295 in → 780 out (584 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-haiku-4.5:medium PASS 100% 127 in → 430 out (457 reasoning)
Hello, Planet!!!
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 295 in → 572 out (572 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 265 in → 14 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 127 in → 456 out (487 reasoning)
Hello, Planet!!!
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 295 in → 590 out (569 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-opus-4.6:high PASS 100% 98 in → 79 out (56 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:high__json_schema PASS 100% 266 in → 85 out (57 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.6:low PASS 100% 98 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:low__json_schema PASS 100% 266 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.6:max PASS 100% 98 in → 159 out (121 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:max__json_schema PASS 100% 266 in → 165 out (133 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.6:medium PASS 100% 98 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 266 in → 14 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.6:none PASS 100% 98 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:none__json_schema PASS 100% 266 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.6:xhigh PASS 100% 98 in → 82 out (58 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 266 in → 114 out (84 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.7:high PASS 100% 163 in → 14 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:high__json_schema PASS 100% 383 in → 25 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.7:low PASS 100% 163 in → 14 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:low__json_schema PASS 100% 383 in → 25 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.7:max PASS 100% 163 in → 245 out (70 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:max__json_schema PASS 100% 383 in → 205 out (91 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.7:medium PASS 100% 163 in → 14 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 383 in → 25 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.7:none PASS 100% 163 in → 14 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:none__json_schema PASS 100% 383 in → 25 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.7:xhigh PASS 100% 163 in → 14 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 383 in → 25 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.8:high PASS 100% 158 in → 218 out (61 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:high__json_schema PASS 100% 378 in → 23 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.8:low PASS 100% 158 in → 92 out (61 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:low__json_schema PASS 100% 378 in → 23 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.8:max PASS 100% 158 in → 574 out (135 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:max__json_schema PASS 100% 378 in → 336 out (122 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-opus-4.8:medium PASS 100% 158 in → 112 out (65 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 378 in → 86 out (57 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.8:none PASS 100% 158 in → 12 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:none__json_schema PASS 100% 378 in → 22 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.8:xhigh PASS 100% 158 in → 396 out (150 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 378 in → 153 out (58 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:high PASS 100% 98 in → 161 out (131 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 266 in → 219 out (183 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:low PASS 100% 98 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 266 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:max PASS 100% 98 in → 185 out (164 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 266 in → 209 out (182 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:medium PASS 100% 98 in → 92 out (66 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 266 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:none PASS 100% 98 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 266 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 98 in → 141 out (118 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 266 in → 167 out (128 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-5:high PASS 100% 158 in → 12 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-5:high__json_schema PASS 100% 378 in → 76 out (41 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-sonnet-5:low PASS 100% 158 in → 12 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-5:low__json_schema PASS 100% 378 in → 23 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-5:medium PASS 100% 158 in → 12 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 378 in → 81 out (34 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-5:none PASS 100% 158 in → 12 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-5:none__json_schema PASS 100% 378 in → 22 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-sonnet-5:xhigh PASS 100% 158 in → 12 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 378 in → 186 out (61 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
arcee-ai/trinity-large-thinking:high PASS 100% 85 in → 1,764 out (1,819 reasoning)
Hello, Planet!!!
arcee-ai/trinity-large-thinking:low PASS 100% 85 in → 725 out (730 reasoning)
Hello, Planet!!!
arcee-ai/trinity-large-thinking:medium PASS 100% 85 in → 545 out (542 reasoning)
Hello, Planet!!!
arcee-ai/trinity-large-thinking:xhigh PASS 100% 85 in → 523 out (521 reasoning)
Hello, Planet!!!
arcee-ai/trinity-mini:low PASS 100% 85 in → 378 out (424 reasoning)
Hello, Planet!!!
arcee-ai/trinity-mini:medium__json_schema PASS 100% 85 in → 367 out (412 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 85 in → 958 out (1,076 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 84 in → 340 out (334 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 163 in → 891 out (951 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 84 in → 533 out (595 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 84 in → 702 out (928 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 176 in → 4,484 out (4,479 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 176 in → 2,539 out (2,743 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 84 in → 491 out (526 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 84 in → 763 out (811 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 169 in → 735 out (771 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 84 in → 373 out (409 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
deepseek/deepseek-v4-flash:high PASS 100% 84 in → 331 out (326 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 84 in → 550 out (527 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash:medium PASS 100% 84 in → 665 out (659 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash:xhigh PASS 100% 163 in → 736 out (846 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:high PASS 100% 84 in → 380 out (373 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 276 in → 300 out (285 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
deepseek/deepseek-v4-pro:low PASS 100% 84 in → 971 out (965 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:medium PASS 100% 84 in → 435 out (429 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:none PASS 100% 84 in → 4 out (0 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 276 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 163 in → 1,077 out (1,071 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 165 in → 730 out (714 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3-flash-preview:high PASS 100% 82 in → 843 out (839 reasoning)
Hello, Planet!!!
google/gemini-3-flash-preview:high__json_schema PASS 100% 142 in → 735 out (725 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3-flash-preview:low PASS 100% 82 in → 716 out (712 reasoning)
Hello, Planet!!!
google/gemini-3-flash-preview:low__json_schema PASS 100% 83 in → 1,767 out (1,757 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3-flash-preview:medium PASS 100% 82 in → 1,120 out (1,116 reasoning)
Hello, Planet!!!
google/gemini-3-flash-preview:medium__json_schema PASS 100% 83 in → 3,010 out (3,000 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3-flash-preview:xhigh PASS 100% 82 in → 265 out (261 reasoning)
Hello, Planet!!!
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 142 in → 3,185 out (3,175 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 83 in → 394 out (390 reasoning)
Hello, Planet!!!
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 83 in → 741 out (730 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 82 in → 137 out (133 reasoning)
Hello, Planet!!!
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 83 in → 671 out (661 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 83 in → 165 out (161 reasoning)
Hello, Planet!!!
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 83 in → 305 out (291 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 82 in → 851 out (847 reasoning)
Hello, Planet!!!
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 83 in → 695 out (684 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.1-pro-preview:high PASS 100% 82 in → 291 out (287 reasoning)
Hello, Planet!!!
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 142 in → 627 out (617 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.1-pro-preview:low PASS 100% 82 in → 285 out (281 reasoning)
Hello, Planet!!!
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 142 in → 103 out (92 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.1-pro-preview:medium PASS 100% 82 in → 265 out (261 reasoning)
Hello, Planet!!!
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 142 in → 100 out (89 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 82 in → 302 out (298 reasoning)
Hello, Planet!!!
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 142 in → 162 out (147 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3.5-flash-lite:high PASS 100% 82 in → 620 out (616 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 142 in → 855 out (845 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.5-flash-lite:max PASS 100% 82 in → 589 out (585 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 142 in → 838 out (828 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.5-flash-lite:medium PASS 100% 83 in → 438 out (434 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 142 in → 479 out (464 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 83 in → 674 out (670 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 83 in → 872 out (862 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.5-flash:high PASS 100% 82 in → 639 out (635 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash:high__json_schema PASS 100% 142 in → 457 out (446 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.5-flash:low PASS 100% 82 in → 263 out (259 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash:low__json_schema PASS 100% 142 in → 198 out (183 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3.5-flash:medium PASS 100% 82 in → 453 out (449 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash:medium__json_schema PASS 100% 142 in → 363 out (352 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.5-flash:xhigh PASS 100% 82 in → 676 out (672 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 142 in → 658 out (647 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.6-flash:high PASS 100% 83 in → 323 out (319 reasoning)
Hello, Planet!!!
google/gemini-3.6-flash:high__json_schema PASS 100% 142 in → 709 out (698 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.6-flash:low PASS 100% 83 in → 179 out (175 reasoning)
Hello, Planet!!!
google/gemini-3.6-flash:low__json_schema PASS 100% 83 in → 10 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.6-flash:max PASS 100% 82 in → 397 out (393 reasoning)
Hello, Planet!!!
google/gemini-3.6-flash:max__json_schema PASS 100% 142 in → 578 out (568 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.6-flash:medium PASS 100% 83 in → 454 out (450 reasoning)
Hello, Planet!!!
google/gemini-3.6-flash:medium__json_schema PASS 100% 83 in → 508 out (498 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.6-flash:xhigh PASS 100% 83 in → 325 out (321 reasoning)
Hello, Planet!!!
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 142 in → 532 out (522 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemma-4-26b-a4b-it:high PASS 100% 98 in → 2,514 out (2,509 reasoning)
Hello, Planet!!!
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 98 in → 1,522 out (1,451 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-26b-a4b-it:low PASS 100% 98 in → 4,262 out (3,992 reasoning)
Hello, Planet!!!
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 97 in → 989 out (971 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-26b-a4b-it:medium PASS 100% 98 in → 2,187 out (2,103 reasoning)
Hello, Planet!!!
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 97 in → 2,884 out (2,822 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-26b-a4b-it:none PASS 100% 95 in → 5 out (0 reasoning)
Hello, Planet!!!
google/gemma-4-26b-a4b-it:xhigh PASS 100% 98 in → 1,983 out (1,978 reasoning)
Hello, Planet!!!
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 98 in → 1,626 out (1,613 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-31b-it:high PASS 100% 98 in → 1,780 out (1 reasoning)
Hello, Planet!!!
google/gemma-4-31b-it:high__json_schema PASS 100% 97 in → 1,561 out (1,509 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-31b-it:medium__json_schema PASS 100% 101 in → 891 out (879 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemma-4-31b-it:none__json_schema PASS 100% 95 in → 12 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemma-4-31b-it:xhigh PASS 100% 98 in → 891 out (1 reasoning)
Hello, Planet!!!
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 97 in → 1,516 out (1,496 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
minimax/minimax-m2.5:high PASS 100% 115 in → 580 out (574 reasoning)
Hello, Planet!!!
minimax/minimax-m2.5:high__json_schema PASS 100% 115 in → 570 out (652 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
minimax/minimax-m2.5:medium PASS 100% 115 in → 719 out (727 reasoning)
Hello, Planet!!!
minimax/minimax-m2.5:medium__json_schema PASS 100% 124 in → 294 out (350 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 115 in → 2,783 out (3,280 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
minimax/minimax-m2.7:high PASS 100% 115 in → 371 out (364 reasoning)
Hello, Planet!!!
minimax/minimax-m2.7:low PASS 100% 115 in → 378 out (453 reasoning)
Hello, Planet!!!
minimax/minimax-m2.7:low__json_schema PASS 100% 248 in → 541 out (529 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
minimax/minimax-m2.7:medium PASS 100% 115 in → 698 out (811 reasoning)
Hello, Planet!!!
minimax/minimax-m2.7:medium__json_schema PASS 100% 248 in → 4,411 out (4,396 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
minimax/minimax-m2.7:xhigh PASS 100% 117 in → 816 out (811 reasoning)
Hello, Planet!!!
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 248 in → 456 out (444 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
minimax/minimax-m3:high PASS 100% 253 in → 248 out (296 reasoning)
Hello, Planet!!!
minimax/minimax-m3:low PASS 100% 253 in → 408 out (463 reasoning)
Hello, Planet!!!
minimax/minimax-m3:xhigh PASS 100% 253 in → 329 out (391 reasoning)
Hello, Planet!!!
mistralai/devstral-2512 PASS 100% 85 in → 5 out
Hello, Planet!!!
mistralai/devstral-2512__json_schema PASS 100% 85 in → 13 out
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
mistralai/mistral-medium-3-5:high PASS 100% 97 in → 1,247 out (1,364 reasoning)
Hello, Planet!!!
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 85 in → 800 out (883 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
mistralai/mistral-medium-3-5:low PASS 100% 97 in → 325 out (343 reasoning)
Hello, Planet!!!
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 85 in → 414 out (450 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
mistralai/mistral-medium-3-5:medium PASS 100% 97 in → 366 out (372 reasoning)
Hello, Planet!!!
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 85 in → 254 out (278 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
mistralai/mistral-medium-3-5:none PASS 100% 97 in → 5 out (0 reasoning)
Hello, Planet!!!
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 85 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 97 in → 776 out (837 reasoning)
Hello, Planet!!!
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 85 in → 566 out (629 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
moonshotai/kimi-k2.6:high__json_schema PASS 100% 84 in → 11 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
moonshotai/kimi-k2.6:low PASS 100% 84 in → 891 out (885 reasoning)
Hello, Planet!!!
moonshotai/kimi-k2.6:low__json_schema PASS 100% 84 in → 708 out (694 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
moonshotai/kimi-k2.6:medium PASS 100% 84 in → 704 out (698 reasoning)
Hello, Planet!!!
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 83 in → 688 out (440 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 84 in → 689 out (827 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 84 in → 549 out (627 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 84 in → 1,615 out (1,609 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:high PASS 100% 161 in → 204 out (185 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:high__json_schema PASS 100% 161 in → 138 out (119 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:low PASS 100% 161 in → 247 out (228 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:low__json_schema PASS 100% 161 in → 178 out (159 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:max PASS 100% 161 in → 173 out (154 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:max__json_schema PASS 100% 161 in → 170 out (151 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:medium PASS 100% 161 in → 325 out (306 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:medium__json_schema PASS 100% 161 in → 294 out (275 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:xhigh PASS 100% 161 in → 179 out (160 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 161 in → 167 out (148 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 98 in → 326 out (315 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 98 in → 210 out (193 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 98 in → 248 out (239 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 98 in → 198 out (178 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 98 in → 260 out (251 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 98 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!\n" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 98 in → 411 out (458 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 98 in → 300 out (281 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
openai/gpt-5.4-mini:high PASS 100% 80 in → 214 out (204 reasoning)
Hello, Planet!!!
openai/gpt-5.4-mini:high__json_schema PASS 100% 116 in → 431 out (412 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4-mini:low PASS 100% 80 in → 150 out (140 reasoning)
Hello, Planet!!!
openai/gpt-5.4-mini:low__json_schema PASS 100% 116 in → 112 out (93 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4-mini:medium PASS 100% 80 in → 322 out (312 reasoning)
Hello, Planet!!!
openai/gpt-5.4-mini:medium__json_schema PASS 100% 116 in → 281 out (262 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4-mini:xhigh PASS 100% 80 in → 851 out (841 reasoning)
Hello, Planet!!!
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 116 in → 3,125 out (3,106 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4-nano:xhigh PASS 100% 80 in → 442 out (432 reasoning)
Hello, Planet!!!
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 116 in → 2,121 out (2,100 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
openai/gpt-5.4:high PASS 100% 80 in → 138 out (128 reasoning)
Hello, Planet!!!
openai/gpt-5.4:high__json_schema PASS 100% 116 in → 75 out (56 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4:low PASS 100% 80 in → 52 out (42 reasoning)
Hello, Planet!!!
openai/gpt-5.4:low__json_schema PASS 100% 116 in → 28 out (9 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4:medium PASS 100% 80 in → 124 out (114 reasoning)
Hello, Planet!!!
openai/gpt-5.4:medium__json_schema PASS 100% 116 in → 80 out (61 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4:none PASS 100% 80 in → 8 out (0 reasoning)
Hello, Planet!!!
openai/gpt-5.4:none__json_schema PASS 100% 116 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4:xhigh PASS 100% 80 in → 254 out (244 reasoning)
Hello, Planet!!!
openai/gpt-5.4:xhigh__json_schema PASS 100% 116 in → 266 out (247 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.5:high PASS 100% 80 in → 75 out (65 reasoning)
Hello, Planet!!!
openai/gpt-5.5:high__json_schema PASS 100% 116 in → 89 out (70 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.5:low PASS 100% 80 in → 43 out (33 reasoning)
Hello, Planet!!!
openai/gpt-5.5:low__json_schema PASS 100% 116 in → 56 out (37 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.5:medium PASS 100% 80 in → 61 out (51 reasoning)
Hello, Planet!!!
openai/gpt-5.5:medium__json_schema PASS 100% 116 in → 77 out (58 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.5:none PASS 100% 80 in → 8 out (0 reasoning)
Hello, Planet!!!
openai/gpt-5.5:none__json_schema PASS 100% 116 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.5:xhigh PASS 100% 80 in → 125 out (115 reasoning)
Hello, Planet!!!
openai/gpt-5.5:xhigh__json_schema PASS 100% 116 in → 128 out (109 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-luna:high PASS 100% 80 in → 92 out (82 reasoning)
Hello, Planet!!!
openai/gpt-5.6-luna:high__json_schema PASS 100% 116 in → 286 out (267 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-luna:low PASS 100% 80 in → 77 out (67 reasoning)
Hello, Planet!!!
openai/gpt-5.6-luna:low__json_schema PASS 100% 116 in → 133 out (112 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
openai/gpt-5.6-luna:max PASS 100% 80 in → 178 out (168 reasoning)
Hello, Planet!!!
openai/gpt-5.6-luna:max__json_schema PASS 100% 116 in → 311 out (292 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-luna:medium PASS 100% 80 in → 75 out (65 reasoning)
Hello, Planet!!!
openai/gpt-5.6-luna:medium__json_schema PASS 100% 116 in → 90 out (71 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-luna:xhigh PASS 100% 80 in → 111 out (101 reasoning)
Hello, Planet!!!
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 116 in → 106 out (87 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:high PASS 100% 80 in → 47 out (37 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:high__json_schema PASS 100% 116 in → 49 out (30 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:low PASS 100% 80 in → 42 out (32 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:low__json_schema PASS 100% 116 in → 45 out (26 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:max PASS 100% 80 in → 56 out (46 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:max__json_schema PASS 100% 116 in → 77 out (58 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:medium PASS 100% 80 in → 51 out (41 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:medium__json_schema PASS 100% 116 in → 52 out (33 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:none PASS 100% 80 in → 8 out (0 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:none__json_schema PASS 100% 116 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:xhigh PASS 100% 80 in → 57 out (47 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 116 in → 117 out (98 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:high PASS 100% 80 in → 47 out (37 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:high__json_schema PASS 100% 116 in → 49 out (30 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:low PASS 100% 80 in → 44 out (34 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:low__json_schema PASS 100% 116 in → 74 out (55 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:max PASS 100% 80 in → 111 out (101 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:max__json_schema PASS 100% 116 in → 138 out (119 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:medium PASS 100% 80 in → 39 out (29 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:medium__json_schema PASS 100% 116 in → 61 out (42 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:none PASS 100% 80 in → 8 out (0 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:none__json_schema PASS 100% 116 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:xhigh PASS 100% 80 in → 54 out (44 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 116 in → 154 out (135 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-oss-120b:high PASS 100% 141 in → 335 out (371 reasoning)
Hello, Planet!!!
openai/gpt-oss-120b:medium__json_schema PASS 100% 139 in → 365 out (406 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-oss-120b:xhigh PASS 100% 143 in → 983 out (1,165 reasoning)
Hello, Planet!!!
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 143 in → 495 out (579 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-oss-20b:high PASS 100% 141 in → 339 out (328 reasoning)
Hello, Planet!!!
openai/gpt-oss-20b:high__json_schema PASS 100% 128 in → 588 out (656 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-oss-20b:low__json_schema PASS 100% 139 in → 147 out (137 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-oss-20b:medium PASS 100% 139 in → 476 out (520 reasoning)
Hello, Planet!!!
openai/gpt-oss-20b:medium__json_schema PASS 100% 143 in → 419 out (455 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-oss-20b:xhigh PASS 100% 128 in → 741 out (832 reasoning)
Hello, Planet!!!
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 128 in → 1,616 out (1,899 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
poolside/laguna-m.1:high PASS 100% 93 in → 297 out (290 reasoning)
Hello, Planet!!!
poolside/laguna-m.1:high__json_schema PASS 100% 93 in → 352 out (336 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
poolside/laguna-m.1:low PASS 100% 93 in → 365 out (358 reasoning)
Hello, Planet!!!
poolside/laguna-m.1:low__json_schema PASS 100% 93 in → 292 out (276 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
poolside/laguna-m.1:medium PASS 100% 93 in → 371 out (364 reasoning)
Hello, Planet!!!
poolside/laguna-m.1:medium__json_schema PASS 100% 93 in → 208 out (191 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
poolside/laguna-m.1:none PASS 100% 93 in → 6 out (0 reasoning)
Hello, Planet!!!
poolside/laguna-m.1:none__json_schema PASS 100% 93 in → 11 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-m.1:xhigh PASS 100% 93 in → 416 out (409 reasoning)
Hello, Planet!!!
poolside/laguna-m.1:xhigh__json_schema PASS 100% 93 in → 303 out (291 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-xs-2.1:high PASS 100% 93 in → 514 out (509 reasoning)
Hello, Planet!!!
poolside/laguna-xs-2.1:high__json_schema PASS 100% 93 in → 1,524 out (1,508 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
poolside/laguna-xs-2.1:low PASS 100% 93 in → 345 out (341 reasoning)
Hello, Planet!!!
poolside/laguna-xs-2.1:low__json_schema PASS 100% 93 in → 791 out (776 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
poolside/laguna-xs-2.1:medium PASS 100% 93 in → 1,093 out (1,088 reasoning)
Hello, Planet!!!
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 93 in → 1,540 out (1,528 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-xs-2.1:xhigh PASS 100% 93 in → 435 out (430 reasoning)
Hello, Planet!!!
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 93 in → 979 out (964 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
poolside/laguna-xs.2:high PASS 100% 130 in → 241 out (234 reasoning)
Hello, Planet!!!
poolside/laguna-xs.2:high__json_schema PASS 100% 130 in → 409 out (397 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-xs.2:low PASS 100% 130 in → 738 out (731 reasoning)
Hello, Planet!!!
poolside/laguna-xs.2:medium PASS 100% 130 in → 311 out (305 reasoning)
Hello, Planet!!!
poolside/laguna-xs.2:medium__json_schema PASS 100% 130 in → 249 out (237 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-xs.2:xhigh PASS 100% 130 in → 284 out (277 reasoning)
Hello, Planet!!!
qwen/qwen3.6-27b:high__json_schema PASS 100% 88 in → 1,494 out (1,561 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-27b:low PASS 100% 88 in → 1,398 out (1,469 reasoning)
Hello, Planet!!!
qwen/qwen3.6-27b:medium PASS 100% 88 in → 997 out (1,028 reasoning)
Hello, Planet!!!
qwen/qwen3.6-27b:none PASS 100% 90 in → 5 out (0 reasoning)
Hello, Planet!!!
qwen/qwen3.6-27b:xhigh PASS 100% 88 in → 1,218 out (1,224 reasoning)
Hello, Planet!!!
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 88 in → 1,967 out (1,964 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-35b-a3b:high PASS 100% 88 in → 1,224 out (1,215 reasoning)
Hello, Planet!!!
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 88 in → 1,592 out (1,651 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-35b-a3b:low PASS 100% 88 in → 2,287 out (2,278 reasoning)
Hello, Planet!!!
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 88 in → 2,186 out (2,177 reasoning)
Hello, Planet!!!
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 88 in → 1,306 out (1,349 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-flash:high PASS 100% 88 in → 1,135 out (1,126 reasoning)
Hello, Planet!!!
qwen/qwen3.6-flash:low PASS 100% 88 in → 1,828 out (1,819 reasoning)
Hello, Planet!!!
qwen/qwen3.6-flash:xhigh PASS 100% 88 in → 2,727 out (2,718 reasoning)
Hello, Planet!!!
qwen/qwen3.7-flash:high PASS 100% 88 in → 2,786 out (2,777 reasoning)
Hello, Planet!!!
qwen/qwen3.7-flash:max PASS 100% 88 in → 1,281 out (1,272 reasoning)
Hello, Planet!!!
qwen/qwen3.7-flash:medium PASS 100% 88 in → 1,714 out (1,705 reasoning)
Hello, Planet!!!
qwen/qwen3.7-max:high PASS 100% 88 in → 749 out (740 reasoning)
Hello, Planet!!!
qwen/qwen3.7-max:low PASS 100% 88 in → 818 out (809 reasoning)
Hello, Planet!!!
qwen/qwen3.7-max:medium PASS 100% 88 in → 395 out (386 reasoning)
Hello, Planet!!!
qwen/qwen3.7-max:xhigh PASS 100% 88 in → 551 out (542 reasoning)
Hello, Planet!!!
qwen/qwen3.7-plus:high PASS 100% 88 in → 919 out (910 reasoning)
Hello, Planet!!!
qwen/qwen3.7-plus:low PASS 100% 88 in → 327 out (318 reasoning)
Hello, Planet!!!
qwen/qwen3.7-plus:medium PASS 100% 88 in → 1,046 out (1,037 reasoning)
Hello, Planet!!!
qwen/qwen3.7-plus:xhigh PASS 100% 88 in → 549 out (540 reasoning)
Hello, Planet!!!
tencent/hy3:high PASS 100% 90 in → 5,081 out (5,075 reasoning)
Hello, Planet!!!
tencent/hy3:high__json_schema PASS 100% 90 in → 8,562 out (8,556 reasoning)
Hello, Planet!!!
tencent/hy3:low PASS 100% 90 in → 983 out (1,035 reasoning)
Hello, Planet!!!
tencent/hy3:low__json_schema PASS 100% 90 in → 1,387 out (1,466 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
tencent/hy3:medium PASS 100% 90 in → 2,295 out (2,424 reasoning)
Hello, Planet!!!
tencent/hy3:none__json_schema PASS 100% 93 in → 11 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
tencent/hy3:xhigh__json_schema PASS 100% 90 in → 5,734 out (6,253 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
thinkingmachines/inkling-small:high PASS 100% 89 in → 216 out (239 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:high__json_schema PASS 100% 89 in → 412 out (472 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:low PASS 100% 89 in → 82 out (78 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:low__json_schema PASS 100% 89 in → 97 out (77 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!\n" }
thinkingmachines/inkling-small:max PASS 100% 89 in → 558 out (548 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:max__json_schema PASS 100% 89 in → 360 out (343 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
thinkingmachines/inkling-small:medium PASS 100% 89 in → 270 out (260 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:medium__json_schema PASS 100% 89 in → 278 out (261 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
thinkingmachines/inkling-small:xhigh PASS 100% 89 in → 278 out (306 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 89 in → 271 out (253 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
thinkingmachines/inkling:high PASS 100% 89 in → 315 out (357 reasoning)
Hello, Planet!!!
thinkingmachines/inkling:high__json_schema PASS 100% 89 in → 226 out (211 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
thinkingmachines/inkling:low__json_schema PASS 100% 89 in → 119 out (104 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
thinkingmachines/inkling:max PASS 100% 89 in → 484 out (474 reasoning)
Hello, Planet!!!
thinkingmachines/inkling:medium PASS 100% 89 in → 243 out (276 reasoning)
Hello, Planet!!!
thinkingmachines/inkling:none PASS 100% 87 in → 8 out (0 reasoning)
Hello, Planet!!!
thinkingmachines/inkling:xhigh PASS 100% 89 in → 355 out (345 reasoning)
Hello, Planet!!!
thinkingmachines/inkling:xhigh__json_schema PASS 100% 89 in → 505 out (487 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.3:high PASS 100% 259 in → 1,531 out (1,527 reasoning)
Hello, Planet!!!
x-ai/grok-4.3:high__json_schema PASS 100% 316 in → 2,143 out (2,134 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
x-ai/grok-4.3:low PASS 100% 265 in → 511 out (507 reasoning)
Hello, Planet!!!
x-ai/grok-4.3:low__json_schema PASS 100% 322 in → 952 out (939 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.3:max__json_schema PASS 100% 316 in → 1,985 out (1,972 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.3:medium PASS 100% 265 in → 746 out (742 reasoning)
Hello, Planet!!!
x-ai/grok-4.3:medium__json_schema PASS 100% 322 in → 1,904 out (1,891 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.3:xhigh PASS 100% 259 in → 1,139 out (1,135 reasoning)
Hello, Planet!!!
x-ai/grok-4.3:xhigh__json_schema PASS 100% 316 in → 1,006 out (993 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.5:high PASS 100% 282 in → 215 out (211 reasoning)
Hello, Planet!!!
x-ai/grok-4.5:high__json_schema PASS 100% 351 in → 491 out (476 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.5:low PASS 100% 282 in → 191 out (187 reasoning)
Hello, Planet!!!
x-ai/grok-4.5:low__json_schema PASS 100% 351 in → 358 out (343 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.5:max PASS 100% 282 in → 233 out (229 reasoning)
Hello, Planet!!!
x-ai/grok-4.5:max__json_schema PASS 100% 351 in → 683 out (672 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
x-ai/grok-4.5:medium PASS 100% 282 in → 454 out (450 reasoning)
Hello, Planet!!!
x-ai/grok-4.5:medium__json_schema PASS 100% 351 in → 354 out (343 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
x-ai/grok-4.5:xhigh PASS 100% 282 in → 247 out (243 reasoning)
Hello, Planet!!!
x-ai/grok-4.5:xhigh__json_schema PASS 100% 351 in → 624 out (609 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
z-ai/glm-4.7-flash:high PASS 100% 79 in → 1,444 out (1,526 reasoning)
Hello, Planet!!!
z-ai/glm-4.7-flash:low PASS 100% 79 in → 3,042 out (3,277 reasoning)
Hello, Planet!!!
z-ai/glm-4.7-flash:xhigh PASS 100% 79 in → 2,569 out (2,769 reasoning)
Hello, Planet!!!
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 79 in → 1,732 out (1,865 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
z-ai/glm-5.2:high PASS 100% 86 in → 608 out (602 reasoning)
Hello, Planet!!!
z-ai/glm-5.2:high__json_schema PASS 100% 86 in → 669 out (680 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
z-ai/glm-5.2:low PASS 100% 86 in → 1,106 out (1,118 reasoning)
Hello, Planet!!!
z-ai/glm-5.2:none PASS 100% 80 in → 4 out (0 reasoning)
Hello, Planet!!!
z-ai/glm-5.2:none__json_schema PASS 100% 80 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
z-ai/glm-5.2:xhigh PASS 100% 86 in → 1,711 out (1,705 reasoning)
Hello, Planet!!!
z-ai/glm-5.2:xhigh__json_schema PASS 100% 86 in → 760 out (745 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
anthropic/claude-haiku-4.5:none FAIL 0% 98 in → 7 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 85 in → 553 out (641 reasoning)
Hello, World!!! Planet!
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-mini:high__json_schema FAIL 0% 85 in → 555 out (631 reasoning)
Hello, World!!! Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!! Planet!" }
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 85 in → 590 out (669 reasoning)
Hello, World!!! Planet!
JSON Schema Structured Output
(raw) { "resolved_content":"Hello, World!!! Planet!" }
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-mini:medium FAIL 0% 85 in → 731 out (837 reasoning)
Hello, World!!! Planet!
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-mini:xhigh FAIL 0% 85 in → 515 out (592 reasoning)
Hello, World!!! Planet!
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 84 in → 5 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 84 in → 15 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash:high__json_schema FAIL 0% 86 in → 260 out (244 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash:low FAIL 0% 84 in → 502 out (542 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash:medium__json_schema FAIL 0%
Invalid JSON. Output: Hello, World!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:none FAIL 0% 84 in → 5 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 84 in → 15 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Hello, Planet!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 86 in → 527 out (493 reasoning)
<<<<<<< HEAD Hello, Planet! ======= Hello, World!!! >>>>>>> feature
JSON Schema Structured Output
(raw) { "resolved_content": "<<<<<<< HEAD\nHello, Planet!\n=======\nHello, World!!!\n>>>>>>> feature" }
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 86 in → 549 out (535 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
google/gemini-3.5-flash-lite:low FAIL 0% 82 in → 6 out (0 reasoning)
Hello, Planet! World!!!
Failure: Content mismatch: ['greeting.txt']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 142 in → 15 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
google/gemma-4-26b-a4b-it:none__json_schema FAIL 0% 95 in → 15 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
google/gemma-4-31b-it:low FAIL 0% 97 in → 1,635 out (1,625 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
google/gemma-4-31b-it:low__json_schema FAIL 0% 101 in → 1,043 out (1,031 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']
google/gemma-4-31b-it:medium FAIL 0% 98 in → 1,470 out (1,398 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
google/gemma-4-31b-it:none FAIL 0% 95 in → 5 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
ibm-granite/granite-4.1-8b FAIL 0% 90 in → 5 out
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
ibm-granite/granite-4.1-8b__json_schema FAIL 0% 90 in → 12 out
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']
inclusionai/ling-2.6-flash FAIL 0% 102 in → 5 out
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
inclusionai/ling-2.6-flash__json_schema FAIL 0% 102 in → 15 out
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
liquid/lfm-2-24b-a2b FAIL 0% 96 in → 5 out
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: Hello, World!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:low FAIL 0% 115 in → 425 out (419 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.5:low__json_schema FAIL 0% 115 in → 530 out (575 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.5:xhigh FAIL 0% 115 in → 422 out (474 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.7:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high__json_schema FAIL 0% 201 in → 18 out (17 reasoning)
Hello, Planet! Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet! Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m3:low__json_schema FAIL 0% 240 in → 14 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m3:medium FAIL 0% 253 in → 221 out (254 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m3:medium__json_schema FAIL 0% 240 in → 11 out (0 reasoning)
Hello, World!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, World!"}
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m3:xhigh__json_schema FAIL 0% 240 in → 12 out (0 reasoning)
Hello, World!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!" }
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.6:high FAIL 0% 84 in → 260 out (250 reasoning)
Hello, Planet!!!
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.6:xhigh FAIL 0% 84 in → 1,137 out (1,375 reasoning)
Hello, Planet!!!
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Hello, World!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
moonshotai/kimi-k2.7-code:high FAIL 0% 84 in → 1,461 out (1,455 reasoning)
Hello, Planet!!!
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.7-code:low FAIL 0% 84 in → 940 out (1,112 reasoning)
Hello, Planet!!!
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.7-code:medium FAIL 0% 83 in → 323 out (317 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.7-code:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:high FAIL 0% 98 in → 1,029 out (1,144 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 0% 98 in → 1,014 out (1,110 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:low FAIL 0% 98 in → 739 out (816 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 98 in → 802 out (902 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:medium FAIL 0% 98 in → 573 out (609 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema FAIL 0% 98 in → 21,864 out (24,206 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 98 in → 5 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 98 in → 15 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 98 in → 828 out (902 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 0% 98 in → 705 out (815 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-super-120b-a12b:medium__json_schema FAIL 0% 98 in → 216 out (198 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-super-120b-a12b:none FAIL 0% 98 in → 5 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-mini:none FAIL 0% 80 in → 14 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-mini:none__json_schema FAIL 0% 116 in → 24 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!\n"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:high FAIL 0% 80 in → 8 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:high__json_schema FAIL 0% 116 in → 19 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World!!!\n"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:low FAIL 0% 80 in → 8 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:low__json_schema FAIL 0% 116 in → 19 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World!!!\n"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:medium FAIL 0% 80 in → 8 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:medium__json_schema FAIL 0% 116 in → 17 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:none FAIL 0% 80 in → 8 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:none__json_schema FAIL 0% 116 in → 19 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World!!!\n"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.6-luna:none FAIL 0% 80 in → 8 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.6-luna:none__json_schema FAIL 0% 116 in → 17 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-oss-120b:high__json_schema FAIL 0% 141 in → 18 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!\n" }
Failure: Content mismatch: ['greeting.txt']
openai/gpt-oss-120b:low FAIL 0% 141 in → 314 out (310 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-oss-120b:low__json_schema FAIL 0% 141 in → 62 out (38 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
openai/gpt-oss-120b:medium FAIL 0% 143 in → 378 out (412 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-oss-20b:low FAIL 0% 143 in → 119 out (116 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
poolside/laguna-xs-2.1:none FAIL 0% 93 in → 4 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 93 in → 11 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
poolside/laguna-xs.2:low__json_schema FAIL 0% 130 in → 427 out (415 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']
poolside/laguna-xs.2:none FAIL 0% 130 in → 6 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
poolside/laguna-xs.2:none__json_schema FAIL 0% 130 in → 11 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']
poolside/laguna-xs.2:xhigh__json_schema FAIL 0% 130 in → 1,147 out (1,135 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-27b:high FAIL 0% 88 in → 1,928 out (1,988 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-27b:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:medium__json_schema FAIL 0% 90 in → 13 out (0 reasoning)
Hello, World! and Planet!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World! and Planet!"}
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 90 in → 16 out (0 reasoning)
Hello, World!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!" }
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:low__json_schema FAIL 0% 88 in → 3,014 out (2,950 reasoning)
Hello, World, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World, Planet!" }
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:medium FAIL 0% 88 in → 2,160 out (2,220 reasoning)
Hello, World!!! Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:medium__json_schema FAIL 0% 88 in → 2,713 out (2,745 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:none FAIL 0% 90 in → 5 out (0 reasoning)
Hello, World!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 90 in → 12 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1b2a64a7-d0c5-99dc-8740-3544704a25d0","request_id":"1b2a64a7-d0c5-99dc-8740-3544704a25d0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3a8c3829-cbfe-9e4e-ad71-bf642203bdc2","request_id":"3a8c3829-cbfe-9e4e-ad71-bf642203bdc2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium FAIL 0% 88 in → 2,687 out (2,676 reasoning)
Hello, World! Planet!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6eeb7069-ada1-94bd-96bf-24475d18e200","request_id":"6eeb7069-ada1-94bd-96bf-24475d18e200"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none FAIL 0% 90 in → 9 out (0 reasoning)
Hello, World!!! Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-134738a6-2e7e-9d5e-81f1-8050e801df70","request_id":"134738a6-2e7e-9d5e-81f1-8050e801df70"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2238702c-daae-9fb8-8634-23a5f3e8fde9","request_id":"2238702c-daae-9fb8-8634-23a5f3e8fde9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-45672f00-bb69-9214-a6e1-707f103613bf"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low FAIL 0% 88 in → 2,193 out (2,184 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f8e1a9db-2689-9912-ac7c-5f96246a1d64"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2fb95d5b-011d-90d4-b54f-661a2fc56a30"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f8c01122-6ae0-97d2-918a-64f6bf11582c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none FAIL 0% 90 in → 4 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c8cfaef3-52ff-948d-b0b5-621fee0571b6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh FAIL 0% 88 in → 2,234 out (2,225 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-00aac1ab-6c4f-99a7-a02f-45beb670cd9c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9044b7d7-15f3-90a4-b6e9-ddd11efd9b5f","request_id":"9044b7d7-15f3-90a4-b6e9-ddd11efd9b5f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-774d1282-3d08-9038-a31e-4d4d8f053cd8","request_id":"774d1282-3d08-9038-a31e-4d4d8f053cd8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ea6e2cae-0ba4-92bf-a22b-f0f0cbbd9525","request_id":"ea6e2cae-0ba4-92bf-a22b-f0f0cbbd9525"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none FAIL 0% 90 in → 4 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9e6fcc43-fda6-90b5-8d8f-14dd800136b3","request_id":"9e6fcc43-fda6-90b5-8d8f-14dd800136b3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c54b2e80-46e8-9691-a2fc-25a8c09de3db","request_id":"c54b2e80-46e8-9691-a2fc-25a8c09de3db"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1b4de303-c893-9194-ab5d-1623286e0314","request_id":"1b4de303-c893-9194-ab5d-1623286e0314"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e0a31d5b-5961-9a0f-9ac1-c7e461e0ef87","request_id":"e0a31d5b-5961-9a0f-9ac1-c7e461e0ef87"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-becb5d4b-cbf7-9380-a992-9e40033b460b","request_id":"becb5d4b-cbf7-9380-a992-9e40033b460b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none FAIL 0% 90 in → 4 out (0 reasoning)
Hello, World!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6f12bccd-c8b8-98a9-b146-6519e64e97f4","request_id":"6f12bccd-c8b8-98a9-b146-6519e64e97f4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-881762bb-6570-9ff6-b27f-9cce2ec5b6ab","request_id":"881762bb-6570-9ff6-b27f-9cce2ec5b6ab"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:medium__json_schema FAIL 0% 90 in → 858 out (938 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']
tencent/hy3:none FAIL 0% 93 in → 9 out (0 reasoning)
Hello, Planet! Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
tencent/hy3:xhigh FAIL 0% 90 in → 1,847 out (1,841 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
thinkingmachines/inkling-small:none FAIL 0% 89 in → 7 out (0 reasoning)
Hello, World!!!<|end_message|>
Failure: Content mismatch: ['greeting.txt']
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: Hello, Planet!!!{ "resolved_content": "Hello, Planet!!!" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low FAIL 0% 89 in → 162 out (149 reasoning)
Hello, World and Planet!!!
Failure: Content mismatch: ['greeting.txt']
thinkingmachines/inkling:max__json_schema FAIL 0%
Invalid JSON. Output: Hello, Planet!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: Hello, Planet!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: Hello, World!!!{ "resolved_content": "Hello, World!!!" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:max FAIL 0% 259 in → 3,522 out (3,518 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
x-ai/grok-4.3:none FAIL 0% 257 in → 4 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
x-ai/grok-4.3:none__json_schema FAIL 0% 314 in → 9 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium FAIL 0% 79 in → 1,629 out (1,724 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 79 in → 4 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 79 in → 18 out (0 reasoning)
Hello, World!!! Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!\nHello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-5.2:low__json_schema FAIL 0% 88 in → 938 out (925 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-5.2:medium FAIL 0% 86 in → 342 out (336 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-5.2:medium__json_schema FAIL 0% 262 in → 600 out (589 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']