f001 — cherry_pick

← cherry_pick
Single-line greeting conflict
Tests ability to resolve a simple single-line cherry-pick conflict in a greeting file. Evaluates basic conflict marker understanding and merge strategy.
easy cherry-pick conflict-resolution single-line basic
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Hello, World!' > greeting.txt
  5. 05 git add greeting.txt
  6. 06 git commit -m 'Initial greeting'
  7. 07 git checkout -b feature
  8. 08 echo 'Hello, World!!!' > greeting.txt
  9. 09 git add greeting.txt
  10. 10 git commit -m 'Add excitement'
  11. 11 git checkout main
  12. 12 echo 'Hello, Planet!' > greeting.txt
  13. 13 git add greeting.txt
  14. 14 git commit -m 'Update greeting'
  15. 15 git cherry-pick $(git rev-parse feature)
Prompt
Resolve the cherry-pick conflict in greeting.txt. The base version was 'Hello, World!'. The incoming commit changed it to 'Hello, World!!!'. The current branch has 'Hello, Planet!'. Provide ONLY the resolved file content, nothing else.
Expected
Hello, Planet!!!
Fixture Metrics

Scoped model quality, cost, API time, and token usage for cherry_pick/f001.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 172 in → 77 out (63 reasoning)
Hello, Planet!!!
anthropic/claude-fable-5:high__json_schema PASS 100% 392 in → 81 out (63 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!\n"}
anthropic/claude-fable-5:low PASS 100% 171 in → 58 out (55 reasoning)
Hello, Planet!!!
anthropic/claude-fable-5:low__json_schema PASS 100% 393 in → 41 out (10 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!\n"}
anthropic/claude-fable-5:max PASS 100% 173 in → 239 out (80 reasoning)
Hello, Planet!!!
anthropic/claude-fable-5:max__json_schema PASS 100% 391 in → 362 out (141 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!\n"}
anthropic/claude-fable-5:medium PASS 100% 172 in → 60 out (58 reasoning)
Hello, Planet!!!
anthropic/claude-fable-5:medium__json_schema PASS 100% 390 in → 69 out (41 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-fable-5:xhigh PASS 100% 173 in → 105 out (77 reasoning)
Hello, Planet!!!
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 393 in → 93 out (72 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!\n"}
anthropic/claude-haiku-4.5:high PASS 100% 138 in → 320 out (332 reasoning)
Hello, Planet!!!
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 307 in → 317 out (298 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-haiku-4.5:low PASS 100% 139 in → 546 out (593 reasoning)
Hello, Planet!!!
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 305 in → 325 out (338 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-haiku-4.5:medium PASS 100% 137 in → 315 out (341 reasoning)
Hello, Planet!!!
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 306 in → 346 out (338 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-haiku-4.5:none PASS 100% 108 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 275 in → 14 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 137 in → 324 out (340 reasoning)
Hello, Planet!!!
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 305 in → 334 out (341 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-opus-4.6:high PASS 100% 109 in → 102 out (73 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:high__json_schema PASS 100% 277 in → 137 out (100 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
anthropic/claude-opus-4.6:low PASS 100% 107 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:low__json_schema PASS 100% 276 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.6:max PASS 100% 109 in → 151 out (122 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:max__json_schema PASS 100% 278 in → 150 out (114 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
anthropic/claude-opus-4.6:medium PASS 100% 110 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 278 in → 16 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
anthropic/claude-opus-4.6:none PASS 100% 109 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:none__json_schema PASS 100% 275 in → 15 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 108 in → 111 out (81 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 277 in → 138 out (101 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
anthropic/claude-opus-4.7:high PASS 100% 179 in → 14 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:high__json_schema PASS 100% 399 in → 25 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.7:low PASS 100% 176 in → 14 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:low__json_schema PASS 100% 397 in → 25 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.7:max PASS 100% 177 in → 262 out (78 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:max__json_schema PASS 100% 398 in → 274 out (74 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.7:medium PASS 100% 178 in → 14 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 396 in → 27 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
anthropic/claude-opus-4.7:none PASS 100% 178 in → 14 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:none__json_schema PASS 100% 397 in → 27 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 178 in → 14 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 397 in → 27 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
anthropic/claude-opus-4.8:high PASS 100% 173 in → 216 out (78 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:high__json_schema PASS 100% 393 in → 150 out (79 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
anthropic/claude-opus-4.8:low PASS 100% 174 in → 76 out (65 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:low__json_schema PASS 100% 392 in → 66 out (41 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.8:max PASS 100% 174 in → 473 out (139 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:max__json_schema PASS 100% 392 in → 338 out (104 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.8:medium PASS 100% 173 in → 253 out (76 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 394 in → 127 out (73 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 392 in → 22 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-opus-4.8:xhigh PASS 100% 172 in → 266 out (73 reasoning)
Hello, Planet!!!
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 391 in → 155 out (66 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!\n"}
anthropic/claude-sonnet-4.6:high PASS 100% 110 in → 179 out (149 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 276 in → 139 out (107 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:low PASS 100% 107 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 278 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:max PASS 100% 109 in → 266 out (186 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 275 in → 240 out (209 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:medium PASS 100% 110 in → 111 out (91 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 278 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:none PASS 100% 108 in → 7 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 278 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 108 in → 181 out (144 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 275 in → 110 out (88 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-5:high PASS 100% 173 in → 97 out (56 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-5:high__json_schema PASS 100% 392 in → 103 out (70 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-sonnet-5:low PASS 100% 170 in → 12 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-5:low__json_schema PASS 100% 392 in → 23 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-sonnet-5:medium PASS 100% 171 in → 12 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 392 in → 23 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
anthropic/claude-sonnet-5:none PASS 100% 173 in → 12 out (0 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-5:none__json_schema PASS 100% 392 in → 22 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
anthropic/claude-sonnet-5:xhigh PASS 100% 172 in → 180 out (59 reasoning)
Hello, Planet!!!
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 392 in → 164 out (59 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 93 in → 1,217 out (1,229 reasoning)
Hello, Planet!!!
arcee-ai/trinity-large-thinking:low PASS 100% 93 in → 906 out (922 reasoning)
Hello, Planet!!!
arcee-ai/trinity-large-thinking:medium PASS 100% 93 in → 766 out (767 reasoning)
Hello, Planet!!!
arcee-ai/trinity-large-thinking:xhigh PASS 100% 92 in → 782 out (790 reasoning)
Hello, Planet!!!
arcee-ai/trinity-mini:low PASS 100% 92 in → 579 out (694 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:high PASS 100% 93 in → 776 out (770 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 92 in → 710 out (776 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 94 in → 895 out (1,003 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 94 in → 549 out (586 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 173 in → 546 out (628 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 184 in → 3,355 out (3,667 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 97 in → 215 out (229 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 92 in → 432 out (492 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 94 in → 891 out (1,024 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 102 in → 616 out (600 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash:high PASS 100% 94 in → 307 out (347 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash:low PASS 100% 92 in → 498 out (492 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 93 in → 232 out (212 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 171 in → 1,281 out (1,275 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:high PASS 100% 93 in → 452 out (446 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:low PASS 100% 92 in → 178 out (173 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 92 in → 14 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
deepseek/deepseek-v4-pro:medium PASS 100% 92 in → 577 out (571 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 283 in → 246 out (232 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
deepseek/deepseek-v4-pro:none PASS 100% 93 in → 5 out (0 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 93 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 170 in → 682 out (676 reasoning)
Hello, Planet!!!
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 175 in → 956 out (940 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3-flash-preview:high PASS 100% 93 in → 325 out (321 reasoning)
Hello, Planet!!!
google/gemini-3-flash-preview:high__json_schema PASS 100% 153 in → 440 out (429 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3-flash-preview:low PASS 100% 93 in → 318 out (314 reasoning)
Hello, Planet!!!
google/gemini-3-flash-preview:low__json_schema PASS 100% 154 in → 635 out (625 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3-flash-preview:medium PASS 100% 94 in → 240 out (236 reasoning)
Hello, Planet!!!
google/gemini-3-flash-preview:medium__json_schema PASS 100% 155 in → 1,769 out (1,758 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3-flash-preview:xhigh PASS 100% 94 in → 467 out (463 reasoning)
Hello, Planet!!!
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 155 in → 619 out (609 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 95 in → 296 out (292 reasoning)
Hello, Planet!!!
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 155 in → 610 out (599 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 152 in → 120 out (105 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 96 in → 340 out (336 reasoning)
Hello, Planet!!!
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 94 in → 193 out (178 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 95 in → 246 out (242 reasoning)
Hello, Planet!!!
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 154 in → 1,040 out (1,029 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.1-pro-preview:high PASS 100% 95 in → 417 out (413 reasoning)
Hello, Planet!!!
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 153 in → 305 out (290 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3.1-pro-preview:low PASS 100% 95 in → 309 out (305 reasoning)
Hello, Planet!!!
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 155 in → 304 out (294 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.1-pro-preview:medium PASS 100% 94 in → 256 out (252 reasoning)
Hello, Planet!!!
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 154 in → 114 out (103 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 94 in → 266 out (262 reasoning)
Hello, Planet!!!
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 154 in → 120 out (110 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.5-flash-lite:high PASS 100% 94 in → 410 out (406 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 96 in → 656 out (646 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.5-flash-lite:low PASS 100% 96 in → 4 out (0 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 95 in → 11 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.5-flash-lite:max PASS 100% 94 in → 313 out (309 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 155 in → 469 out (459 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.5-flash-lite:medium PASS 100% 96 in → 342 out (338 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 94 in → 526 out (515 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 95 in → 399 out (395 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 96 in → 669 out (659 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.5-flash:high PASS 100% 93 in → 744 out (740 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash:high__json_schema PASS 100% 154 in → 386 out (376 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.5-flash:low PASS 100% 94 in → 369 out (365 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash:low__json_schema PASS 100% 155 in → 129 out (114 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3.5-flash:medium PASS 100% 93 in → 560 out (556 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash:medium__json_schema PASS 100% 153 in → 512 out (497 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemini-3.5-flash:xhigh PASS 100% 92 in → 503 out (499 reasoning)
Hello, Planet!!!
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 154 in → 958 out (947 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
google/gemini-3.6-flash:high PASS 100% 94 in → 363 out (359 reasoning)
Hello, Planet!!!
google/gemini-3.6-flash:high__json_schema PASS 100% 96 in → 1,119 out (1,109 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.6-flash:low PASS 100% 94 in → 257 out (253 reasoning)
Hello, Planet!!!
google/gemini-3.6-flash:low__json_schema PASS 100% 95 in → 10 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.6-flash:max PASS 100% 95 in → 274 out (270 reasoning)
Hello, Planet!!!
google/gemini-3.6-flash:max__json_schema PASS 100% 94 in → 621 out (611 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.6-flash:medium PASS 100% 96 in → 336 out (332 reasoning)
Hello, Planet!!!
google/gemini-3.6-flash:medium__json_schema PASS 100% 151 in → 592 out (582 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemini-3.6-flash:xhigh PASS 100% 96 in → 291 out (287 reasoning)
Hello, Planet!!!
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 96 in → 625 out (615 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
google/gemma-4-26b-a4b-it:high PASS 100% 111 in → 608 out (574 reasoning)
Hello, Planet!!!
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 107 in → 549 out (526 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-26b-a4b-it:low PASS 100% 111 in → 870 out (852 reasoning)
Hello, Planet!!!
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 109 in → 630 out (604 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-26b-a4b-it:medium PASS 100% 108 in → 811 out (782 reasoning)
Hello, Planet!!!
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 111 in → 513 out (499 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-26b-a4b-it:none PASS 100% 107 in → 5 out (0 reasoning)
Hello, Planet!!!
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 108 in → 15 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 108 in → 405 out (378 reasoning)
Hello, Planet!!!
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 111 in → 1,653 out (1,671 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-31b-it:high PASS 100% 109 in → 1,177 out (1,215 reasoning)
Hello, Planet!!!
google/gemma-4-31b-it:high__json_schema PASS 100% 108 in → 1,096 out (1 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-31b-it:low PASS 100% 110 in → 1,536 out (1,503 reasoning)
Hello, Planet!!!
google/gemma-4-31b-it:medium PASS 100% 112 in → 1,004 out (999 reasoning)
Hello, Planet!!!
google/gemma-4-31b-it:medium__json_schema PASS 100% 109 in → 563 out (558 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
google/gemma-4-31b-it:xhigh PASS 100% 109 in → 571 out (579 reasoning)
Hello, Planet!!!
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 109 in → 1,170 out (1,139 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
inclusionai/ling-2.6-flash PASS 100% 115 in → 5 out
Hello, Planet!!!
inclusionai/ling-2.6-flash__json_schema PASS 100% 115 in → 16 out
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
minimax/minimax-m2.7:high PASS 100% 126 in → 428 out (423 reasoning)
Hello, Planet!!!
minimax/minimax-m2.7:high__json_schema PASS 100% 258 in → 701 out (689 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
minimax/minimax-m2.7:low PASS 100% 125 in → 1,492 out (1,768 reasoning)
Hello, Planet!!!
minimax/minimax-m2.7:low__json_schema PASS 100% 258 in → 1,027 out (1,015 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
minimax/minimax-m2.7:medium__json_schema PASS 100% 257 in → 1,679 out (1,667 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
minimax/minimax-m2.7:xhigh PASS 100% 125 in → 667 out (661 reasoning)
Hello, Planet!!!
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 257 in → 2,413 out (2,401 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
minimax/minimax-m3:high PASS 100% 262 in → 189 out (211 reasoning)
Hello, Planet!!!
minimax/minimax-m3:medium PASS 100% 261 in → 207 out (248 reasoning)
Hello, Planet!!!
minimax/minimax-m3:xhigh PASS 100% 263 in → 280 out (344 reasoning)
Hello, Planet!!!
mistralai/devstral-2512 PASS 100% 96 in → 5 out
Hello, Planet!!!
mistralai/devstral-2512__json_schema PASS 100% 96 in → 13 out
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
mistralai/mistral-medium-3-5:high PASS 100% 108 in → 807 out (883 reasoning)
Hello, Planet!!!
mistralai/mistral-medium-3-5:low PASS 100% 109 in → 595 out (643 reasoning)
Hello, Planet!!!
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 97 in → 333 out (336 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
mistralai/mistral-medium-3-5:medium PASS 100% 107 in → 2,280 out (2,492 reasoning)
Hello, Planet!!!
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 97 in → 2,044 out (2,270 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
mistralai/mistral-medium-3-5:none PASS 100% 109 in → 5 out (0 reasoning)
Hello, Planet!!!
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 97 in → 13 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 106 in → 600 out (674 reasoning)
Hello, Planet!!!
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 96 in → 362 out (388 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
moonshotai/kimi-k2.6:low PASS 100% 91 in → 2,281 out (2,747 reasoning)
Hello, Planet!!!
moonshotai/kimi-k2.6:low__json_schema PASS 100% 91 in → 1,602 out (1,445 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
moonshotai/kimi-k2.6:medium PASS 100% 94 in → 753 out (749 reasoning)
Hello, Planet!!!
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 93 in → 744 out (856 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
moonshotai/kimi-k2.6:xhigh PASS 100% 94 in → 1,219 out (1,215 reasoning)
Hello, Planet!!!
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 276 in → 1,005 out (994 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 94 in → 740 out (874 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
moonshotai/kimi-k2.7-code:low PASS 100% 93 in → 1,229 out (1,223 reasoning)
Hello, Planet!!!
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 92 in → 415 out (401 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
moonshotai/kimi-k3:high PASS 100% 171 in → 163 out (144 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:high__json_schema PASS 100% 169 in → 237 out (218 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:low PASS 100% 171 in → 260 out (241 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:low__json_schema PASS 100% 169 in → 210 out (191 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:max PASS 100% 171 in → 188 out (169 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:max__json_schema PASS 100% 169 in → 115 out (96 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:medium PASS 100% 170 in → 186 out (167 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:medium__json_schema PASS 100% 171 in → 180 out (161 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:xhigh PASS 100% 171 in → 219 out (200 reasoning)
Hello, Planet!!!
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 171 in → 186 out (167 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 110 in → 1,137 out (1,304 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 107 in → 2,295 out (2,677 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 110 in → 204 out (194 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 108 in → 1,024 out (986 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 108 in → 154 out (168 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 109 in → 4,893 out (4,611 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 109 in → 280 out (267 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 109 in → 205 out (186 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 109 in → 5 out (0 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 108 in → 15 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 109 in → 168 out (187 reasoning)
Hello, Planet!!!
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 110 in → 124 out (105 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
openai/gpt-5.4-mini:high PASS 100% 88 in → 227 out (217 reasoning)
Hello, Planet!!!
openai/gpt-5.4-mini:high__json_schema PASS 100% 124 in → 325 out (306 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4-mini:low PASS 100% 90 in → 76 out (66 reasoning)
Hello, Planet!!!
openai/gpt-5.4-mini:low__json_schema PASS 100% 125 in → 113 out (94 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4-mini:medium PASS 100% 90 in → 179 out (169 reasoning)
Hello, Planet!!!
openai/gpt-5.4-mini:medium__json_schema PASS 100% 125 in → 136 out (117 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4-mini:none PASS 100% 89 in → 8 out (0 reasoning)
Hello, Planet!!!
openai/gpt-5.4-mini:xhigh PASS 100% 89 in → 1,226 out (1,216 reasoning)
Hello, Planet!!!
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 125 in → 4,906 out (4,887 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4-nano:medium__json_schema PASS 100% 125 in → 19 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
openai/gpt-5.4-nano:none__json_schema PASS 100% 126 in → 19 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 88 in → 428 out (418 reasoning)
Hello, Planet!!!
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 126 in → 287 out (266 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!\n"}
openai/gpt-5.4:high PASS 100% 90 in → 122 out (112 reasoning)
Hello, Planet!!!
openai/gpt-5.4:high__json_schema PASS 100% 125 in → 163 out (144 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4:low PASS 100% 88 in → 45 out (35 reasoning)
Hello, Planet!!!
openai/gpt-5.4:low__json_schema PASS 100% 125 in → 71 out (52 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4:medium PASS 100% 89 in → 66 out (56 reasoning)
Hello, Planet!!!
openai/gpt-5.4:medium__json_schema PASS 100% 124 in → 62 out (43 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4:none PASS 100% 88 in → 8 out (0 reasoning)
Hello, Planet!!!
openai/gpt-5.4:none__json_schema PASS 100% 124 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.4:xhigh PASS 100% 88 in → 92 out (82 reasoning)
Hello, Planet!!!
openai/gpt-5.4:xhigh__json_schema PASS 100% 125 in → 173 out (154 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.5:high PASS 100% 88 in → 102 out (92 reasoning)
Hello, Planet!!!
openai/gpt-5.5:high__json_schema PASS 100% 125 in → 135 out (116 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.5:low PASS 100% 90 in → 70 out (60 reasoning)
Hello, Planet!!!
openai/gpt-5.5:low__json_schema PASS 100% 124 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.5:medium PASS 100% 87 in → 91 out (81 reasoning)
Hello, Planet!!!
openai/gpt-5.5:medium__json_schema PASS 100% 124 in → 91 out (72 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.5:none PASS 100% 91 in → 8 out (0 reasoning)
Hello, Planet!!!
openai/gpt-5.5:none__json_schema PASS 100% 126 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.5:xhigh PASS 100% 88 in → 123 out (113 reasoning)
Hello, Planet!!!
openai/gpt-5.5:xhigh__json_schema PASS 100% 125 in → 163 out (144 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-luna:high PASS 100% 89 in → 264 out (254 reasoning)
Hello, Planet!!!
openai/gpt-5.6-luna:high__json_schema PASS 100% 123 in → 224 out (205 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-luna:max PASS 100% 89 in → 494 out (484 reasoning)
Hello, Planet!!!
openai/gpt-5.6-luna:max__json_schema PASS 100% 124 in → 292 out (273 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-luna:medium__json_schema PASS 100% 124 in → 141 out (122 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 125 in → 207 out (188 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:high PASS 100% 88 in → 35 out (25 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:high__json_schema PASS 100% 125 in → 76 out (57 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:low PASS 100% 91 in → 36 out (26 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:low__json_schema PASS 100% 124 in → 63 out (44 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:max PASS 100% 88 in → 46 out (36 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:max__json_schema PASS 100% 125 in → 128 out (109 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:medium PASS 100% 89 in → 37 out (27 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:medium__json_schema PASS 100% 124 in → 65 out (46 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:none PASS 100% 89 in → 8 out (0 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:none__json_schema PASS 100% 123 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-sol:xhigh PASS 100% 89 in → 35 out (25 reasoning)
Hello, Planet!!!
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 123 in → 99 out (80 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:high PASS 100% 89 in → 8 out (0 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:high__json_schema PASS 100% 125 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:low PASS 100% 89 in → 32 out (22 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:low__json_schema PASS 100% 126 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:max PASS 100% 89 in → 58 out (48 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:max__json_schema PASS 100% 126 in → 221 out (202 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:medium PASS 100% 91 in → 33 out (23 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:medium__json_schema PASS 100% 125 in → 59 out (40 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:none PASS 100% 88 in → 8 out (0 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:none__json_schema PASS 100% 125 in → 17 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-5.6-terra:xhigh PASS 100% 89 in → 56 out (46 reasoning)
Hello, Planet!!!
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 127 in → 102 out (83 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-oss-120b:high PASS 100% 153 in → 462 out (542 reasoning)
Hello, Planet!!!
openai/gpt-oss-120b:high__json_schema PASS 100% 150 in → 573 out (551 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
openai/gpt-oss-120b:low PASS 100% 146 in → 62 out (61 reasoning)
Hello, Planet!!!
openai/gpt-oss-120b:low__json_schema PASS 100% 139 in → 88 out (78 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-oss-120b:medium PASS 100% 150 in → 174 out (193 reasoning)
Hello, Planet!!!
openai/gpt-oss-120b:medium__json_schema PASS 100% 148 in → 211 out (231 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
openai/gpt-oss-120b:xhigh PASS 100% 153 in → 559 out (546 reasoning)
Hello, Planet!!!
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 151 in → 712 out (694 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
openai/gpt-oss-20b:high PASS 100% 153 in → 1,073 out (1,249 reasoning)
Hello, Planet!!!
openai/gpt-oss-20b:high__json_schema PASS 100% 146 in → 1,915 out (2,270 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
openai/gpt-oss-20b:medium PASS 100% 149 in → 287 out (279 reasoning)
Hello, Planet!!!
openai/gpt-oss-20b:xhigh PASS 100% 136 in → 786 out (905 reasoning)
Hello, Planet!!!
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 147 in → 2,405 out (2,791 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
poolside/laguna-m.1:high PASS 100% 105 in → 343 out (336 reasoning)
Hello, Planet!!!
poolside/laguna-m.1:high__json_schema PASS 100% 104 in → 248 out (231 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
poolside/laguna-m.1:low PASS 100% 105 in → 457 out (450 reasoning)
Hello, Planet!!!
poolside/laguna-m.1:low__json_schema PASS 100% 105 in → 412 out (400 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-m.1:medium PASS 100% 104 in → 458 out (451 reasoning)
Hello, Planet!!!
poolside/laguna-m.1:medium__json_schema PASS 100% 101 in → 648 out (636 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-m.1:none PASS 100% 103 in → 6 out (0 reasoning)
Hello, Planet!!!
poolside/laguna-m.1:none__json_schema PASS 100% 104 in → 11 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-m.1:xhigh PASS 100% 102 in → 470 out (463 reasoning)
Hello, Planet!!!
poolside/laguna-m.1:xhigh__json_schema PASS 100% 104 in → 419 out (405 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-xs-2.1:high PASS 100% 105 in → 819 out (814 reasoning)
Hello, Planet!!!
poolside/laguna-xs-2.1:high__json_schema PASS 100% 103 in → 417 out (405 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-xs-2.1:low PASS 100% 104 in → 280 out (275 reasoning)
Hello, Planet!!!
poolside/laguna-xs-2.1:low__json_schema PASS 100% 103 in → 998 out (982 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
poolside/laguna-xs-2.1:medium PASS 100% 104 in → 572 out (568 reasoning)
Hello, Planet!!!
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 105 in → 412 out (395 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
poolside/laguna-xs-2.1:xhigh PASS 100% 104 in → 600 out (597 reasoning)
Hello, Planet!!!
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 103 in → 320 out (304 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
poolside/laguna-xs.2:high PASS 100% 140 in → 383 out (376 reasoning)
Hello, Planet!!!
poolside/laguna-xs.2:high__json_schema PASS 100% 141 in → 454 out (442 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-xs.2:low PASS 100% 140 in → 291 out (284 reasoning)
Hello, Planet!!!
poolside/laguna-xs.2:low__json_schema PASS 100% 141 in → 271 out (259 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-xs.2:medium__json_schema PASS 100% 142 in → 462 out (450 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-xs.2:none PASS 100% 141 in → 6 out (0 reasoning)
Hello, Planet!!!
poolside/laguna-xs.2:none__json_schema PASS 100% 142 in → 11 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 140 in → 400 out (389 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
qwen/qwen3.6-27b:high PASS 100% 99 in → 1,442 out (1,550 reasoning)
Hello, Planet!!!
qwen/qwen3.6-27b:high__json_schema PASS 100% 101 in → 1,258 out (1,312 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-27b:low PASS 100% 100 in → 931 out (986 reasoning)
Hello, Planet!!!
qwen/qwen3.6-27b:low__json_schema PASS 100% 102 in → 12 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
qwen/qwen3.6-27b:medium PASS 100% 99 in → 1,161 out (1,216 reasoning)
Hello, Planet!!!
qwen/qwen3.6-27b:medium__json_schema PASS 100% 101 in → 5,845 out (5,977 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-27b:none PASS 100% 100 in → 5 out (0 reasoning)
Hello, Planet!!!
qwen/qwen3.6-27b:none__json_schema PASS 100% 101 in → 16 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-27b:xhigh PASS 100% 101 in → 1,410 out (1,529 reasoning)
Hello, Planet!!!
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 101 in → 1,195 out (1,254 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-35b-a3b:low PASS 100% 101 in → 3,572 out (3,563 reasoning)
Hello, Planet!!!
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 100 in → 1,464 out (1,544 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 101 in → 1,309 out (1,375 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 99 in → 1,995 out (2,040 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
qwen/qwen3.6-flash:high PASS 100% 101 in → 1,059 out (1,050 reasoning)
Hello, Planet!!!
qwen/qwen3.6-flash:low PASS 100% 100 in → 2,412 out (2,403 reasoning)
Hello, Planet!!!
qwen/qwen3.6-flash:medium PASS 100% 101 in → 2,032 out (2,023 reasoning)
Hello, Planet!!!
qwen/qwen3.6-flash:none PASS 100% 103 in → 4 out (0 reasoning)
Hello, Planet!!!
qwen/qwen3.6-flash:xhigh PASS 100% 101 in → 895 out (886 reasoning)
Hello, Planet!!!
qwen/qwen3.7-flash:high PASS 100% 101 in → 2,213 out (2,204 reasoning)
Hello, Planet!!!
qwen/qwen3.7-flash:max PASS 100% 99 in → 1,277 out (1,268 reasoning)
Hello, Planet!!!
qwen/qwen3.7-flash:medium PASS 100% 98 in → 1,111 out (1,102 reasoning)
Hello, Planet!!!
qwen/qwen3.7-flash:xhigh PASS 100% 99 in → 1,193 out (1,184 reasoning)
Hello, Planet!!!
qwen/qwen3.7-max:high PASS 100% 99 in → 563 out (554 reasoning)
Hello, Planet!!!
qwen/qwen3.7-max:low PASS 100% 101 in → 686 out (677 reasoning)
Hello, Planet!!!
qwen/qwen3.7-max:medium PASS 100% 100 in → 431 out (422 reasoning)
Hello, Planet!!!
qwen/qwen3.7-max:xhigh PASS 100% 101 in → 644 out (635 reasoning)
Hello, Planet!!!
qwen/qwen3.7-plus:high PASS 100% 101 in → 559 out (550 reasoning)
Hello, Planet!!!
qwen/qwen3.7-plus:low PASS 100% 99 in → 191 out (182 reasoning)
Hello, Planet!!!
qwen/qwen3.7-plus:medium PASS 100% 101 in → 527 out (518 reasoning)
Hello, Planet!!!
qwen/qwen3.7-plus:xhigh PASS 100% 101 in → 420 out (411 reasoning)
Hello, Planet!!!
tencent/hy3:high PASS 100% 98 in → 3,574 out (3,568 reasoning)
Hello, Planet!!!
tencent/hy3:low PASS 100% 97 in → 1,269 out (1,263 reasoning)
Hello, Planet!!!
tencent/hy3:low__json_schema PASS 100% 99 in → 2,046 out (2,143 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
tencent/hy3:medium__json_schema PASS 100% 98 in → 1,609 out (1,702 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
tencent/hy3:xhigh PASS 100% 98 in → 1,603 out (1,597 reasoning)
Hello, Planet!!!
tencent/hy3:xhigh__json_schema PASS 100% 98 in → 4,434 out (4,850 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
thinkingmachines/inkling-small:high PASS 100% 100 in → 237 out (227 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:high__json_schema PASS 100% 99 in → 204 out (186 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
thinkingmachines/inkling-small:low PASS 100% 97 in → 140 out (153 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:low__json_schema PASS 100% 97 in → 102 out (86 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
thinkingmachines/inkling-small:max PASS 100% 96 in → 211 out (201 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:max__json_schema PASS 100% 98 in → 309 out (292 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
thinkingmachines/inkling-small:medium PASS 100% 99 in → 135 out (125 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:medium__json_schema PASS 100% 97 in → 170 out (152 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
thinkingmachines/inkling-small:none PASS 100% 96 in → 7 out (0 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:xhigh PASS 100% 98 in → 269 out (303 reasoning)
Hello, Planet!!!
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 98 in → 347 out (329 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
thinkingmachines/inkling:high PASS 100% 97 in → 420 out (410 reasoning)
Hello, Planet!!!
thinkingmachines/inkling:low PASS 100% 97 in → 95 out (84 reasoning)
Hello, Planet!!!
thinkingmachines/inkling:max PASS 100% 97 in → 533 out (522 reasoning)
Hello, Planet!!!
thinkingmachines/inkling:medium PASS 100% 97 in → 180 out (170 reasoning)
Hello, Planet!!!
thinkingmachines/inkling:medium__json_schema PASS 100% 99 in → 217 out (201 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
thinkingmachines/inkling:xhigh PASS 100% 99 in → 378 out (367 reasoning)
Hello, Planet!!!
thinkingmachines/inkling:xhigh__json_schema PASS 100% 99 in → 349 out (331 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.3:high PASS 100% 266 in → 790 out (786 reasoning)
Hello, Planet!!!
x-ai/grok-4.3:high__json_schema PASS 100% 323 in → 1,859 out (1,846 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.3:low PASS 100% 273 in → 397 out (393 reasoning)
Hello, Planet!!!
x-ai/grok-4.3:low__json_schema PASS 100% 330 in → 552 out (539 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.3:max PASS 100% 269 in → 985 out (981 reasoning)
Hello, Planet!!!
x-ai/grok-4.3:max__json_schema PASS 100% 325 in → 994 out (981 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.3:medium PASS 100% 274 in → 553 out (549 reasoning)
Hello, Planet!!!
x-ai/grok-4.3:medium__json_schema PASS 100% 328 in → 1,099 out (1,086 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.3:none PASS 100% 266 in → 4 out (0 reasoning)
Hello, Planet!!!
x-ai/grok-4.3:none__json_schema PASS 100% 321 in → 9 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
x-ai/grok-4.3:xhigh PASS 100% 266 in → 1,499 out (1,495 reasoning)
Hello, Planet!!!
x-ai/grok-4.3:xhigh__json_schema PASS 100% 323 in → 849 out (840 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
x-ai/grok-4.5:high PASS 100% 290 in → 250 out (246 reasoning)
Hello, Planet!!!
x-ai/grok-4.5:high__json_schema PASS 100% 360 in → 450 out (435 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.5:low PASS 100% 292 in → 237 out (233 reasoning)
Hello, Planet!!!
x-ai/grok-4.5:low__json_schema PASS 100% 359 in → 295 out (280 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.5:max PASS 100% 290 in → 315 out (311 reasoning)
Hello, Planet!!!
x-ai/grok-4.5:max__json_schema PASS 100% 359 in → 480 out (465 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.5:medium PASS 100% 291 in → 575 out (571 reasoning)
Hello, Planet!!!
x-ai/grok-4.5:medium__json_schema PASS 100% 360 in → 599 out (584 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
x-ai/grok-4.5:xhigh PASS 100% 292 in → 372 out (368 reasoning)
Hello, Planet!!!
x-ai/grok-4.5:xhigh__json_schema PASS 100% 360 in → 583 out (568 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
z-ai/glm-4.7-flash:low__json_schema PASS 100% 88 in → 2,745 out (2,876 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
z-ai/glm-4.7-flash:medium PASS 100% 87 in → 979 out (1,026 reasoning)
Hello, Planet!!!
z-ai/glm-5.2:high PASS 100% 95 in → 315 out (308 reasoning)
Hello, Planet!!!
z-ai/glm-5.2:high__json_schema PASS 100% 89 in → 175 out (156 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
z-ai/glm-5.2:low PASS 100% 95 in → 263 out (278 reasoning)
Hello, Planet!!!
z-ai/glm-5.2:low__json_schema PASS 100% 95 in → 317 out (300 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
z-ai/glm-5.2:medium PASS 100% 96 in → 210 out (228 reasoning)
Hello, Planet!!!
z-ai/glm-5.2:medium__json_schema PASS 100% 272 in → 189 out (177 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!!!"}
z-ai/glm-5.2:none PASS 100% 88 in → 4 out (0 reasoning)
Hello, Planet!!!
z-ai/glm-5.2:none__json_schema PASS 100% 89 in → 12 out (0 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!!"}
z-ai/glm-5.2:xhigh PASS 100% 95 in → 608 out (0 reasoning)
Hello, Planet!!!
z-ai/glm-5.2:xhigh__json_schema PASS 100% 88 in → 196 out (184 reasoning)
Hello, Planet!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!!!" }
anthropic/claude-opus-4.8:none FAIL 0% 172 in → 258 out (0 reasoning)
I need to resolve this conflict, but there's genuine ambiguity here that I should flag rather than silently guess. The two changes affect different aspects: - **Current branch (HEAD)** changed the noun: `World` → `Planet` - **Incoming commit** changed the punctuation/excitement: `!` → `!!!` Since these edits are on different parts of the same line, a reasonable merge that preserves *both* intents would be: ``` Hello, Planet!!! ``` However, I can't be certain this is what you want—you may prefer to keep only one side (e.g., discard the "Planet" rename, or discard the added excitement). If you want strictly one side, tell me which, and I'll give you exactly `Hello, Planet!` or `Hello, World!!!` instead.
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 92 in → 1,382 out (1,559 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-mini:high__json_schema FAIL 0% 92 in → 681 out (798 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 94 in → 442 out (511 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-mini:medium FAIL 0% 92 in → 471 out (555 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 93 in → 393 out (456 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-mini:xhigh FAIL 0% 92 in → 827 out (957 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 94 in → 457 out (527 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 93 in → 5 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 94 in → 16 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!\n" }
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash:high__json_schema FAIL 0% 95 in → 485 out (469 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash:medium FAIL 0% 92 in → 561 out (555 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash:medium__json_schema FAIL 0%
Invalid JSON. Output: Hello, Planet!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:none FAIL 0% 92 in → 5 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 92 in → 14 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Hello, Planet!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:high__json_schema FAIL 0%
Invalid JSON. Output: Hello, World!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
google/gemini-3.1-flash-lite-preview:low FAIL 0% 93 in → 124 out (120 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
google/gemma-4-31b-it:low__json_schema FAIL 0% 114 in → 1,286 out (1,274 reasoning)
}Hello, Planet!!!
JSON Schema Structured Output
(raw) {"resolved_content":"}Hello, Planet!!!"}
Failure: Content mismatch: ['greeting.txt']
google/gemma-4-31b-it:none FAIL 0% 108 in → 5 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
google/gemma-4-31b-it:none__json_schema FAIL 0% 108 in → 12 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
ibm-granite/granite-4.1-8b FAIL 0% 99 in → 5 out
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
ibm-granite/granite-4.1-8b__json_schema FAIL 0% 99 in → 16 out
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
liquid/lfm-2-24b-a2b FAIL 0% 105 in → 9 out
Hello, Planet! Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: Hello, Planet! Hello, World!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:high FAIL 0% 122 in → 420 out (414 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.5:high__json_schema FAIL 0% 133 in → 458 out (446 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.5:low FAIL 0% 124 in → 398 out (469 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.5:low__json_schema FAIL 0% 123 in → 311 out (360 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.5:medium FAIL 0% 127 in → 400 out (401 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.5:medium__json_schema FAIL 0% 123 in → 462 out (533 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.5:xhigh FAIL 0% 125 in → 561 out (718 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.5:xhigh__json_schema FAIL 0% 124 in → 399 out (384 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m2.7:medium FAIL 0% 123 in → 437 out (430 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m3:high__json_schema FAIL 0% 249 in → 12 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m3:low FAIL 0% 262 in → 151 out (166 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m3:low__json_schema FAIL 0% 250 in → 17 out (0 reasoning)
Hello, Planet! + Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet! + Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m3:medium__json_schema FAIL 0% 249 in → 13 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
minimax/minimax-m3:xhigh__json_schema FAIL 0% 250 in → 14 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
mistralai/mistral-medium-3-5:high__json_schema FAIL 0% 97 in → 688 out (770 reasoning)
Hello, Planet!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, Planet!!"}
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.6:high FAIL 0% 93 in → 672 out (666 reasoning)
Hello, Planet!!!
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.6:high__json_schema FAIL 0% 94 in → 14 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.7-code:high FAIL 0% 94 in → 472 out (466 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.7-code:low__json_schema FAIL 0% 93 in → 523 out (582 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.7-code:medium FAIL 0% 93 in → 986 out (980 reasoning)
Hello, Planet!!!
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.7-code:xhigh FAIL 0% 93 in → 639 out (633 reasoning)
Hello, Planet!!!
Failure: Content mismatch: ['greeting.txt']
moonshotai/kimi-k2.7-code:xhigh__json_schema FAIL 0% 92 in → 908 out (1,071 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 0% 108 in → 656 out (765 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:low FAIL 0% 110 in → 473 out (554 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 110 in → 7,129 out (8,166 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:medium FAIL 0% 109 in → 973 out (1,126 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 108 in → 5 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 109 in → 14 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, Planet!" }
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 110 in → 290 out (335 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 0% 108 in → 465 out (542 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-mini:none__json_schema FAIL 0% 124 in → 18 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!\n"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:high FAIL 0% 90 in → 8 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:high__json_schema FAIL 0% 125 in → 18 out (0 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!\n"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:low FAIL 0% 89 in → 8 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:low__json_schema FAIL 0% 125 in → 19 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World!!!\n"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:medium FAIL 0% 89 in → 8 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.4-nano:none FAIL 0% 90 in → 8 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.6-luna:low FAIL 0% 89 in → 8 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.6-luna:low__json_schema FAIL 0% 124 in → 106 out (87 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.6-luna:medium FAIL 0% 88 in → 8 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.6-luna:none FAIL 0% 87 in → 8 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.6-luna:none__json_schema FAIL 0% 124 in → 17 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-5.6-luna:xhigh FAIL 0% 89 in → 388 out (378 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-oss-20b:low FAIL 0% 147 in → 84 out (78 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
openai/gpt-oss-20b:low__json_schema FAIL 0% 150 in → 110 out (92 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
openai/gpt-oss-20b:medium__json_schema FAIL 0% 150 in → 289 out (276 reasoning)
Hello, Planet!
JSON Schema Structured Output
(raw) {"resolved_content":"Hello, Planet!"}
Failure: Content mismatch: ['greeting.txt']
poolside/laguna-xs-2.1:none FAIL 0% 105 in → 4 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 104 in → 18 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!\n" }
Failure: Content mismatch: ['greeting.txt']
poolside/laguna-xs.2:medium FAIL 0% 142 in → 406 out (399 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
poolside/laguna-xs.2:xhigh FAIL 0% 138 in → 317 out (311 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:high FAIL 0% 101 in → 2,252 out (2,395 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:high__json_schema FAIL 0% 100 in → 1,251 out (1,325 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:medium FAIL 0% 100 in → 2,619 out (2,786 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:none FAIL 0% 101 in → 4 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 103 in → 12 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) {"resolved_content": "Hello, World!!!"}
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-35b-a3b:xhigh FAIL 0% 101 in → 1,653 out (1,644 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c1826e71-0981-9990-a141-1c60535e7671","request_id":"c1826e71-0981-9990-a141-1c60535e7671"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-42a123eb-7951-94ec-88ef-ece2eca2fa89","request_id":"42a123eb-7951-94ec-88ef-ece2eca2fa89"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-803d3930-7205-9f35-84e6-46350ae29f70","request_id":"803d3930-7205-9f35-84e6-46350ae29f70"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4da9ccf8-a99d-99e1-9385-1a8255f617fb","request_id":"4da9ccf8-a99d-99e1-9385-1a8255f617fb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-63401672-8c25-9f00-94e9-27dd5fd0fa3f","request_id":"63401672-8c25-9f00-94e9-27dd5fd0fa3f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-cadebe20-c7c9-9cb1-b584-33b11e0f5084"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low FAIL 0% 101 in → 2,063 out (2,054 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-92621838-710e-9361-b91a-a0b9c5e73c18"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-27b7ba79-dcd4-9a0d-b9a0-04fbd99fe89d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-88dde569-add9-9c7a-b52d-d59eef827421"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none FAIL 0% 99 in → 4 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.7-flash:none__json_schema FAIL 0% 102 in → 4 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1f15950f-3018-90b8-9d5b-8cbe3fd94e82"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c6765591-a18e-9290-bd22-f219bbdabd35","request_id":"c6765591-a18e-9290-bd22-f219bbdabd35"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9a033813-68d5-9ff6-bbbc-f40d1725f478","request_id":"9a033813-68d5-9ff6-bbbc-f40d1725f478"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0a858048-337c-931b-b2c9-a5b6d6486df9","request_id":"0a858048-337c-931b-b2c9-a5b6d6486df9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none FAIL 0% 102 in → 4 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2afe6874-cff0-915d-9b5e-1f5c2899e1ff","request_id":"2afe6874-cff0-915d-9b5e-1f5c2899e1ff"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6081ad1f-a88a-9f4a-b64e-d20454d09c95","request_id":"6081ad1f-a88a-9f4a-b64e-d20454d09c95"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b145de20-96ac-9a7a-b20c-4104f2ee8dd7","request_id":"b145de20-96ac-9a7a-b20c-4104f2ee8dd7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f232a353-2b3d-989e-a7a4-fd9c7c130657","request_id":"f232a353-2b3d-989e-a7a4-fd9c7c130657"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a0211470-add7-95b8-93ea-521e8e610710","request_id":"a0211470-add7-95b8-93ea-521e8e610710"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none FAIL 0% 100 in → 4 out (0 reasoning)
Hello, Planet!
Failure: Content mismatch: ['greeting.txt']
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a52eb28e-8e7a-9a2a-a7db-4002fd43f224","request_id":"a52eb28e-8e7a-9a2a-a7db-4002fd43f224"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b39665b6-79ed-9fd4-a7dc-bb1b020a07bb","request_id":"b39665b6-79ed-9fd4-a7dc-bb1b020a07bb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: OpenAI-compatible response contained provider error: Provider returned error
tencent/hy3:medium FAIL 0% 99 in → 1,272 out (1,272 reasoning)
(empty output)
Failure: Content mismatch: ['greeting.txt']
tencent/hy3:none FAIL 0% 101 in → 5 out (0 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
tencent/hy3:none__json_schema FAIL 0% 101 in → 13 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: Hello, World!!!{ "resolved_content": "Hello, World!!!" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: Hello, Planet!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: Hello, Planet!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:max__json_schema FAIL 0%
Invalid JSON. Output: Hello, Planet!!!
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 98 in → 7 out (0 reasoning)
Hello, Planet!!!<|end_message|>
Failure: Content mismatch: ['greeting.txt']
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: Hello, World!!!{ "resolved_content": "Hello, World!!!" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:high FAIL 0% 90 in → 1,631 out (1,839 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-4.7-flash:high__json_schema FAIL 0% 89 in → 2,054 out (2,122 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!" }
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-4.7-flash:low FAIL 0% 88 in → 417 out (440 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 89 in → 8 out (0 reasoning)
Hello, Planet! Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 89 in → 15 out (0 reasoning)
Hello, World!!!
JSON Schema Structured Output
(raw) { "resolved_content": "Hello, World!!!\n" }
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-4.7-flash:xhigh FAIL 0% 89 in → 1,623 out (1,830 reasoning)
Hello, World!!!
Failure: Content mismatch: ['greeting.txt']
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)