f003 — merge_conflicts

← merge_conflicts
Single-line status message conflict
Tests ability to resolve a status-message merge conflict. Evaluates choosing between contradictory status values.
easy merge-conflict status-message resolution
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Status: OK' > status.txt
  5. 05 git add status.txt
  6. 06 git commit -m 'Initial status'
  7. 07 git checkout -b fix
  8. 08 echo 'Status: Resolved' > status.txt
  9. 09 git add status.txt
  10. 10 git commit -m 'Fix status'
  11. 11 git checkout main
  12. 12 echo 'Status: Failed' > status.txt
  13. 13 git add status.txt
  14. 14 git commit -m 'Update status'
  15. 15 git merge fix
Prompt
Resolve the merge conflict in status.txt. The base was 'Status: OK'. Branch A changed it to 'Status: Resolved'. Branch B changed it to 'Status: Failed'. Provide ONLY the resolved file content.
Expected
Status: Resolved
Fixture Metrics

Scoped model quality, cost, API time, and token usage for merge_conflicts/f003.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 141 in → 222 out (83 reasoning)
Status: Resolved
anthropic/claude-fable-5:high__json_schema PASS 100% 361 in → 82 out (42 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
anthropic/claude-fable-5:low PASS 100% 141 in → 62 out (42 reasoning)
Status: Resolved
anthropic/claude-fable-5:low__json_schema PASS 100% 361 in → 49 out (22 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-fable-5:max__json_schema PASS 100% 361 in → 316 out (75 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
anthropic/claude-fable-5:medium PASS 100% 141 in → 92 out (65 reasoning)
Status: Resolved
anthropic/claude-fable-5:medium__json_schema PASS 100% 361 in → 65 out (42 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-fable-5:xhigh PASS 100% 141 in → 416 out (116 reasoning)
Status: Resolved
anthropic/claude-haiku-4.5:high PASS 100% 120 in → 973 out (647 reasoning)
Status: Resolved
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 288 in → 986 out (655 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-haiku-4.5:low PASS 100% 120 in → 876 out (596 reasoning)
``` Status: Resolved ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 288 in → 731 out (540 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-haiku-4.5:medium PASS 100% 120 in → 885 out (751 reasoning)
``` Status: Resolved ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 288 in → 988 out (656 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 258 in → 14 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 120 in → 613 out (570 reasoning)
``` Status: Resolved ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 288 in → 2,417 out (910 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-opus-4.6:high__json_schema PASS 100% 259 in → 298 out (251 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.6:low__json_schema PASS 100% 259 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 259 in → 203 out (183 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 259 in → 57 out (28 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.6:none__json_schema PASS 100% 259 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.6:xhigh PASS 100% 91 in → 191 out (172 reasoning)
Status: Resolved
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 259 in → 209 out (191 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.7:high PASS 100% 146 in → 11 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.7:high__json_schema PASS 100% 366 in → 24 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.7:low PASS 100% 146 in → 11 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.7:max PASS 100% 146 in → 1,105 out (160 reasoning)
Status: Resolved
anthropic/claude-opus-4.7:max__json_schema PASS 100% 366 in → 541 out (94 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.7:medium PASS 100% 146 in → 11 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 366 in → 23 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 361 in → 98 out (50 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.8:low__json_schema PASS 100% 361 in → 69 out (37 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.8:max__json_schema PASS 100% 361 in → 478 out (144 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 361 in → 106 out (45 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 361 in → 19 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 361 in → 209 out (63 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
anthropic/claude-sonnet-4.6:high PASS 100% 91 in → 482 out (269 reasoning)
Status: Resolved
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 259 in → 275 out (199 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 91 in → 26 out (4 reasoning)
Status: Resolved
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 259 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 259 in → 431 out (167 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 259 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-sonnet-4.6:none PASS 100% 91 in → 11 out (0 reasoning)
``` Status: Resolved ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 259 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 259 in → 491 out (262 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-sonnet-5:high PASS 100% 141 in → 349 out (91 reasoning)
Status: Resolved
anthropic/claude-sonnet-5:high__json_schema PASS 100% 361 in → 287 out (85 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-sonnet-5:low__json_schema PASS 100% 361 in → 20 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-sonnet-5:medium PASS 100% 141 in → 9 out (0 reasoning)
Status: Resolved
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 361 in → 20 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-sonnet-5:none PASS 100% 141 in → 9 out (0 reasoning)
Status: Resolved
anthropic/claude-sonnet-5:none__json_schema PASS 100% 361 in → 19 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-sonnet-5:xhigh PASS 100% 141 in → 595 out (125 reasoning)
Status: Resolved
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 361 in → 181 out (70 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 78 in → 1,540 out (1,564 reasoning)
Status: Resolved
arcee-ai/trinity-large-thinking:low PASS 100% 78 in → 2,511 out (2,539 reasoning)
Status: Resolved
arcee-ai/trinity-large-thinking:medium PASS 100% 78 in → 1,677 out (1,701 reasoning)
Status: Resolved
arcee-ai/trinity-large-thinking:xhigh PASS 100% 78 in → 1,846 out (1,869 reasoning)
Status: Resolved
arcee-ai/trinity-mini:high__json_schema PASS 100% 78 in → 1,680 out (1,987 reasoning)
Status: Resolved
arcee-ai/trinity-mini:low PASS 100% 78 in → 1,009 out (1,217 reasoning)
Status: Resolved
arcee-ai/trinity-mini:low__json_schema PASS 100% 78 in → 1,131 out (1,340 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
arcee-ai/trinity-mini:medium PASS 100% 78 in → 911 out (1,057 reasoning)
Status: Resolved
arcee-ai/trinity-mini:xhigh PASS 100% 78 in → 842 out (989 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:high PASS 100% 158 in → 1,747 out (1,891 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 79 in → 16 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 79 in → 897 out (997 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 79 in → 1,268 out (1,366 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 158 in → 562 out (628 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 158 in → 1,142 out (1,212 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 79 in → 1,031 out (1,112 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 79 in → 723 out (763 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 79 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 79 in → 500 out (551 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 81 in → 1,734 out (1,718 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash:high PASS 100% 79 in → 442 out (436 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 81 in → 898 out (882 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash:low PASS 100% 79 in → 450 out (491 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 271 in → 1,614 out (1,601 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
deepseek/deepseek-v4-flash:medium PASS 100% 79 in → 349 out (344 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash:none PASS 100% 79 in → 5 out (0 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 147 in → 12 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
deepseek/deepseek-v4-flash:xhigh PASS 100% 158 in → 5,355 out (5,346 reasoning)
``` Status: Resolved ```
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 176 in → 12 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
deepseek/deepseek-v4-pro:high PASS 100% 79 in → 775 out (769 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 79 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-pro:low PASS 100% 79 in → 643 out (704 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:medium PASS 100% 79 in → 613 out (678 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 81 in → 382 out (366 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-pro:none PASS 100% 79 in → 4 out (0 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:xhigh PASS 100% 79 in → 538 out (589 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 349 in → 2,992 out (2,979 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3-flash-preview:high PASS 100% 73 in → 1,127 out (1,124 reasoning)
Status: Resolved
google/gemini-3-flash-preview:high__json_schema PASS 100% 133 in → 1,024 out (1,014 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3-flash-preview:low PASS 100% 73 in → 806 out (803 reasoning)
Status: Resolved
google/gemini-3-flash-preview:low__json_schema PASS 100% 133 in → 500 out (491 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3-flash-preview:medium PASS 100% 73 in → 658 out (655 reasoning)
Status: Resolved
google/gemini-3-flash-preview:medium__json_schema PASS 100% 133 in → 694 out (684 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3-flash-preview:xhigh PASS 100% 73 in → 579 out (576 reasoning)
Status: Resolved
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 133 in → 591 out (582 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 73 in → 725 out (722 reasoning)
Status: Resolved
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 74 in → 1,029 out (1,019 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 73 in → 136 out (133 reasoning)
Status: Resolved
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 74 in → 114 out (99 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 74 in → 447 out (444 reasoning)
Status: Resolved
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 133 in → 500 out (490 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 73 in → 1,135 out (1,132 reasoning)
Status: Resolved
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 74 in → 480 out (470 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.1-pro-preview:high PASS 100% 73 in → 522 out (519 reasoning)
Status: Resolved
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 133 in → 365 out (356 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.1-pro-preview:low PASS 100% 73 in → 464 out (461 reasoning)
Status: Resolved
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 133 in → 387 out (378 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.1-pro-preview:medium PASS 100% 73 in → 542 out (539 reasoning)
Status: Resolved
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 133 in → 306 out (296 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 73 in → 677 out (674 reasoning)
Status: Resolved
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 133 in → 428 out (418 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.5-flash-lite:high PASS 100% 74 in → 1,006 out (1,003 reasoning)
Status: Resolved
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 74 in → 1,012 out (1,003 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.5-flash-lite:low PASS 100% 73 in → 3 out (0 reasoning)
Status: Resolved
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 133 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.5-flash-lite:max PASS 100% 73 in → 1,117 out (1,114 reasoning)
Status: Resolved
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 133 in → 835 out (826 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.5-flash-lite:medium PASS 100% 73 in → 737 out (734 reasoning)
Status: Resolved
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 133 in → 670 out (655 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 73 in → 974 out (971 reasoning)
Status: Resolved
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 133 in → 786 out (771 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.5-flash:high PASS 100% 73 in → 1,277 out (1,274 reasoning)
Status: Resolved
google/gemini-3.5-flash:high__json_schema PASS 100% 133 in → 640 out (630 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.5-flash:low PASS 100% 73 in → 359 out (356 reasoning)
Status: Resolved
google/gemini-3.5-flash:low__json_schema PASS 100% 133 in → 237 out (222 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.5-flash:medium PASS 100% 73 in → 594 out (591 reasoning)
Status: Resolved
google/gemini-3.5-flash:medium__json_schema PASS 100% 133 in → 392 out (383 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.5-flash:xhigh PASS 100% 73 in → 978 out (975 reasoning)
Status: Resolved
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 133 in → 560 out (551 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.6-flash:high PASS 100% 74 in → 1,120 out (1,117 reasoning)
Status: Resolved
google/gemini-3.6-flash:high__json_schema PASS 100% 74 in → 942 out (933 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.6-flash:low PASS 100% 74 in → 349 out (346 reasoning)
Status: Resolved
google/gemini-3.6-flash:low__json_schema PASS 100% 133 in → 349 out (340 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.6-flash:max PASS 100% 73 in → 1,405 out (1,402 reasoning)
Status: Resolved
google/gemini-3.6-flash:max__json_schema PASS 100% 133 in → 1,198 out (1,188 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.6-flash:medium PASS 100% 74 in → 885 out (882 reasoning)
Status: Resolved
google/gemini-3.6-flash:medium__json_schema PASS 100% 74 in → 871 out (862 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.6-flash:xhigh PASS 100% 73 in → 848 out (845 reasoning)
Status: Resolved
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 74 in → 1,483 out (1,473 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemma-4-26b-a4b-it:high PASS 100% 89 in → 1,388 out (1,382 reasoning)
Status: Resolved
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 89 in → 1,040 out (1,018 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-26b-a4b-it:low PASS 100% 89 in → 378 out (360 reasoning)
Status: Resolved
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 89 in → 653 out (633 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-26b-a4b-it:medium PASS 100% 89 in → 1,334 out (1,330 reasoning)
Status: Resolved
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 89 in → 815 out (789 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-26b-a4b-it:none PASS 100% 86 in → 4 out (0 reasoning)
Status: Resolved
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 86 in → 16 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 89 in → 691 out (653 reasoning)
Status: Resolved
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 88 in → 1,026 out (1,001 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-31b-it:high PASS 100% 89 in → 581 out (1 reasoning)
Status: Resolved
google/gemma-4-31b-it:high__json_schema PASS 100% 88 in → 398 out (391 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-31b-it:low PASS 100% 89 in → 263 out (251 reasoning)
Status: Resolved
google/gemma-4-31b-it:medium PASS 100% 89 in → 164 out (149 reasoning)
Status: Resolved
google/gemma-4-31b-it:medium__json_schema PASS 100% 92 in → 244 out (233 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemma-4-31b-it:none PASS 100% 86 in → 4 out (0 reasoning)
Status: Resolved
google/gemma-4-31b-it:none__json_schema PASS 100% 86 in → 16 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-31b-it:xhigh PASS 100% 89 in → 505 out (1 reasoning)
Status: Resolved
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 89 in → 233 out (221 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
ibm-granite/granite-4.1-8b PASS 100% 83 in → 5 out
Status: Resolved
liquid/lfm-2-24b-a2b PASS 100% 91 in → 5 out
Status: Resolved
minimax/minimax-m2.5:high PASS 100% 112 in → 396 out (391 reasoning)
Status: Resolved
minimax/minimax-m2.5:high__json_schema PASS 100% 110 in → 223 out (244 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m2.5:low__json_schema PASS 100% 110 in → 1,141 out (1,292 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m2.5:medium PASS 100% 110 in → 758 out (755 reasoning)
Status: Resolved
minimax/minimax-m2.5:medium__json_schema PASS 100% 110 in → 642 out (696 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m2.5:xhigh PASS 100% 110 in → 572 out (648 reasoning)
Status: Resolved
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 110 in → 1,791 out (2,057 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m2.7:high PASS 100% 110 in → 1,233 out (1,404 reasoning)
Status: Resolved
minimax/minimax-m2.7:low PASS 100% 112 in → 493 out (488 reasoning)
Status: Resolved
minimax/minimax-m2.7:low__json_schema PASS 100% 109 in → 905 out (1,028 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m2.7:medium PASS 100% 112 in → 616 out (611 reasoning)
Status: Resolved
minimax/minimax-m2.7:medium__json_schema PASS 100% 243 in → 1,290 out (1,279 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
minimax/minimax-m2.7:xhigh PASS 100% 110 in → 854 out (962 reasoning)
Status: Resolved
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 243 in → 1,481 out (1,470 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
minimax/minimax-m3:high PASS 100% 248 in → 255 out (308 reasoning)
Status: Resolved
minimax/minimax-m3:high__json_schema PASS 100% 235 in → 14 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m3:low PASS 100% 248 in → 424 out (486 reasoning)
Status: Resolved
minimax/minimax-m3:low__json_schema PASS 100% 235 in → 11 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
minimax/minimax-m3:medium PASS 100% 248 in → 264 out (301 reasoning)
Status: Resolved
minimax/minimax-m3:medium__json_schema PASS 100% 235 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
minimax/minimax-m3:xhigh PASS 100% 248 in → 278 out (320 reasoning)
Status: Resolved
minimax/minimax-m3:xhigh__json_schema PASS 100% 196 in → 16 out (13 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
mistralai/devstral-2512 PASS 100% 79 in → 5 out
Status: Resolved
mistralai/devstral-2512__json_schema PASS 100% 79 in → 12 out
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
mistralai/mistral-medium-3-5:high PASS 100% 91 in → 660 out (725 reasoning)
Status: Resolved
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 79 in → 3,923 out (4,486 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
mistralai/mistral-medium-3-5:low PASS 100% 91 in → 1,501 out (1,713 reasoning)
Status: Resolved
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 79 in → 1,670 out (1,900 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
mistralai/mistral-medium-3-5:medium PASS 100% 91 in → 1,428 out (1,615 reasoning)
Status: Resolved
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 79 in → 1,408 out (1,597 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 91 in → 2,367 out (2,724 reasoning)
Status: Resolved
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 79 in → 1,543 out (1,744 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
moonshotai/kimi-k2.6:high PASS 100% 77 in → 2,266 out (2,263 reasoning)
Status: Resolved
moonshotai/kimi-k2.6:high__json_schema PASS 100% 77 in → 1,567 out (1,881 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
moonshotai/kimi-k2.6:low PASS 100% 76 in → 1,938 out (2,309 reasoning)
Status: Resolved
moonshotai/kimi-k2.6:low__json_schema PASS 100% 77 in → 1,222 out (1,208 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 76 in → 1,314 out (1,299 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 77 in → 1,609 out (1,938 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
moonshotai/kimi-k2.7-code:low PASS 100% 77 in → 440 out (517 reasoning)
``` Status: Resolved ```
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 77 in → 280 out (326 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
moonshotai/kimi-k2.7-code:medium PASS 100% 77 in → 583 out (578 reasoning)
Status: Resolved
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 77 in → 293 out (279 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 76 in → 446 out (441 reasoning)
Status: Resolved
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 77 in → 194 out (180 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
moonshotai/kimi-k3:high PASS 100% 154 in → 362 out (344 reasoning)
Status: Resolved
moonshotai/kimi-k3:high__json_schema PASS 100% 154 in → 706 out (688 reasoning)
Status: Resolved
moonshotai/kimi-k3:low__json_schema PASS 100% 154 in → 601 out (583 reasoning)
Status: Resolved
moonshotai/kimi-k3:max PASS 100% 154 in → 591 out (573 reasoning)
Status: Resolved
moonshotai/kimi-k3:max__json_schema PASS 100% 154 in → 917 out (899 reasoning)
Status: Resolved
moonshotai/kimi-k3:medium PASS 100% 154 in → 923 out (905 reasoning)
Status: Resolved
moonshotai/kimi-k3:medium__json_schema PASS 100% 251 in → 2,000 out (1,977 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
moonshotai/kimi-k3:xhigh PASS 100% 154 in → 487 out (469 reasoning)
Status: Resolved
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 251 in → 450 out (427 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 92 in → 1,114 out (1,253 reasoning)
Status: Resolved
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 92 in → 2,066 out (2,340 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 92 in → 151 out (151 reasoning)
Status: Resolved
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 92 in → 1,491 out (1,642 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 92 in → 1,933 out (2,179 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 92 in → 1,122 out (1,250 reasoning)
Status: Resolved
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 92 in → 815 out (891 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 92 in → 264 out (293 reasoning)
Status: Resolved
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 92 in → 856 out (824 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 92 in → 512 out (490 reasoning)
Status: Resolved
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 92 in → 481 out (461 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 92 in → 893 out (863 reasoning)
Status: Resolved
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 92 in → 544 out (522 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 92 in → 5 out (0 reasoning)
Status: Resolved
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 92 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 92 in → 370 out (416 reasoning)
Status: Resolved
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 92 in → 368 out (351 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
openai/gpt-5.4-mini:high PASS 100% 75 in → 420 out (410 reasoning)
Status: Resolved
openai/gpt-5.4-mini:high__json_schema PASS 100% 111 in → 773 out (754 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-mini:low PASS 100% 75 in → 129 out (119 reasoning)
Status: Resolved
openai/gpt-5.4-mini:low__json_schema PASS 100% 111 in → 180 out (161 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-mini:medium PASS 100% 75 in → 1,582 out (1,572 reasoning)
Status: Resolved
openai/gpt-5.4-mini:medium__json_schema PASS 100% 111 in → 252 out (233 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-mini:none PASS 100% 75 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.4-mini:none__json_schema PASS 100% 111 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-mini:xhigh PASS 100% 75 in → 3,116 out (3,106 reasoning)
Status: Resolved
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 111 in → 4,587 out (4,568 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-nano:high PASS 100% 75 in → 109 out (99 reasoning)
Status: Resolved
openai/gpt-5.4-nano:high__json_schema PASS 100% 111 in → 178 out (159 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-nano:low PASS 100% 75 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.4-nano:low__json_schema PASS 100% 111 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-nano:medium PASS 100% 75 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.4-nano:medium__json_schema PASS 100% 111 in → 118 out (98 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
openai/gpt-5.4-nano:none PASS 100% 75 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.4-nano:none__json_schema PASS 100% 111 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-nano:xhigh PASS 100% 75 in → 932 out (922 reasoning)
Status: Resolved
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 111 in → 978 out (958 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
openai/gpt-5.4:high PASS 100% 75 in → 212 out (202 reasoning)
Status: Resolved
openai/gpt-5.4:high__json_schema PASS 100% 111 in → 404 out (385 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4:low PASS 100% 75 in → 175 out (165 reasoning)
Status: Resolved
openai/gpt-5.4:low__json_schema PASS 100% 111 in → 155 out (136 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4:medium PASS 100% 75 in → 188 out (178 reasoning)
Status: Resolved
openai/gpt-5.4:medium__json_schema PASS 100% 111 in → 233 out (214 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4:none__json_schema PASS 100% 111 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4:xhigh PASS 100% 75 in → 829 out (819 reasoning)
Status: Resolved
openai/gpt-5.4:xhigh__json_schema PASS 100% 111 in → 5,529 out (5,510 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.5:high PASS 100% 75 in → 526 out (516 reasoning)
Status: Resolved
openai/gpt-5.5:high__json_schema PASS 100% 111 in → 397 out (378 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.5:low PASS 100% 75 in → 146 out (136 reasoning)
Status: Resolved
openai/gpt-5.5:low__json_schema PASS 100% 111 in → 105 out (86 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.5:medium PASS 100% 75 in → 141 out (131 reasoning)
Status: Resolved
openai/gpt-5.5:medium__json_schema PASS 100% 111 in → 301 out (281 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
openai/gpt-5.5:none PASS 100% 75 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.5:none__json_schema PASS 100% 111 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.5:xhigh PASS 100% 75 in → 519 out (509 reasoning)
Status: Resolved
openai/gpt-5.5:xhigh__json_schema PASS 100% 111 in → 535 out (516 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:high PASS 100% 75 in → 92 out (82 reasoning)
Status: Resolved
openai/gpt-5.6-luna:high__json_schema PASS 100% 111 in → 155 out (136 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:low PASS 100% 75 in → 144 out (134 reasoning)
Status: Resolved
openai/gpt-5.6-luna:low__json_schema PASS 100% 111 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:max PASS 100% 75 in → 166 out (156 reasoning)
Status: Resolved
openai/gpt-5.6-luna:max__json_schema PASS 100% 111 in → 355 out (336 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:medium PASS 100% 75 in → 112 out (102 reasoning)
Status: Resolved
openai/gpt-5.6-luna:medium__json_schema PASS 100% 111 in → 99 out (80 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:none PASS 100% 75 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-luna:none__json_schema PASS 100% 111 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:xhigh PASS 100% 75 in → 145 out (135 reasoning)
Status: Resolved
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 111 in → 219 out (200 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-sol:high PASS 100% 75 in → 131 out (121 reasoning)
Status: Resolved
openai/gpt-5.6-sol:high__json_schema PASS 100% 111 in → 125 out (106 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-sol:low PASS 100% 75 in → 77 out (67 reasoning)
Status: Resolved
openai/gpt-5.6-sol:max PASS 100% 75 in → 115 out (105 reasoning)
Status: Resolved
openai/gpt-5.6-sol:max__json_schema PASS 100% 111 in → 200 out (181 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-sol:medium PASS 100% 75 in → 72 out (62 reasoning)
Status: Resolved
openai/gpt-5.6-sol:medium__json_schema PASS 100% 111 in → 151 out (132 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-sol:none PASS 100% 75 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-sol:none__json_schema PASS 100% 111 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-sol:xhigh PASS 100% 75 in → 201 out (191 reasoning)
Status: Resolved
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 111 in → 143 out (124 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:high PASS 100% 75 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-terra:high__json_schema PASS 100% 111 in → 96 out (77 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:low PASS 100% 75 in → 60 out (50 reasoning)
Status: Resolved
openai/gpt-5.6-terra:low__json_schema PASS 100% 111 in → 77 out (58 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:max PASS 100% 75 in → 120 out (110 reasoning)
Status: Resolved
openai/gpt-5.6-terra:max__json_schema PASS 100% 111 in → 275 out (256 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:medium PASS 100% 75 in → 56 out (46 reasoning)
Status: Resolved
openai/gpt-5.6-terra:medium__json_schema PASS 100% 111 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:none PASS 100% 75 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-terra:none__json_schema PASS 100% 111 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:xhigh PASS 100% 75 in → 113 out (103 reasoning)
Status: Resolved
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 111 in → 79 out (60 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-oss-120b:high PASS 100% 136 in → 1,649 out (2,011 reasoning)
Status: Resolved
openai/gpt-oss-120b:low PASS 100% 123 in → 68 out (61 reasoning)
Status: Resolved
openai/gpt-oss-120b:medium PASS 100% 147 in → 204 out (0 reasoning)
Status: Resolved
openai/gpt-oss-120b:medium__json_schema PASS 100% 123 in → 215 out (219 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 134 in → 1,069 out (1,258 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
openai/gpt-oss-20b:high PASS 100% 136 in → 1,392 out (1,395 reasoning)
Status: Resolved
openai/gpt-oss-20b:high__json_schema PASS 100% 134 in → 1,086 out (1,227 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-oss-20b:low PASS 100% 138 in → 73 out (71 reasoning)
Status: Resolved
openai/gpt-oss-20b:low__json_schema PASS 100% 123 in → 94 out (81 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-oss-20b:medium PASS 100% 123 in → 860 out (963 reasoning)
Status: Resolved
openai/gpt-oss-20b:medium__json_schema PASS 100% 136 in → 670 out (658 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-oss-20b:xhigh PASS 100% 138 in → 1,111 out (1,290 reasoning)
Status: Resolved
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 134 in → 980 out (1,094 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
poolside/laguna-m.1:high PASS 100% 84 in → 904 out (898 reasoning)
Status: Resolved
poolside/laguna-m.1:high__json_schema PASS 100% 84 in → 1,406 out (1,392 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-m.1:low PASS 100% 84 in → 1,204 out (1,199 reasoning)
Status: Resolved
poolside/laguna-m.1:medium PASS 100% 84 in → 1,040 out (1,034 reasoning)
Status: Resolved
poolside/laguna-m.1:medium__json_schema PASS 100% 84 in → 1,187 out (1,172 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-m.1:none PASS 100% 84 in → 5 out (0 reasoning)
Status: Resolved
poolside/laguna-m.1:none__json_schema PASS 100% 84 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-m.1:xhigh PASS 100% 84 in → 898 out (892 reasoning)
Status: Resolved
poolside/laguna-m.1:xhigh__json_schema PASS 100% 84 in → 232 out (216 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
poolside/laguna-xs-2.1:high PASS 100% 84 in → 816 out (812 reasoning)
Status: Resolved
poolside/laguna-xs-2.1:high__json_schema PASS 100% 84 in → 1,497 out (1,482 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-xs-2.1:low PASS 100% 84 in → 1,415 out (1,411 reasoning)
Status: Resolved
poolside/laguna-xs-2.1:low__json_schema PASS 100% 84 in → 1,623 out (1,613 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
poolside/laguna-xs-2.1:medium PASS 100% 84 in → 2,193 out (2,189 reasoning)
Status: Resolved
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 84 in → 1,443 out (1,430 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-xs-2.1:none__json_schema PASS 100% 84 in → 10 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
poolside/laguna-xs-2.1:xhigh PASS 100% 84 in → 3,946 out (3,945 reasoning)
Status: Resolved
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 84 in → 627 out (611 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-xs.2:high PASS 100% 121 in → 686 out (680 reasoning)
Status: Resolved
poolside/laguna-xs.2:high__json_schema PASS 100% 121 in → 357 out (346 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
poolside/laguna-xs.2:medium PASS 100% 121 in → 698 out (692 reasoning)
Status: Resolved
poolside/laguna-xs.2:medium__json_schema PASS 100% 121 in → 522 out (511 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 121 in → 586 out (576 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
qwen/qwen3.6-27b:high PASS 100% 81 in → 1,143 out (1,160 reasoning)
Status: Resolved
qwen/qwen3.6-27b:high__json_schema PASS 100% 81 in → 1,897 out (1,931 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-27b:low PASS 100% 81 in → 1,871 out (1,945 reasoning)
Status: Resolved
qwen/qwen3.6-27b:low__json_schema PASS 100% 81 in → 2,032 out (1 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-27b:medium PASS 100% 81 in → 1,730 out (1,738 reasoning)
Status: Resolved
qwen/qwen3.6-27b:medium__json_schema PASS 100% 81 in → 1,642 out (1,677 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-27b:none PASS 100% 83 in → 5 out (0 reasoning)
Status: Resolved
qwen/qwen3.6-27b:xhigh PASS 100% 81 in → 1,104 out (1,095 reasoning)
Status: Resolved
qwen/qwen3.6-35b-a3b:high PASS 100% 81 in → 1,494 out (1,530 reasoning)
Status: Resolved
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 81 in → 2,311 out (2,418 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-35b-a3b:low PASS 100% 81 in → 1,281 out (1,314 reasoning)
Status: Resolved
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 81 in → 2,204 out (2,305 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 81 in → 1,876 out (1,867 reasoning)
Status: Resolved
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 81 in → 1,639 out (1,679 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 81 in → 2,096 out (2,228 reasoning)
Status: Resolved
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 81 in → 2,222 out (2,184 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-flash:high PASS 100% 81 in → 1,551 out (1,542 reasoning)
Status: Resolved
qwen/qwen3.6-flash:low PASS 100% 81 in → 1,828 out (1,819 reasoning)
Status: Resolved
qwen/qwen3.6-flash:medium PASS 100% 81 in → 1,642 out (1,633 reasoning)
Status: Resolved
qwen/qwen3.6-flash:xhigh PASS 100% 81 in → 2,626 out (2,617 reasoning)
Status: Resolved
qwen/qwen3.7-flash:high PASS 100% 81 in → 1,681 out (1,672 reasoning)
Status: Resolved
qwen/qwen3.7-flash:low PASS 100% 81 in → 1,705 out (1,696 reasoning)
Status: Resolved
qwen/qwen3.7-flash:max PASS 100% 81 in → 1,331 out (1,322 reasoning)
Status: Resolved
qwen/qwen3.7-flash:medium PASS 100% 81 in → 1,699 out (1,690 reasoning)
Status: Resolved
qwen/qwen3.7-flash:none PASS 100% 83 in → 4 out (0 reasoning)
Status: Resolved
qwen/qwen3.7-flash:xhigh PASS 100% 81 in → 2,109 out (2,100 reasoning)
Status: Resolved
qwen/qwen3.7-max:high PASS 100% 81 in → 760 out (751 reasoning)
Status: Resolved
qwen/qwen3.7-max:low PASS 100% 81 in → 445 out (436 reasoning)
Status: Resolved
qwen/qwen3.7-max:medium PASS 100% 81 in → 369 out (360 reasoning)
Status: Resolved
qwen/qwen3.7-max:xhigh PASS 100% 81 in → 533 out (524 reasoning)
Status: Resolved
qwen/qwen3.7-plus:high PASS 100% 81 in → 777 out (768 reasoning)
Status: Resolved
qwen/qwen3.7-plus:low PASS 100% 81 in → 623 out (614 reasoning)
Status: Resolved
qwen/qwen3.7-plus:medium PASS 100% 81 in → 945 out (936 reasoning)
Status: Resolved
qwen/qwen3.7-plus:none PASS 100% 83 in → 4 out (0 reasoning)
Status: Resolved
qwen/qwen3.7-plus:xhigh PASS 100% 81 in → 723 out (714 reasoning)
Status: Resolved
tencent/hy3:high__json_schema PASS 100% 85 in → 5,244 out (5,769 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
tencent/hy3:low PASS 100% 85 in → 2,072 out (2,220 reasoning)
Status: Resolved
tencent/hy3:medium PASS 100% 85 in → 1,214 out (1,285 reasoning)
Status: Resolved
tencent/hy3:medium__json_schema PASS 100% 85 in → 1,535 out (1,550 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
tencent/hy3:none PASS 100% 88 in → 5 out (0 reasoning)
Status: Resolved
tencent/hy3:xhigh PASS 100% 85 in → 4,402 out (4,396 reasoning)
Status: Resolved
thinkingmachines/inkling-small:high PASS 100% 84 in → 466 out (456 reasoning)
Status: Resolved
thinkingmachines/inkling-small:high__json_schema PASS 100% 84 in → 847 out (965 reasoning)
Status: Resolved
thinkingmachines/inkling-small:low PASS 100% 84 in → 95 out (85 reasoning)
Status: Resolved
thinkingmachines/inkling-small:low__json_schema PASS 100% 84 in → 82 out (65 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
thinkingmachines/inkling-small:max PASS 100% 84 in → 638 out (715 reasoning)
Status: Resolved
thinkingmachines/inkling-small:max__json_schema PASS 100% 84 in → 1,538 out (1,520 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
thinkingmachines/inkling-small:medium PASS 100% 84 in → 439 out (429 reasoning)
Status: Resolved
thinkingmachines/inkling-small:medium__json_schema PASS 100% 84 in → 336 out (318 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
thinkingmachines/inkling-small:xhigh PASS 100% 84 in → 1,472 out (1,657 reasoning)
Status: Resolved
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 84 in → 754 out (737 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
thinkingmachines/inkling:high PASS 100% 84 in → 2,985 out (2,975 reasoning)
Status: Resolved
thinkingmachines/inkling:high__json_schema PASS 100% 84 in → 3,192 out (3,178 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
thinkingmachines/inkling:low PASS 100% 84 in → 121 out (129 reasoning)
Status: Resolved
thinkingmachines/inkling:low__json_schema PASS 100% 84 in → 115 out (96 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
thinkingmachines/inkling:max PASS 100% 84 in → 3,847 out (4,492 reasoning)
Status: Resolved
thinkingmachines/inkling:max__json_schema PASS 100% 84 in → 5,433 out (5,418 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
thinkingmachines/inkling:medium PASS 100% 84 in → 329 out (319 reasoning)
Status: Resolved
thinkingmachines/inkling:medium__json_schema PASS 100% 84 in → 919 out (901 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
thinkingmachines/inkling:xhigh PASS 100% 84 in → 5,527 out (5,517 reasoning)
Status: Resolved
thinkingmachines/inkling:xhigh__json_schema PASS 100% 84 in → 5,395 out (5,376 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
x-ai/grok-4.3:high PASS 100% 252 in → 1,607 out (1,604 reasoning)
Status: Resolved
x-ai/grok-4.3:high__json_schema PASS 100% 309 in → 1,669 out (1,661 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
x-ai/grok-4.3:low PASS 100% 258 in → 517 out (514 reasoning)
Status: Resolved
x-ai/grok-4.3:low__json_schema PASS 100% 315 in → 758 out (746 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
x-ai/grok-4.3:max PASS 100% 252 in → 1,142 out (1,139 reasoning)
Status: Resolved
x-ai/grok-4.3:max__json_schema PASS 100% 309 in → 1,711 out (1,703 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
x-ai/grok-4.3:medium PASS 100% 258 in → 778 out (775 reasoning)
Status: Resolved
x-ai/grok-4.3:medium__json_schema PASS 100% 315 in → 1,161 out (1,149 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
x-ai/grok-4.3:none__json_schema PASS 100% 307 in → 9 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
x-ai/grok-4.3:xhigh PASS 100% 252 in → 1,996 out (1,993 reasoning)
Status: Resolved
x-ai/grok-4.3:xhigh__json_schema PASS 100% 309 in → 1,779 out (1,767 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
x-ai/grok-4.5:high PASS 100% 274 in → 564 out (561 reasoning)
Status: Resolved
x-ai/grok-4.5:high__json_schema PASS 100% 343 in → 545 out (530 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
x-ai/grok-4.5:low PASS 100% 274 in → 226 out (223 reasoning)
Status: Resolved
x-ai/grok-4.5:low__json_schema PASS 100% 343 in → 510 out (495 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
x-ai/grok-4.5:max PASS 100% 274 in → 626 out (623 reasoning)
Status: Resolved
x-ai/grok-4.5:max__json_schema PASS 100% 343 in → 645 out (632 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
x-ai/grok-4.5:medium PASS 100% 274 in → 637 out (634 reasoning)
Status: Resolved
x-ai/grok-4.5:medium__json_schema PASS 100% 343 in → 669 out (656 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
x-ai/grok-4.5:xhigh PASS 100% 274 in → 560 out (557 reasoning)
Status: Resolved
x-ai/grok-4.5:xhigh__json_schema PASS 100% 343 in → 739 out (729 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
z-ai/glm-4.7-flash:high PASS 100% 74 in → 1,655 out (1,719 reasoning)
Status: Resolved
z-ai/glm-4.7-flash:high__json_schema PASS 100% 74 in → 1,602 out (1,596 reasoning)
Status: Resolved
z-ai/glm-4.7-flash:low PASS 100% 74 in → 785 out (811 reasoning)
Status: Resolved
z-ai/glm-4.7-flash:medium PASS 100% 74 in → 1,536 out (1,641 reasoning)
Status: Resolved
z-ai/glm-4.7-flash:none PASS 100% 74 in → 5 out (0 reasoning)
Status: Resolved
z-ai/glm-4.7-flash:none__json_schema PASS 100% 74 in → 11 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
z-ai/glm-4.7-flash:xhigh PASS 100% 74 in → 1,332 out (1,427 reasoning)
Status: Resolved
z-ai/glm-5.2:high PASS 100% 81 in → 644 out (638 reasoning)
Status: Resolved
z-ai/glm-5.2:high__json_schema PASS 100% 83 in → 597 out (583 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
z-ai/glm-5.2:low PASS 100% 81 in → 633 out (627 reasoning)
Status: Resolved
z-ai/glm-5.2:low__json_schema PASS 100% 81 in → 659 out (693 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
z-ai/glm-5.2:medium PASS 100% 81 in → 395 out (391 reasoning)
Status: Resolved
z-ai/glm-5.2:xhigh PASS 100% 81 in → 542 out (536 reasoning)
Status: Resolved
z-ai/glm-5.2:xhigh__json_schema PASS 100% 81 in → 1,727 out (1,797 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
anthropic/claude-fable-5:max FAIL 0% 141 in → 4 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['status.txt']
anthropic/claude-fable-5:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
anthropic/claude-haiku-4.5:none FAIL 0% 91 in → 10 out (0 reasoning)
``` Status: Failed ```
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.6:high FAIL 0% 91 in → 242 out (192 reasoning)
``` Status: Resolved ``` **Reasoning:** The HEAD branch reported a failure (`Status: Failed`), and the `fix` branch was explicitly created to address it (`Status: Resolved`). Accepting the fix branch's change is the logical resolution.
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.6:low FAIL 0% 91 in → 6 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.6:max FAIL 0% 91 in → 262 out (176 reasoning)
``` Status: Resolved ``` **Reasoning:** The `fix` branch explicitly resolved the status, changing it to "Resolved." The `Failed` state on HEAD represents a problem, while the `fix` branch's purpose was to address it. Accepting the fix branch's change is the logical resolution.
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.6:medium FAIL 0% 91 in → 7 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.6:none FAIL 0% 91 in → 6 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.7:low__json_schema FAIL 0% 366 in → 23 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.7:medium__json_schema FAIL 0% 366 in → 23 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.7:none FAIL 0% 146 in → 10 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.7:none__json_schema FAIL 0% 366 in → 21 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.7:xhigh FAIL 0% 146 in → 10 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.8:high FAIL 0% 141 in → 583 out (93 reasoning)
These two changes are semantically contradictory—a status can't be both "Failed" and "Resolved"—so there's no mechanical way to merge them. This one genuinely needs a human decision about which state is correct. If you can tell me which outcome actually reflects reality (did the issue get fixed, or did it fail?), I'll give you the clean resolved file. Otherwise, the two candidate resolutions are: **If resolved:** ``` Status: Resolved ``` **If failed:** ``` Status: Failed ```
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.8:low FAIL 0% 141 in → 329 out (84 reasoning)
These two changes are semantically contradictory ("Resolved" vs "Failed"), so this isn't a conflict I can safely auto-resolve without knowing the actual system state—picking one could assert something false. You need to determine the real status and choose accordingly: - If the issue was fixed: ``` Status: Resolved ``` - If it actually failed: ``` Status: Failed ``` Please confirm which reflects reality, and I'll give you just the final file content.
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.8:max FAIL 0% 141 in → 1,750 out (276 reasoning)
I can't resolve this one for you the way the instruction implies, and I want to be upfront about why rather than just picking one. This isn't a formatting/structural conflict where the "correct" merge is deducible from the diff. The two branches make **directly contradictory factual claims**: - Branch A: `Status: Resolved` (success) - Branch B: `Status: Failed` (failure) There's nothing in the base (`Status: OK`) or the conflict markers that indicates which reflects the true state. If I just output one, I'd be fabricating a decision and presenting a coin-flip as an answer — and if this status drives anything downstream, guessing wrong could be genuinely misleading. To give you the clean file, I need one piece of info: **which status is actually true right now?** (Or, if there's authority/recency ordering between the branches, tell me that.) Once you confirm, the resolved content is trivially one of: ``` Status: Resolved ``` or ``` Status: Failed ```
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.8:medium FAIL 0% 141 in → 686 out (63 reasoning)
This conflict can't be resolved automatically, because the two branches make **contradictory factual claims** ("Failed" vs. "Resolved"), and there's no way to know which reflects the actual state of whatever `status.txt` describes. Unlike a formatting or additive conflict, picking one here is a substantive decision that could be wrong. To resolve it correctly, you need to answer: **what is the true status?** - If the underlying issue was actually fixed → keep `Status: Resolved` - If it actually failed → keep `Status: Failed` If you can tell me which reflects reality (or the intent of the branch that should win), I'll give you the clean file. If I had to guess with zero context I couldn't do so reliably, so I'd rather not hand you a plausible-looking but possibly incorrect status. That said, if you just want the mechanical result of taking **Branch A (fix)**, it would be: ``` Status: Resolved ``` Or taking **Branch B (HEAD)**: ``` Status: Failed ``` Let me know which is correct and I'll confirm the final content.
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.8:none FAIL 0% 141 in → 364 out (0 reasoning)
I need to resolve this merge conflict, but I should point out something important first: **I don't have enough information to know which resolution is correct.** Here's the issue: "Resolved" and "Failed" are semantically **opposite** outcomes. Choosing wrong could hide a real failure or falsely report success. This isn't a formatting conflict I can safely auto-resolve—it depends on facts I don't have: - **What is the actual status of the thing being tracked?** Did the process/fix/test actually succeed or fail? - Which branch reflects reality? (Branch names can be misleading—"fix" might have failed, or "HEAD" might be stale.) Since you asked for *only* the resolved content, and if I had to pick, "Failed" is arguably the safer default (fail-closed—it prompts investigation rather than silently claiming success). But I'd strongly recommend you verify before committing. If you can tell me the actual outcome, I'll give you the correct file. Otherwise, here is the content assuming the safer fail-closed default: ``` Status: Failed ``` Please confirm the real status before you commit this.
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.8:xhigh FAIL 0% 141 in → 1,198 out (158 reasoning)
I can't reliably resolve this one for you, and I want to flag why rather than just guessing. This is a **semantic conflict**, not a mechanical one. "Status: Resolved" and "Status: Failed" are mutually exclusive claims about the actual state of something. Unlike conflicts where you can combine both changes (e.g., two different lines added), here only one can be true, and picking one arbitrarily could propagate wrong information. To resolve it correctly, someone needs to answer: **what is the real status?** - If the underlying issue was actually fixed → keep `Status: Resolved` (from branch `fix`) - If it actually failed → keep `Status: Failed` (from `HEAD`) Once you tell me which reflects reality (or if there's context that makes one obviously correct), I'll give you the clean one-line file. I just don't want to hand you a confidently-wrong status.
Failure: Content mismatch: ['status.txt']
anthropic/claude-sonnet-4.6:max FAIL 0% 91 in → 635 out (251 reasoning)
``` Status: Resolved ``` **Reasoning:** - **Base:** `Status: OK` - **HEAD (Branch B):** `Status: Failed` - **fix (Branch A):** `Status: Resolved` Both branches diverged from the base with conflicting changes. The `fix` branch change (`Resolved`) is chosen as it semantically aligns with the branch's purpose — fixing and resolving an issue — making it the more meaningful and intentional state to preserve.
Failure: Content mismatch: ['status.txt']
anthropic/claude-sonnet-4.6:medium FAIL 0% 91 in → 444 out (216 reasoning)
``` Status: Resolved ``` **Reasoning:** Both branches modified the base (`Status: OK`), creating a true conflict. The `fix` branch intentionally set the status to `Resolved`, which semantically aligns with the branch's purpose (a fix that resolves the issue). The `HEAD` branch value `Failed` is treated as the superseded state being corrected by the fix branch.
Failure: Content mismatch: ['status.txt']
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 91 in → 576 out (262 reasoning)
``` Status: Resolved ``` **Resolution rationale:** Both branches modified the base (`Status: OK`), creating a true conflict. The `fix` branch change (`Status: Resolved`) was chosen as it semantically aligns with a fix being completed, while `Status: Failed` (HEAD) contradicts the purpose of a fix branch. Manual review should confirm which state is correct for the intended workflow.
Failure: Content mismatch: ['status.txt']
anthropic/claude-sonnet-5:low FAIL 0% 141 in → 8 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 78 in → 865 out (1,040 reasoning)
Status: OK
Failure: Content mismatch: ['status.txt']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 78 in → 754 out (900 reasoning)
Status: Resolvedand Failed
Failure: Content mismatch: ['status.txt']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 78 in → 1,104 out (1,294 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 85 in → 4 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
deepseek/deepseek-v4-flash:medium__json_schema FAIL 0% 81 in → 810 out (795 reasoning)
Status: OK
JSON Schema Structured Output
(raw) { "resolved_content": "Status: OK" }
Failure: Content mismatch: ['status.txt']
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 79 in → 28 out (0 reasoning)
<<<<<<< HEAD ======= Status: Resolved >>>>>>> fix
JSON Schema Structured Output
(raw) { "resolved_content": "<<<<<<< HEAD\n=======\nStatus: Resolved\n>>>>>>> fix\n" }
Failure: Content mismatch: ['status.txt']
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 79 in → 17 out (0 reasoning)
Status: Failed and Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed and Resolved" }
Failure: Content mismatch: ['status.txt']
google/gemma-4-31b-it:low__json_schema FAIL 0% 92 in → 437 out (426 reasoning)
}Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"}Status: Resolved"}
Failure: Content mismatch: ['status.txt']
ibm-granite/granite-4.1-8b__json_schema FAIL 0% 83 in → 14 out
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
Failure: Content mismatch: ['status.txt']
inclusionai/ling-2.6-flash FAIL 0% 95 in → 4 out
Status: Failed
Failure: Content mismatch: ['status.txt']
inclusionai/ling-2.6-flash__json_schema FAIL 0% 95 in → 15 out
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
Failure: Content mismatch: ['status.txt']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:low FAIL 0% 115 in → 460 out (536 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
minimax/minimax-m2.7:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
mistralai/mistral-medium-3-5:none FAIL 0% 91 in → 4 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
mistralai/mistral-medium-3-5:none__json_schema FAIL 0% 79 in → 11 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
Failure: Content mismatch: ['status.txt']
moonshotai/kimi-k2.6:medium FAIL 0% 77 in → 996 out (996 reasoning)
(empty output)
Failure: Content mismatch: ['status.txt']
moonshotai/kimi-k2.6:xhigh FAIL 0% 77 in → 1,968 out (2,027 reasoning)
Status: Resolved
Failure: Content mismatch: ['status.txt']
moonshotai/kimi-k2.7-code:high FAIL 0% 77 in → 1,757 out (2,105 reasoning)
Status: Resolved
Failure: Content mismatch: ['status.txt']
moonshotai/kimi-k2.7-code:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k3:low FAIL 0% 154 in → 1,875 out (1,701 reasoning)
This conflict can't be resolved correctly from the information given — it's a semantic conflict, not a mechanical one. Both branches changed the same line in contradictory ways (`Failed` vs. `Resolved`), and a status generally can't be both. There's no merge rule that determines which value is right; that depends on what the status actually *should* be. The two valid resolutions are: ``` Status: Resolved ``` — if the `fix` branch genuinely fixed the issue, or: ``` Status: Failed ``` — if the failure recorded on HEAD is still the true state. Tell me which status reflects reality (or check with the branch authors / the underlying system), and the resolved file is simply that single line with the conflict markers removed. Arbitrarily picking one would risk silently recording the wrong state.
Failure: Content mismatch: ['status.txt']
nvidia/nemotron-3-nano-30b-a3b:medium FAIL 0% 92 in → 32,768 out (376 reasoning)
(empty output)
Failure: Content mismatch: ['status.txt']
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 92 in → 12 out (0 reasoning)
Status: Resolved Status: Resolved
Failure: Content mismatch: ['status.txt']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 92 in → 14 out (0 reasoning)
Status: OK
JSON Schema Structured Output
(raw) { "resolved_content": "Status: OK" }
Failure: Content mismatch: ['status.txt']
openai/gpt-5.4:none FAIL 0% 75 in → 10 out (0 reasoning)
Status: Resolved / Failed
Failure: Content mismatch: ['status.txt']
openai/gpt-5.6-sol:low__json_schema FAIL 0% 111 in → 86 out (68 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
Failure: Content mismatch: ['status.txt']
openai/gpt-oss-120b:high__json_schema FAIL 0% 138 in → 1,386 out (1,668 reasoning)
Which status should the final file contain—"Status: Resolved" or "Status: Failed"?
JSON Schema Structured Output
(raw) {"resolved_content":"Which status should the final file contain—\"Status: Resolved\" or \"Status: Failed\"?"}
Failure: Content mismatch: ['status.txt']
openai/gpt-oss-120b:low__json_schema FAIL 0%
Invalid structured output. Output: [ "Status: Resolved" ]
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
openai/gpt-oss-120b:xhigh FAIL 0% 139 in → 872 out (1,035 reasoning)
Status
Failure: Content mismatch: ['status.txt']
poolside/laguna-m.1:low__json_schema FAIL 0% 84 in → 791 out (762 reasoning)
<<<<<<< HEAD Status: Resolved ======= Status: Resolved >>>>>>> fix
JSON Schema Structured Output
(raw) { "resolved_content": "<<<<<<< HEAD\nStatus: Resolved\n=======\nStatus: Resolved\n>>>>>>> fix" }
Failure: Content mismatch: ['status.txt']
poolside/laguna-xs-2.1:none FAIL 0% 84 in → 8 out (0 reasoning)
Status: Resolved </think>Status: Resolved
Failure: Content mismatch: ['status.txt']
poolside/laguna-xs.2:low FAIL 0% 121 in → 278 out (272 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
poolside/laguna-xs.2:low__json_schema FAIL 0% 121 in → 452 out (436 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
Failure: Content mismatch: ['status.txt']
poolside/laguna-xs.2:none FAIL 0% 121 in → 52 out (0 reasoning)
I'll resolve this merge conflict by choosing one of the changes. Since this appears to be a status file where both branches have modified the status, I'll select 'Status: Resolved' as the final content. ``` Status: Resolved ```
Failure: Content mismatch: ['status.txt']
poolside/laguna-xs.2:none__json_schema FAIL 0% 121 in → 10 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
Failure: Content mismatch: ['status.txt']
poolside/laguna-xs.2:xhigh FAIL 0% 121 in → 352 out (346 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
qwen/qwen3.6-27b:none__json_schema FAIL 0% 83 in → 15 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
Failure: Content mismatch: ['status.txt']
qwen/qwen3.6-27b:xhigh__json_schema FAIL 0% 81 in → 1,268 out (1,351 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
Failure: Content mismatch: ['status.txt']
qwen/qwen3.6-35b-a3b:none FAIL 0% 83 in → 4 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 83 in → 15 out (0 reasoning)
Status: OK
JSON Schema Structured Output
(raw) { "resolved_content": "Status: OK" }
Failure: Content mismatch: ['status.txt']
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3a1b7b6b-ed5f-9a18-b65f-a9e9a1847e7f","request_id":"3a1b7b6b-ed5f-9a18-b65f-a9e9a1847e7f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b62ae2b1-5deb-92de-a4ba-5266e51136ff","request_id":"b62ae2b1-5deb-92de-a4ba-5266e51136ff"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3c654ffc-88da-9f2a-9f47-8efbd97e313a","request_id":"3c654ffc-88da-9f2a-9f47-8efbd97e313a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none FAIL 0% 83 in → 3 out (0 reasoning)
Status: OK
Failure: Content mismatch: ['status.txt']
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5acd4910-9264-9fdb-ab71-1e2a517f4bc6","request_id":"5acd4910-9264-9fdb-ab71-1e2a517f4bc6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fc0bb92d-ca6b-9c08-8e44-d8f525553aa6","request_id":"fc0bb92d-ca6b-9c08-8e44-d8f525553aa6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-81917f33-7834-9c0b-92fc-7726ee87b3a2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-735af62e-2441-9486-a052-ca8d15ce391c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2e21c65b-e196-98b4-9424-1fdf2fe7f38a"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6b0d932b-a198-9cb5-b7eb-dec650cd6952"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7724dbf0-c6d7-953e-b7b7-ccbb7d9c8b57"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-18af46ef-2148-9e49-a453-a92636e9d846"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8e0c360d-eec7-9f05-b509-03aae9d3e8b5","request_id":"8e0c360d-eec7-9f05-b509-03aae9d3e8b5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-14b3b855-012a-9cfe-a589-0f5bcdef821f","request_id":"14b3b855-012a-9cfe-a589-0f5bcdef821f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f1f334fa-562c-9e28-ad90-3bc064aeec79","request_id":"f1f334fa-562c-9e28-ad90-3bc064aeec79"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none FAIL 0% 83 in → 3 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-035c3125-8268-92ea-9383-0c056155e0e7","request_id":"035c3125-8268-92ea-9383-0c056155e0e7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-13d62664-5ada-99b0-8900-ad81f88fca0d","request_id":"13d62664-5ada-99b0-8900-ad81f88fca0d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d6ab952d-029b-9f1b-95a4-533393569669","request_id":"d6ab952d-029b-9f1b-95a4-533393569669"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-81895ea6-e072-90de-bb6d-c74ced8488bf","request_id":"81895ea6-e072-90de-bb6d-c74ced8488bf"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-066e9ab8-d05b-9b67-9478-7126e3b331d6","request_id":"066e9ab8-d05b-9b67-9478-7126e3b331d6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1907ec83-db88-9e17-be3a-9b893af653b5","request_id":"1907ec83-db88-9e17-be3a-9b893af653b5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-94823773-74f1-9d2c-9801-acc6beb949c7","request_id":"94823773-74f1-9d2c-9801-acc6beb949c7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:high FAIL 0% 85 in → 3,134 out (3,124 reasoning)
Status: Failed Status: Resolved
Failure: Content mismatch: ['status.txt']
tencent/hy3:low__json_schema FAIL 0% 85 in → 1,930 out (2,046 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
Failure: Content mismatch: ['status.txt']
tencent/hy3:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Model 'tencent/hy3:none' violated the reasoning_level=none invariant: provider reported 2577 reasoning token(s)
tencent/hy3:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling-small:none FAIL 0% 84 in → 7 out (0 reasoning)
Status: Resolved<|end_message|>
Failure: Content mismatch: ['status.txt']
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved{ "resolved_content": "Status: Resolved" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 82 in → 12 out (0 reasoning)
Status: Resolved Status: Failed
Failure: Content mismatch: ['status.txt']
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved{ "resolved_content": "Status: Resolved" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.3:none FAIL 0% 250 in → 3 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:medium__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:none FAIL 0% 75 in → 4 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
z-ai/glm-5.2:none__json_schema FAIL 0% 77 in → 17 out (0 reasoning)
Status: Resolved Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\nStatus: Failed" }
Failure: Content mismatch: ['status.txt']