f003 — cherry_pick

← cherry_pick
Single-line status message conflict
Tests ability to resolve a status-message conflict. Evaluates choosing between conflicting status values from different branches.
easy cherry-pick conflict-resolution status-message text-merge
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Status: OK' > status.txt
  5. 05 git add status.txt
  6. 06 git commit -m 'Initial status'
  7. 07 git checkout -b fix
  8. 08 echo 'Status: Resolved' > status.txt
  9. 09 git add status.txt
  10. 10 git commit -m 'Fix status'
  11. 11 git checkout main
  12. 12 echo 'Status: Failed' > status.txt
  13. 13 git add status.txt
  14. 14 git commit -m 'Update status'
  15. 15 git cherry-pick $(git rev-parse fix)
Prompt
Resolve the cherry-pick conflict in status.txt. The base was 'Status: OK'. The incoming commit changed it to 'Status: Resolved'. The current branch has 'Status: Failed'. Provide ONLY the resolved file content.
Expected
Status: Resolved
Fixture Metrics

Scoped model quality, cost, API time, and token usage for cherry_pick/f003.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 153 in → 53 out (26 reasoning)
Status: Resolved
anthropic/claude-fable-5:high__json_schema PASS 100% 376 in → 55 out (16 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
anthropic/claude-fable-5:low PASS 100% 156 in → 39 out (16 reasoning)
Status: Resolved
anthropic/claude-fable-5:low__json_schema PASS 100% 374 in → 20 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-fable-5:max__json_schema PASS 100% 374 in → 304 out (117 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
anthropic/claude-fable-5:medium PASS 100% 155 in → 78 out (51 reasoning)
Status: Resolved
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 373 in → 104 out (55 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
anthropic/claude-haiku-4.5:high PASS 100% 132 in → 713 out (566 reasoning)
``` Status: Resolved ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 297 in → 1,021 out (663 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-haiku-4.5:low PASS 100% 131 in → 485 out (523 reasoning)
``` Status: Resolved ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 299 in → 882 out (570 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-haiku-4.5:medium PASS 100% 130 in → 664 out (620 reasoning)
``` Status: Resolved ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 300 in → 801 out (555 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-haiku-4.5:none PASS 100% 101 in → 11 out (0 reasoning)
``` Status: Resolved ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 267 in → 14 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 130 in → 608 out (588 reasoning)
``` Status: Resolved ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 298 in → 1,743 out (835 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-opus-4.6:high PASS 100% 103 in → 101 out (84 reasoning)
Status: Resolved
anthropic/claude-opus-4.6:high__json_schema PASS 100% 270 in → 119 out (95 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.6:low PASS 100% 101 in → 7 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.6:low__json_schema PASS 100% 269 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 271 in → 178 out (158 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.6:medium PASS 100% 102 in → 7 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 269 in → 14 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.6:none PASS 100% 102 in → 7 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.6:none__json_schema PASS 100% 268 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 271 in → 175 out (157 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.7:high PASS 100% 158 in → 11 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.7:high__json_schema PASS 100% 378 in → 24 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.7:low PASS 100% 161 in → 11 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.7:low__json_schema PASS 100% 380 in → 24 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.7:max PASS 100% 159 in → 395 out (66 reasoning)
Status: Resolved
anthropic/claude-opus-4.7:max__json_schema PASS 100% 378 in → 272 out (87 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.7:medium PASS 100% 158 in → 11 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 379 in → 24 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.7:none PASS 100% 157 in → 11 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.7:none__json_schema PASS 100% 378 in → 24 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 161 in → 11 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 380 in → 24 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.8:high PASS 100% 153 in → 179 out (78 reasoning)
Status: Resolved
anthropic/claude-opus-4.8:high__json_schema PASS 100% 373 in → 20 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.8:low PASS 100% 154 in → 54 out (26 reasoning)
Status: Resolved
anthropic/claude-opus-4.8:low__json_schema PASS 100% 374 in → 22 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-opus-4.8:max__json_schema PASS 100% 375 in → 214 out (63 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-opus-4.8:medium PASS 100% 153 in → 141 out (59 reasoning)
Status: Resolved
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 374 in → 20 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-opus-4.8:none PASS 100% 152 in → 9 out (0 reasoning)
Status: Resolved
anthropic/claude-opus-4.8:none__json_schema PASS 100% 375 in → 21 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 153 in → 389 out (79 reasoning)
Status: Resolved
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 375 in → 126 out (61 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-sonnet-4.6:high PASS 100% 101 in → 161 out (140 reasoning)
Status: Resolved
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 269 in → 161 out (138 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 101 in → 7 out (0 reasoning)
Status: Resolved
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 270 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-sonnet-4.6:max PASS 100% 101 in → 191 out (183 reasoning)
Status: Resolved
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 269 in → 365 out (202 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 102 in → 84 out (68 reasoning)
Status: Resolved
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 269 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-sonnet-4.6:none PASS 100% 100 in → 7 out (0 reasoning)
Status: Resolved
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 271 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 101 in → 216 out (195 reasoning)
Status: Resolved
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 271 in → 129 out (100 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-sonnet-5:high PASS 100% 153 in → 177 out (69 reasoning)
Status: Resolved
anthropic/claude-sonnet-5:high__json_schema PASS 100% 375 in → 120 out (71 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
anthropic/claude-sonnet-5:low PASS 100% 155 in → 9 out (0 reasoning)
Status: Resolved
anthropic/claude-sonnet-5:low__json_schema PASS 100% 374 in → 22 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
anthropic/claude-sonnet-5:medium PASS 100% 154 in → 9 out (0 reasoning)
Status: Resolved
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 374 in → 43 out (17 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
anthropic/claude-sonnet-5:none PASS 100% 153 in → 9 out (0 reasoning)
Status: Resolved
anthropic/claude-sonnet-5:none__json_schema PASS 100% 374 in → 19 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
anthropic/claude-sonnet-5:xhigh PASS 100% 154 in → 320 out (176 reasoning)
Status: Resolved
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 375 in → 273 out (97 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 87 in → 892 out (913 reasoning)
Status: Resolved
arcee-ai/trinity-large-thinking:low PASS 100% 86 in → 1,159 out (1,176 reasoning)
Status: Resolved
arcee-ai/trinity-large-thinking:medium PASS 100% 84 in → 1,073 out (1,107 reasoning)
Status: Resolved
arcee-ai/trinity-large-thinking:xhigh PASS 100% 85 in → 1,432 out (1,484 reasoning)
Status: Resolved
arcee-ai/trinity-mini:high__json_schema PASS 100% 88 in → 1,033 out (1,214 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
arcee-ai/trinity-mini:low__json_schema PASS 100% 86 in → 1,051 out (1,229 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 86 in → 591 out (683 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 86 in → 630 out (730 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 166 in → 835 out (830 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 233 in → 256 out (262 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 89 in → 275 out (313 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 88 in → 404 out (442 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 186 in → 4,859 out (5,450 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 167 in → 1,353 out (1,518 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 90 in → 227 out (222 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 88 in → 520 out (565 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 86 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 168 in → 1,524 out (1,519 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 89 in → 755 out (865 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash:high PASS 100% 89 in → 391 out (436 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 90 in → 274 out (258 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash:low PASS 100% 86 in → 308 out (303 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 89 in → 334 out (318 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash:medium PASS 100% 87 in → 575 out (652 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 88 in → 318 out (348 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash:none PASS 100% 87 in → 4 out (0 reasoning)
Status: Resolved
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 88 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 89 in → 320 out (363 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:high PASS 100% 88 in → 5 out (0 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 90 in → 445 out (429 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-pro:low PASS 100% 87 in → 665 out (659 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 89 in → 547 out (530 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
deepseek/deepseek-v4-pro:medium PASS 100% 89 in → 380 out (374 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 89 in → 744 out (729 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-pro:none PASS 100% 87 in → 4 out (0 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 89 in → 14 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 88 in → 5 out (0 reasoning)
Status: Resolved
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 360 in → 860 out (847 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3-flash-preview:high PASS 100% 85 in → 565 out (562 reasoning)
Status: Resolved
google/gemini-3-flash-preview:high__json_schema PASS 100% 144 in → 485 out (476 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3-flash-preview:low PASS 100% 85 in → 330 out (327 reasoning)
Status: Resolved
google/gemini-3-flash-preview:low__json_schema PASS 100% 144 in → 361 out (352 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3-flash-preview:medium PASS 100% 84 in → 387 out (384 reasoning)
Status: Resolved
google/gemini-3-flash-preview:medium__json_schema PASS 100% 145 in → 349 out (335 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3-flash-preview:xhigh PASS 100% 85 in → 552 out (549 reasoning)
Status: Resolved
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 145 in → 624 out (615 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 87 in → 261 out (258 reasoning)
Status: Resolved
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 86 in → 682 out (673 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 85 in → 152 out (149 reasoning)
Status: Resolved
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 84 in → 151 out (136 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 86 in → 109 out (106 reasoning)
Status: Resolved
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 84 in → 143 out (133 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 85 in → 311 out (308 reasoning)
Status: Resolved
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 146 in → 216 out (206 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.1-pro-preview:high PASS 100% 82 in → 651 out (648 reasoning)
Status: Resolved
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 145 in → 612 out (603 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.1-pro-preview:low PASS 100% 85 in → 441 out (438 reasoning)
Status: Resolved
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 144 in → 367 out (357 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.1-pro-preview:medium PASS 100% 85 in → 578 out (575 reasoning)
Status: Resolved
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 146 in → 518 out (508 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 84 in → 337 out (334 reasoning)
Status: Resolved
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 145 in → 316 out (306 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.5-flash-lite:high PASS 100% 86 in → 941 out (938 reasoning)
Status: Resolved
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 85 in → 1,020 out (1,010 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemini-3.5-flash-lite:low PASS 100% 85 in → 3 out (0 reasoning)
Status: Resolved
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 145 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.5-flash-lite:max PASS 100% 87 in → 969 out (966 reasoning)
Status: Resolved
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 86 in → 791 out (776 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.5-flash-lite:medium PASS 100% 84 in → 661 out (658 reasoning)
Status: Resolved
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 146 in → 798 out (789 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 85 in → 914 out (911 reasoning)
Status: Resolved
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 85 in → 911 out (896 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.5-flash:high PASS 100% 85 in → 1,070 out (1,067 reasoning)
Status: Resolved
google/gemini-3.5-flash:high__json_schema PASS 100% 145 in → 1,057 out (1,042 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.5-flash:low PASS 100% 86 in → 379 out (376 reasoning)
Status: Resolved
google/gemini-3.5-flash:low__json_schema PASS 100% 145 in → 242 out (227 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.5-flash:medium PASS 100% 85 in → 735 out (732 reasoning)
Status: Resolved
google/gemini-3.5-flash:medium__json_schema PASS 100% 145 in → 645 out (636 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.5-flash:xhigh PASS 100% 85 in → 782 out (779 reasoning)
Status: Resolved
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 144 in → 862 out (850 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
google/gemini-3.6-flash:high PASS 100% 86 in → 507 out (504 reasoning)
Status: Resolved
google/gemini-3.6-flash:high__json_schema PASS 100% 145 in → 1,207 out (1,198 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemini-3.6-flash:low PASS 100% 86 in → 269 out (266 reasoning)
Status: Resolved
google/gemini-3.6-flash:low__json_schema PASS 100% 87 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemini-3.6-flash:max PASS 100% 84 in → 818 out (815 reasoning)
Status: Resolved
google/gemini-3.6-flash:max__json_schema PASS 100% 143 in → 1,112 out (1,101 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
google/gemini-3.6-flash:medium PASS 100% 86 in → 814 out (811 reasoning)
Status: Resolved
google/gemini-3.6-flash:medium__json_schema PASS 100% 86 in → 736 out (725 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
google/gemini-3.6-flash:xhigh PASS 100% 85 in → 867 out (864 reasoning)
Status: Resolved
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 85 in → 1,707 out (1,696 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
google/gemma-4-26b-a4b-it:high PASS 100% 99 in → 392 out (385 reasoning)
Status: Resolved
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 100 in → 515 out (493 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-26b-a4b-it:low PASS 100% 101 in → 1,144 out (1,148 reasoning)
Status: Resolved
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 101 in → 457 out (436 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-26b-a4b-it:medium PASS 100% 102 in → 798 out (796 reasoning)
Status: Resolved
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 99 in → 545 out (534 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-26b-a4b-it:none PASS 100% 99 in → 4 out (0 reasoning)
Status: Resolved
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 97 in → 11 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemma-4-26b-a4b-it:xhigh PASS 100% 101 in → 711 out (734 reasoning)
Status: Resolved
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 102 in → 514 out (512 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-31b-it:high PASS 100% 102 in → 201 out (192 reasoning)
Status: Resolved
google/gemma-4-31b-it:high__json_schema PASS 100% 101 in → 197 out (178 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
google/gemma-4-31b-it:low PASS 100% 101 in → 267 out (265 reasoning)
Status: Resolved
google/gemma-4-31b-it:low__json_schema PASS 100% 105 in → 253 out (242 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
google/gemma-4-31b-it:medium PASS 100% 101 in → 202 out (203 reasoning)
Status: Resolved
google/gemma-4-31b-it:medium__json_schema PASS 100% 101 in → 255 out (226 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-31b-it:none PASS 100% 98 in → 4 out (0 reasoning)
Status: Resolved
google/gemma-4-31b-it:none__json_schema PASS 100% 97 in → 16 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
google/gemma-4-31b-it:xhigh PASS 100% 100 in → 238 out (235 reasoning)
Status: Resolved
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 102 in → 219 out (201 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
ibm-granite/granite-4.1-8b PASS 100% 92 in → 5 out
Status: Resolved
ibm-granite/granite-4.1-8b__json_schema PASS 100% 92 in → 15 out
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
inclusionai/ling-2.6-flash PASS 100% 108 in → 5 out
Status: Resolved
liquid/lfm-2-24b-a2b PASS 100% 101 in → 8 out
status.txt Status: Resolved
minimax/minimax-m2.5:high PASS 100% 125 in → 367 out (417 reasoning)
Status: Resolved
minimax/minimax-m2.5:high__json_schema PASS 100% 128 in → 231 out (217 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m2.5:low PASS 100% 120 in → 548 out (542 reasoning)
Status: Resolved
minimax/minimax-m2.5:low__json_schema PASS 100% 119 in → 705 out (819 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m2.5:medium PASS 100% 120 in → 313 out (346 reasoning)
Status: Resolved
minimax/minimax-m2.5:medium__json_schema PASS 100% 119 in → 416 out (467 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m2.5:xhigh PASS 100% 129 in → 365 out (359 reasoning)
Status: Resolved
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 119 in → 577 out (673 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m2.7:high PASS 100% 119 in → 543 out (652 reasoning)
Status: Resolved
minimax/minimax-m2.7:high__json_schema PASS 100% 252 in → 917 out (902 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m2.7:low PASS 100% 121 in → 474 out (548 reasoning)
Status: Resolved
minimax/minimax-m2.7:low__json_schema PASS 100% 120 in → 14 out (12 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
minimax/minimax-m2.7:medium PASS 100% 124 in → 376 out (427 reasoning)
Status: Resolved
minimax/minimax-m2.7:medium__json_schema PASS 100% 251 in → 441 out (430 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
minimax/minimax-m2.7:xhigh PASS 100% 119 in → 703 out (844 reasoning)
Status: Resolved
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 250 in → 620 out (609 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
minimax/minimax-m3:high PASS 100% 257 in → 182 out (208 reasoning)
Status: Resolved
minimax/minimax-m3:high__json_schema PASS 100% 244 in → 14 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m3:low PASS 100% 257 in → 150 out (158 reasoning)
``` Status: Resolved ```
minimax/minimax-m3:low__json_schema PASS 100% 245 in → 11 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
minimax/minimax-m3:medium PASS 100% 256 in → 404 out (470 reasoning)
Status: Resolved
minimax/minimax-m3:medium__json_schema PASS 100% 244 in → 14 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
minimax/minimax-m3:xhigh PASS 100% 257 in → 91 out (101 reasoning)
Status: Resolved
minimax/minimax-m3:xhigh__json_schema PASS 100% 205 in → 17 out (12 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
mistralai/devstral-2512 PASS 100% 90 in → 5 out
Status: Resolved
mistralai/devstral-2512__json_schema PASS 100% 91 in → 12 out
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
mistralai/mistral-medium-3-5:high PASS 100% 99 in → 851 out (971 reasoning)
Status: Resolved
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 89 in → 658 out (725 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
mistralai/mistral-medium-3-5:low PASS 100% 101 in → 768 out (845 reasoning)
Status: Resolved
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 88 in → 228 out (246 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
mistralai/mistral-medium-3-5:medium PASS 100% 102 in → 834 out (952 reasoning)
Status: Resolved
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 90 in → 1,084 out (1,188 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
mistralai/mistral-medium-3-5:none PASS 100% 102 in → 5 out (0 reasoning)
Status: Resolved
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 91 in → 12 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 103 in → 501 out (558 reasoning)
Status: Resolved
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 90 in → 1,148 out (1,278 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
moonshotai/kimi-k2.6:high PASS 100% 86 in → 643 out (638 reasoning)
Status: Resolved
moonshotai/kimi-k2.6:high__json_schema PASS 100% 85 in → 1,182 out (997 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
moonshotai/kimi-k2.6:low PASS 100% 86 in → 1,087 out (1,083 reasoning)
Status: Resolved
moonshotai/kimi-k2.6:low__json_schema PASS 100% 84 in → 956 out (1,075 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
moonshotai/kimi-k2.6:medium PASS 100% 86 in → 611 out (606 reasoning)
Status: Resolved
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 86 in → 512 out (500 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
moonshotai/kimi-k2.6:xhigh PASS 100% 85 in → 951 out (948 reasoning)
Status: Resolved
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 88 in → 2,232 out (2,220 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
moonshotai/kimi-k2.7-code:high PASS 100% 84 in → 185 out (180 reasoning)
Status: Resolved
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 86 in → 218 out (208 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
moonshotai/kimi-k2.7-code:low PASS 100% 85 in → 685 out (680 reasoning)
Status: Resolved
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 171 in → 259 out (247 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 87 in → 250 out (296 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 86 in → 386 out (381 reasoning)
Status: Resolved
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 84 in → 352 out (334 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
moonshotai/kimi-k3:high PASS 100% 163 in → 579 out (561 reasoning)
Status: Resolved
moonshotai/kimi-k3:high__json_schema PASS 100% 163 in → 232 out (214 reasoning)
Status: Resolved
moonshotai/kimi-k3:low PASS 100% 164 in → 264 out (246 reasoning)
Status: Resolved
moonshotai/kimi-k3:low__json_schema PASS 100% 260 in → 215 out (192 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
moonshotai/kimi-k3:max PASS 100% 162 in → 459 out (441 reasoning)
Status: Resolved
moonshotai/kimi-k3:max__json_schema PASS 100% 164 in → 396 out (378 reasoning)
Status: Resolved
moonshotai/kimi-k3:medium PASS 100% 165 in → 533 out (515 reasoning)
Status: Resolved
moonshotai/kimi-k3:medium__json_schema PASS 100% 163 in → 697 out (679 reasoning)
Status: Resolved
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 162 in → 238 out (220 reasoning)
Status: Resolved
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 103 in → 504 out (578 reasoning)
Status: Resolved
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 102 in → 659 out (730 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 104 in → 337 out (385 reasoning)
Status: Resolved
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 103 in → 1,937 out (2,083 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 104 in → 358 out (393 reasoning)
Status: Resolved
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 102 in → 332 out (355 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 102 in → 5 out (0 reasoning)
Status: Resolved
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 102 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 102 in → 729 out (818 reasoning)
Status: Resolved
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 103 in → 630 out (703 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 102 in → 394 out (376 reasoning)
Status: Resolved
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 102 in → 328 out (310 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 101 in → 503 out (588 reasoning)
Status: Resolved
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 103 in → 138 out (119 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 103 in → 450 out (437 reasoning)
Status: Resolved
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 103 in → 161 out (142 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 104 in → 5 out (0 reasoning)
Status: Resolved
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 102 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 103 in → 280 out (329 reasoning)
Status: Resolved
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 103 in → 225 out (208 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
openai/gpt-5.4-mini:high PASS 100% 83 in → 170 out (160 reasoning)
Status: Resolved
openai/gpt-5.4-mini:high__json_schema PASS 100% 120 in → 259 out (240 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-mini:low PASS 100% 84 in → 99 out (89 reasoning)
Status: Resolved
openai/gpt-5.4-mini:low__json_schema PASS 100% 120 in → 93 out (73 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
openai/gpt-5.4-mini:medium PASS 100% 85 in → 220 out (210 reasoning)
Status: Resolved
openai/gpt-5.4-mini:medium__json_schema PASS 100% 120 in → 217 out (198 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-mini:none PASS 100% 83 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.4-mini:none__json_schema PASS 100% 119 in → 23 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-mini:xhigh PASS 100% 84 in → 526 out (516 reasoning)
Status: Resolved
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 121 in → 983 out (964 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4-nano:high PASS 100% 84 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.4-nano:high__json_schema PASS 100% 120 in → 18 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
openai/gpt-5.4-nano:low PASS 100% 83 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.4-nano:low__json_schema PASS 100% 120 in → 18 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
openai/gpt-5.4-nano:medium PASS 100% 87 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.4-nano:medium__json_schema PASS 100% 119 in → 18 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
openai/gpt-5.4-nano:none PASS 100% 85 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.4-nano:none__json_schema PASS 100% 120 in → 18 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 82 in → 176 out (166 reasoning)
Status: Resolved
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 123 in → 453 out (433 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
openai/gpt-5.4:high PASS 100% 84 in → 935 out (925 reasoning)
Status: Resolved
openai/gpt-5.4:high__json_schema PASS 100% 119 in → 687 out (668 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4:low PASS 100% 84 in → 80 out (70 reasoning)
Status: Resolved
openai/gpt-5.4:low__json_schema PASS 100% 120 in → 91 out (72 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4:medium PASS 100% 85 in → 243 out (233 reasoning)
Status: Resolved
openai/gpt-5.4:medium__json_schema PASS 100% 119 in → 233 out (214 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4:none PASS 100% 83 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.4:none__json_schema PASS 100% 120 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.4:xhigh PASS 100% 82 in → 526 out (516 reasoning)
Status: Resolved
openai/gpt-5.4:xhigh__json_schema PASS 100% 119 in → 2,304 out (2,285 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.5:high PASS 100% 83 in → 355 out (345 reasoning)
Status: Resolved
openai/gpt-5.5:high__json_schema PASS 100% 119 in → 293 out (274 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.5:low PASS 100% 83 in → 92 out (82 reasoning)
Status: Resolved
openai/gpt-5.5:low__json_schema PASS 100% 121 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.5:medium PASS 100% 83 in → 215 out (205 reasoning)
Status: Resolved
openai/gpt-5.5:medium__json_schema PASS 100% 119 in → 151 out (132 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.5:none PASS 100% 85 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.5:none__json_schema PASS 100% 120 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.5:xhigh PASS 100% 85 in → 199 out (189 reasoning)
Status: Resolved
openai/gpt-5.5:xhigh__json_schema PASS 100% 118 in → 394 out (375 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:high PASS 100% 86 in → 109 out (99 reasoning)
Status: Resolved
openai/gpt-5.6-luna:high__json_schema PASS 100% 119 in → 74 out (55 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:low PASS 100% 84 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-luna:low__json_schema PASS 100% 121 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:max PASS 100% 83 in → 76 out (66 reasoning)
Status: Resolved
openai/gpt-5.6-luna:max__json_schema PASS 100% 120 in → 121 out (102 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:medium PASS 100% 84 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-luna:medium__json_schema PASS 100% 121 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:none PASS 100% 83 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-luna:none__json_schema PASS 100% 120 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-luna:xhigh PASS 100% 82 in → 64 out (54 reasoning)
Status: Resolved
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 119 in → 141 out (122 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-sol:high PASS 100% 83 in → 72 out (62 reasoning)
Status: Resolved
openai/gpt-5.6-sol:high__json_schema PASS 100% 119 in → 143 out (124 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-sol:low PASS 100% 85 in → 71 out (61 reasoning)
Status: Resolved
openai/gpt-5.6-sol:low__json_schema PASS 100% 121 in → 90 out (71 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-sol:max PASS 100% 83 in → 82 out (72 reasoning)
Status: Resolved
openai/gpt-5.6-sol:max__json_schema PASS 100% 120 in → 186 out (166 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved\n"}
openai/gpt-5.6-sol:medium PASS 100% 84 in → 85 out (75 reasoning)
Status: Resolved
openai/gpt-5.6-sol:medium__json_schema PASS 100% 121 in → 88 out (69 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-sol:none PASS 100% 84 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-sol:none__json_schema PASS 100% 120 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-sol:xhigh PASS 100% 85 in → 107 out (97 reasoning)
Status: Resolved
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 120 in → 144 out (125 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:high PASS 100% 83 in → 70 out (60 reasoning)
Status: Resolved
openai/gpt-5.6-terra:high__json_schema PASS 100% 121 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:low PASS 100% 86 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-terra:low__json_schema PASS 100% 119 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:max PASS 100% 85 in → 213 out (203 reasoning)
Status: Resolved
openai/gpt-5.6-terra:max__json_schema PASS 100% 119 in → 505 out (486 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:medium PASS 100% 85 in → 52 out (42 reasoning)
Status: Resolved
openai/gpt-5.6-terra:medium__json_schema PASS 100% 121 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:none PASS 100% 84 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-terra:none__json_schema PASS 100% 121 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-5.6-terra:xhigh PASS 100% 83 in → 8 out (0 reasoning)
Status: Resolved
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 120 in → 17 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-oss-120b:high PASS 100% 145 in → 151 out (147 reasoning)
Status: Resolved
openai/gpt-oss-120b:high__json_schema PASS 100% 143 in → 1,338 out (1,315 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
openai/gpt-oss-120b:low PASS 100% 147 in → 172 out (188 reasoning)
Status: Resolved
openai/gpt-oss-120b:low__json_schema PASS 100% 214 in → 59 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
openai/gpt-oss-120b:medium PASS 100% 144 in → 285 out (331 reasoning)
Status: Resolved
openai/gpt-oss-120b:medium__json_schema PASS 100% 214 in → 268 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
openai/gpt-oss-120b:xhigh PASS 100% 147 in → 377 out (433 reasoning)
Status: Resolved
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 131 in → 1,583 out (1,907 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
openai/gpt-oss-20b:high PASS 100% 144 in → 1,255 out (1,252 reasoning)
Status: Resolved
openai/gpt-oss-20b:high__json_schema PASS 100% 198 in → 1,004 out (1,183 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-oss-20b:low PASS 100% 130 in → 87 out (82 reasoning)
Status: Resolved
openai/gpt-oss-20b:low__json_schema PASS 100% 147 in → 52 out (35 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
openai/gpt-oss-20b:medium PASS 100% 145 in → 268 out (255 reasoning)
Status: Resolved
openai/gpt-oss-20b:xhigh PASS 100% 144 in → 875 out (1,021 reasoning)
Status: Resolved
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 147 in → 1,000 out (1,210 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
poolside/laguna-m.1:high PASS 100% 95 in → 877 out (871 reasoning)
Status: Resolved
poolside/laguna-m.1:high__json_schema PASS 100% 95 in → 756 out (741 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-m.1:low PASS 100% 94 in → 575 out (569 reasoning)
Status: Resolved
poolside/laguna-m.1:low__json_schema PASS 100% 93 in → 943 out (928 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-m.1:medium PASS 100% 96 in → 486 out (480 reasoning)
Status: Resolved
poolside/laguna-m.1:medium__json_schema PASS 100% 96 in → 623 out (608 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-m.1:none PASS 100% 94 in → 5 out (0 reasoning)
Status: Resolved
poolside/laguna-m.1:none__json_schema PASS 100% 95 in → 10 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
poolside/laguna-m.1:xhigh PASS 100% 94 in → 467 out (461 reasoning)
Status: Resolved
poolside/laguna-m.1:xhigh__json_schema PASS 100% 95 in → 1,193 out (1,183 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
poolside/laguna-xs-2.1:high PASS 100% 95 in → 574 out (570 reasoning)
Status: Resolved
poolside/laguna-xs-2.1:high__json_schema PASS 100% 96 in → 577 out (562 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-xs-2.1:low PASS 100% 94 in → 330 out (327 reasoning)
Status: Resolved
poolside/laguna-xs-2.1:low__json_schema PASS 100% 94 in → 766 out (752 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-xs-2.1:medium PASS 100% 94 in → 1,020 out (1,016 reasoning)
Status: Resolved
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 95 in → 536 out (521 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-xs-2.1:none__json_schema PASS 100% 96 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
poolside/laguna-xs-2.1:xhigh PASS 100% 96 in → 408 out (405 reasoning)
Status: Resolved
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 96 in → 357 out (341 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-xs.2:high PASS 100% 133 in → 464 out (458 reasoning)
Status: Resolved
poolside/laguna-xs.2:high__json_schema PASS 100% 133 in → 436 out (420 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
poolside/laguna-xs.2:low PASS 100% 132 in → 233 out (228 reasoning)
Status: Resolved
poolside/laguna-xs.2:low__json_schema PASS 100% 132 in → 372 out (361 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
poolside/laguna-xs.2:medium PASS 100% 132 in → 192 out (186 reasoning)
Status: Resolved
poolside/laguna-xs.2:medium__json_schema PASS 100% 133 in → 380 out (369 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
poolside/laguna-xs.2:xhigh PASS 100% 133 in → 348 out (342 reasoning)
Status: Resolved
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 132 in → 294 out (283 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
qwen/qwen3.6-27b:high PASS 100% 94 in → 1,119 out (1,172 reasoning)
Status: Resolved
qwen/qwen3.6-27b:low PASS 100% 92 in → 1,317 out (1,395 reasoning)
Status: Resolved
qwen/qwen3.6-27b:low__json_schema PASS 100% 94 in → 1,181 out (1,226 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-27b:medium PASS 100% 93 in → 1,142 out (1,133 reasoning)
Status: Resolved
qwen/qwen3.6-27b:none PASS 100% 92 in → 4 out (0 reasoning)
Status: Resolved
qwen/qwen3.6-27b:none__json_schema PASS 100% 94 in → 16 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-27b:xhigh PASS 100% 92 in → 803 out (794 reasoning)
Status: Resolved
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 95 in → 15 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-35b-a3b:high PASS 100% 93 in → 949 out (993 reasoning)
Status: Resolved
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 93 in → 1,343 out (1,426 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-35b-a3b:low PASS 100% 93 in → 1,025 out (1,091 reasoning)
Status: Resolved
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 94 in → 709 out (708 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 93 in → 869 out (892 reasoning)
Status: Resolved
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 94 in → 2,293 out (2,426 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-35b-a3b:none PASS 100% 93 in → 5 out (0 reasoning)
Status: Resolved
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 95 in → 12 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved\n"}
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 93 in → 1,363 out (1,451 reasoning)
Status: Resolved
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 94 in → 1,310 out (1,382 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
qwen/qwen3.6-flash:high PASS 100% 92 in → 534 out (525 reasoning)
Status: Resolved
qwen/qwen3.6-flash:low PASS 100% 94 in → 1,377 out (1,368 reasoning)
Status: Resolved
qwen/qwen3.6-flash:medium PASS 100% 94 in → 1,123 out (1,114 reasoning)
Status: Resolved
qwen/qwen3.6-flash:none PASS 100% 95 in → 4 out (0 reasoning)
Status: Resolved
qwen/qwen3.6-flash:xhigh PASS 100% 93 in → 1,115 out (1,106 reasoning)
Status: Resolved
qwen/qwen3.7-flash:high PASS 100% 92 in → 549 out (540 reasoning)
Status: Resolved
qwen/qwen3.7-flash:low PASS 100% 92 in → 1,464 out (1,455 reasoning)
Status: Resolved
qwen/qwen3.7-flash:max PASS 100% 93 in → 406 out (397 reasoning)
Status: Resolved
qwen/qwen3.7-flash:medium PASS 100% 94 in → 1,230 out (1,221 reasoning)
Status: Resolved
qwen/qwen3.7-flash:none PASS 100% 95 in → 4 out (0 reasoning)
Status: Resolved
qwen/qwen3.7-flash:xhigh PASS 100% 93 in → 982 out (973 reasoning)
Status: Resolved
qwen/qwen3.7-max:high PASS 100% 93 in → 194 out (185 reasoning)
Status: Resolved
qwen/qwen3.7-max:low PASS 100% 93 in → 392 out (383 reasoning)
Status: Resolved
qwen/qwen3.7-max:medium PASS 100% 93 in → 315 out (306 reasoning)
Status: Resolved
qwen/qwen3.7-max:none PASS 100% 96 in → 4 out (0 reasoning)
Status: Resolved
qwen/qwen3.7-max:xhigh PASS 100% 93 in → 270 out (261 reasoning)
Status: Resolved
qwen/qwen3.7-plus:high PASS 100% 91 in → 1,516 out (1,507 reasoning)
Status: Resolved
qwen/qwen3.7-plus:low PASS 100% 94 in → 525 out (516 reasoning)
Status: Resolved
qwen/qwen3.7-plus:medium PASS 100% 93 in → 616 out (607 reasoning)
Status: Resolved
qwen/qwen3.7-plus:none PASS 100% 95 in → 4 out (0 reasoning)
Status: Resolved
qwen/qwen3.7-plus:xhigh PASS 100% 92 in → 672 out (663 reasoning)
Status: Resolved
tencent/hy3:high PASS 100% 93 in → 1,162 out (1,156 reasoning)
Status: Resolved
tencent/hy3:high__json_schema PASS 100% 94 in → 1,014 out (1,102 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
tencent/hy3:low PASS 100% 94 in → 1,776 out (1,899 reasoning)
Status: Resolved
tencent/hy3:low__json_schema PASS 100% 93 in → 1,499 out (1,591 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
tencent/hy3:medium PASS 100% 93 in → 815 out (809 reasoning)
Status: Resolved
tencent/hy3:medium__json_schema PASS 100% 94 in → 837 out (889 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
tencent/hy3:none PASS 100% 98 in → 5 out (0 reasoning)
Status: Resolved
tencent/hy3:none__json_schema PASS 100% 96 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
tencent/hy3:xhigh PASS 100% 95 in → 1,460 out (1,454 reasoning)
Status: Resolved
tencent/hy3:xhigh__json_schema PASS 100% 93 in → 1,883 out (2,103 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
thinkingmachines/inkling-small:high PASS 100% 92 in → 529 out (519 reasoning)
Status: Resolved
thinkingmachines/inkling-small:high__json_schema PASS 100% 94 in → 230 out (212 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
thinkingmachines/inkling-small:low PASS 100% 93 in → 108 out (113 reasoning)
Status: Resolved
thinkingmachines/inkling-small:low__json_schema PASS 100% 92 in → 83 out (65 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
thinkingmachines/inkling-small:max PASS 100% 94 in → 539 out (529 reasoning)
Status: Resolved
thinkingmachines/inkling-small:max__json_schema PASS 100% 93 in → 654 out (637 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
thinkingmachines/inkling-small:medium PASS 100% 92 in → 209 out (239 reasoning)
Status: Resolved
thinkingmachines/inkling-small:medium__json_schema PASS 100% 93 in → 257 out (240 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
thinkingmachines/inkling-small:xhigh PASS 100% 92 in → 574 out (564 reasoning)
Status: Resolved
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 93 in → 330 out (311 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
thinkingmachines/inkling:high PASS 100% 92 in → 2,331 out (2,737 reasoning)
Status: Resolved
thinkingmachines/inkling:low PASS 100% 92 in → 182 out (172 reasoning)
Status: Resolved
thinkingmachines/inkling:max PASS 100% 91 in → 4,148 out (4,139 reasoning)
Status: Resolved
thinkingmachines/inkling:max__json_schema PASS 100% 92 in → 7,342 out (7,323 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
thinkingmachines/inkling:medium PASS 100% 91 in → 906 out (1,070 reasoning)
Status: Resolved
thinkingmachines/inkling:none PASS 100% 90 in → 7 out (0 reasoning)
Status: Resolved
thinkingmachines/inkling:xhigh PASS 100% 95 in → 2,504 out (2,495 reasoning)
Status: Resolved
x-ai/grok-4.3:high PASS 100% 260 in → 1,125 out (1,122 reasoning)
Status: Resolved
x-ai/grok-4.3:high__json_schema PASS 100% 316 in → 806 out (798 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
x-ai/grok-4.3:low PASS 100% 265 in → 710 out (707 reasoning)
Status: Resolved
x-ai/grok-4.3:low__json_schema PASS 100% 322 in → 697 out (688 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
x-ai/grok-4.3:max PASS 100% 259 in → 761 out (758 reasoning)
Status: Resolved
x-ai/grok-4.3:max__json_schema PASS 100% 316 in → 995 out (987 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
x-ai/grok-4.3:medium PASS 100% 269 in → 1,076 out (1,073 reasoning)
Status: Resolved
x-ai/grok-4.3:medium__json_schema PASS 100% 324 in → 875 out (867 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
x-ai/grok-4.3:none PASS 100% 257 in → 3 out (0 reasoning)
Status: Resolved
x-ai/grok-4.3:none__json_schema PASS 100% 313 in → 8 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
x-ai/grok-4.3:xhigh PASS 100% 262 in → 963 out (960 reasoning)
Status: Resolved
x-ai/grok-4.3:xhigh__json_schema PASS 100% 316 in → 788 out (780 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Resolved"}
x-ai/grok-4.5:high PASS 100% 283 in → 279 out (276 reasoning)
Status: Resolved
x-ai/grok-4.5:high__json_schema PASS 100% 352 in → 752 out (739 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
x-ai/grok-4.5:low PASS 100% 282 in → 298 out (295 reasoning)
Status: Resolved
x-ai/grok-4.5:low__json_schema PASS 100% 352 in → 283 out (268 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved\n" }
x-ai/grok-4.5:max PASS 100% 282 in → 395 out (392 reasoning)
Status: Resolved
x-ai/grok-4.5:max__json_schema PASS 100% 350 in → 729 out (716 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
x-ai/grok-4.5:medium PASS 100% 283 in → 510 out (507 reasoning)
Status: Resolved
x-ai/grok-4.5:medium__json_schema PASS 100% 351 in → 578 out (565 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
x-ai/grok-4.5:xhigh PASS 100% 282 in → 535 out (532 reasoning)
Status: Resolved
x-ai/grok-4.5:xhigh__json_schema PASS 100% 353 in → 311 out (301 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Resolved"}
z-ai/glm-4.7-flash:high PASS 100% 82 in → 366 out (373 reasoning)
Status: Resolved
z-ai/glm-4.7-flash:high__json_schema PASS 100% 82 in → 392 out (403 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
z-ai/glm-4.7-flash:low PASS 100% 84 in → 935 out (1,030 reasoning)
Status: Resolved
z-ai/glm-4.7-flash:low__json_schema PASS 100% 84 in → 567 out (585 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
z-ai/glm-4.7-flash:medium PASS 100% 83 in → 928 out (1,006 reasoning)
Status: Resolved
z-ai/glm-4.7-flash:none PASS 100% 84 in → 4 out (0 reasoning)
Status: Resolved
z-ai/glm-4.7-flash:none__json_schema PASS 100% 82 in → 14 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
z-ai/glm-4.7-flash:xhigh PASS 100% 82 in → 2,032 out (2,230 reasoning)
Status: Resolved
z-ai/glm-5.2:high PASS 100% 91 in → 276 out (305 reasoning)
Status: Resolved
z-ai/glm-5.2:high__json_schema PASS 100% 90 in → 730 out (788 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
z-ai/glm-5.2:low PASS 100% 90 in → 460 out (454 reasoning)
Status: Resolved
z-ai/glm-5.2:low__json_schema PASS 100% 91 in → 220 out (205 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
z-ai/glm-5.2:medium PASS 100% 92 in → 192 out (204 reasoning)
Status: Resolved
z-ai/glm-5.2:medium__json_schema PASS 100% 90 in → 361 out (347 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
z-ai/glm-5.2:none PASS 100% 83 in → 5 out (0 reasoning)
Status: Resolved
z-ai/glm-5.2:none__json_schema PASS 100% 86 in → 13 out (0 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
z-ai/glm-5.2:xhigh PASS 100% 90 in → 840 out (834 reasoning)
Status: Resolved
z-ai/glm-5.2:xhigh__json_schema PASS 100% 91 in → 13 out (12 reasoning)
Status: Resolved
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Resolved" }
anthropic/claude-fable-5:max FAIL 0% 154 in → 115 out (38 reasoning)
(empty output)
Failure: Content mismatch: ['status.txt']
anthropic/claude-fable-5:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
anthropic/claude-fable-5:xhigh FAIL 0% 153 in → 48 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.6:max FAIL 0% 103 in → 210 out (131 reasoning)
``` Status: Resolved ``` The cherry-picked commit intentionally changed `Status: OK` → `Status: Resolved`, representing a deliberate fix. The current branch's `Status: Failed` is the state that needs fixing. Accepting the incoming change (`Status: Resolved`) honors the purpose of the cherry-picked commit.
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.6:xhigh FAIL 0% 101 in → 228 out (150 reasoning)
``` Status: Resolved ``` The cherry-picked commit's intent was to update the status to "Resolved" (changing it from "OK" → "Resolved"). Since we're cherry-picking that fix, we accept the incoming change (`Status: Resolved`), which represents the desired fix being applied to the current branch.
Failure: Content mismatch: ['status.txt']
anthropic/claude-opus-4.8:max FAIL 0% 156 in → 574 out (137 reasoning)
``` Status: Resolved ``` **Reasoning:** The purpose of cherry-picking the "Fix status" commit is to apply its change. The base was `OK`, the current branch regressed it to `Failed`, and the incoming fix sets it to `Resolved`. Since the fix (`Resolved`) is the intent you're pulling in and it supersedes the `Failed` state, you keep the incoming side and discard both conflict markers.
Failure: Content mismatch: ['status.txt']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 85 in → 596 out (682 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
arcee-ai/trinity-mini:low FAIL 0% 86 in → 1,001 out (1,164 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
arcee-ai/trinity-mini:medium FAIL 0% 87 in → 1,204 out (1,352 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
arcee-ai/trinity-mini:xhigh FAIL 0% 86 in → 702 out (837 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 86 in → 4 out (0 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
inclusionai/ling-2.6-flash__json_schema FAIL 0% 108 in → 15 out
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
Failure: Content mismatch: ['status.txt']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: status.txt Status: Resolved
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:medium FAIL 0% 85 in → 279 out (346 reasoning)
Status: Resolved
Failure: Content mismatch: ['status.txt']
moonshotai/kimi-k3:xhigh FAIL 0%
(empty output)
Failure: OpenAI-compatible response contained provider error: Provider returned error
openai/gpt-oss-20b:medium__json_schema FAIL 0% 145 in → 11 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
Failure: Content mismatch: ['status.txt']
poolside/laguna-xs-2.1:none FAIL 0% 95 in → 33 out (0 reasoning)
Status: Resolved --- status.txt >>> HEAD Status: Failed ======= Status: Resolved >>> a39647b (Fix status)
Failure: Expected file heading before file content
poolside/laguna-xs.2:none FAIL 0% 131 in → 27 out (0 reasoning)
I'll resolve this cherry-pick conflict by choosing the incoming commit's changes since it represents the intended fix. Status: Resolved
Failure: Content mismatch: ['status.txt']
poolside/laguna-xs.2:none__json_schema FAIL 0% 132 in → 10 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
Failure: Content mismatch: ['status.txt']
qwen/qwen3.6-27b:high__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1aab4e03-9a54-98a0-9242-8994502156ea","request_id":"1aab4e03-9a54-98a0-9242-8994502156ea"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-84ae9974-721f-9258-a751-a65dc7765c28","request_id":"84ae9974-721f-9258-a751-a65dc7765c28"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-91efee51-6552-97d8-8886-f8266cd46b6b","request_id":"91efee51-6552-97d8-8886-f8266cd46b6b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1593ef42-8f90-9028-afe8-3af1b1411589","request_id":"1593ef42-8f90-9028-afe8-3af1b1411589"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-66b1f780-86cb-938f-9d20-09fac79637df","request_id":"66b1f780-86cb-938f-9d20-09fac79637df"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ee32e67b-b7df-95f1-b803-14abb9ba7296"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9f1068ed-2ee8-9998-83c3-2cab04adf0ea"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-bfa1dedf-0c8b-9de9-b560-16e775a753c8"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7ec95506-aef4-9e84-8a68-a248da017673"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3bf9c144-6f4e-9882-a19d-da4508656505"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-577792ca-a4cd-98f3-9bea-dc88593c93fd"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2e2cf0b1-48f2-9f15-8dce-5e46d969a4a4","request_id":"2e2cf0b1-48f2-9f15-8dce-5e46d969a4a4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-278c6a85-29b8-9dfc-a27c-2e17dcf67b90","request_id":"278c6a85-29b8-9dfc-a27c-2e17dcf67b90"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-46a768cf-d81b-92d2-8cc1-555c82034e1f","request_id":"46a768cf-d81b-92d2-8cc1-555c82034e1f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9e62a0a5-f282-9991-92ca-eef6e0ae3ad7","request_id":"9e62a0a5-f282-9991-92ca-eef6e0ae3ad7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c09572ca-4453-9ece-8d9e-ce049a934e0e","request_id":"c09572ca-4453-9ece-8d9e-ce049a934e0e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2731205a-f140-94df-bbd1-6530b5037f96","request_id":"2731205a-f140-94df-bbd1-6530b5037f96"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e69036d7-74c8-94b9-a11c-c69607587c99","request_id":"e69036d7-74c8-94b9-a11c-c69607587c99"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6724c416-59d5-92c2-98d8-0a878358f3ee","request_id":"6724c416-59d5-92c2-98d8-0a878358f3ee"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8f06987a-5c20-9cb2-8c76-e321c18d4f29","request_id":"8f06987a-5c20-9cb2-8c76-e321c18d4f29"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bb75ea17-03b4-96c8-a754-4cd48498b847","request_id":"bb75ea17-03b4-96c8-a754-4cd48498b847"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 93 in → 7 out (0 reasoning)
Status: Resolved<|end_message|>
Failure: Content mismatch: ['status.txt']
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved{ "resolved_content": "Status: Resolved" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved{ "resolved_content": "Status: Resolved" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Status: Resolved
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)