f003 — rebase

← rebase
Single-line status message conflict
Tests ability to resolve a status-message conflict during rebase. Evaluates choosing between contradictory values in rebase context.
easy rebase status-message conflict-resolution
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Status: OK' > status.txt
  5. 05 git add status.txt
  6. 06 git commit -m 'Initial status'
  7. 07 git checkout -b fix
  8. 08 echo 'Status: Resolved' > status.txt
  9. 09 git add status.txt
  10. 10 git commit -m 'Fix status'
  11. 11 git checkout main
  12. 12 echo 'Status: Failed' > status.txt
  13. 13 git add status.txt
  14. 14 git commit -m 'Update status'
  15. 15 git rebase fix
Prompt
Resolve the rebase conflict in status.txt. The base was 'Status: OK'. The upstream fix branch changed it to 'Status: Resolved'. The current branch being rebased (main) changed it to 'Status: Failed'. Keep the current branch value. Provide ONLY the resolved file content.
Expected
Status: Failed
Fixture Metrics

Scoped model quality, cost, API time, and token usage for rebase/f003.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 171 in → 63 out (52 reasoning)
Status: Failed
anthropic/claude-fable-5:high__json_schema PASS 100% 390 in → 46 out (26 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed\n"}
anthropic/claude-fable-5:low PASS 100% 171 in → 38 out (28 reasoning)
Status: Failed
anthropic/claude-fable-5:low__json_schema PASS 100% 391 in → 19 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
anthropic/claude-fable-5:max__json_schema PASS 100% 392 in → 124 out (58 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
anthropic/claude-fable-5:medium PASS 100% 174 in → 52 out (43 reasoning)
Status: Failed
anthropic/claude-fable-5:medium__json_schema PASS 100% 394 in → 36 out (17 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
anthropic/claude-fable-5:xhigh PASS 100% 169 in → 79 out (62 reasoning)
Status: Failed
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 391 in → 83 out (46 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed\n"}
anthropic/claude-haiku-4.5:high PASS 100% 145 in → 954 out (671 reasoning)
``` Status: Failed ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 312 in → 929 out (568 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
anthropic/claude-haiku-4.5:low PASS 100% 143 in → 867 out (632 reasoning)
``` Status: Failed ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 314 in → 259 out (238 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
anthropic/claude-haiku-4.5:medium PASS 100% 146 in → 423 out (389 reasoning)
``` Status: Failed ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 314 in → 219 out (201 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
anthropic/claude-haiku-4.5:none PASS 100% 115 in → 6 out (0 reasoning)
Status: Failed
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 283 in → 15 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed\n"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 144 in → 442 out (427 reasoning)
``` Status: Failed ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 314 in → 2,900 out (1,132 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.6:high PASS 100% 117 in → 6 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.6:high__json_schema PASS 100% 285 in → 14 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.6:low PASS 100% 116 in → 6 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.6:low__json_schema PASS 100% 283 in → 14 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.6:max PASS 100% 114 in → 220 out (190 reasoning)
Status: Failed
anthropic/claude-opus-4.6:max__json_schema PASS 100% 283 in → 214 out (177 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.6:medium PASS 100% 115 in → 6 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 282 in → 14 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.6:none PASS 100% 114 in → 6 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.6:none__json_schema PASS 100% 283 in → 14 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 116 in → 6 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 282 in → 14 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.7:high PASS 100% 175 in → 10 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.7:high__json_schema PASS 100% 397 in → 23 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.7:low PASS 100% 177 in → 10 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.7:low__json_schema PASS 100% 394 in → 23 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.7:max PASS 100% 177 in → 196 out (68 reasoning)
Status: Failed
anthropic/claude-opus-4.7:max__json_schema PASS 100% 396 in → 250 out (88 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.7:medium PASS 100% 177 in → 10 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 394 in → 23 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.7:none PASS 100% 175 in → 10 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.7:none__json_schema PASS 100% 395 in → 23 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 176 in → 10 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 395 in → 23 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.8:high PASS 100% 169 in → 89 out (57 reasoning)
Status: Failed
anthropic/claude-opus-4.8:high__json_schema PASS 100% 389 in → 74 out (45 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.8:low PASS 100% 169 in → 65 out (54 reasoning)
Status: Failed
anthropic/claude-opus-4.8:low__json_schema PASS 100% 390 in → 19 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-opus-4.8:max PASS 100% 172 in → 613 out (143 reasoning)
Status: Failed
anthropic/claude-opus-4.8:max__json_schema PASS 100% 390 in → 318 out (77 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed\n"}
anthropic/claude-opus-4.8:medium PASS 100% 173 in → 75 out (53 reasoning)
Status: Failed
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 392 in → 73 out (47 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-opus-4.8:none PASS 100% 171 in → 8 out (0 reasoning)
Status: Failed
anthropic/claude-opus-4.8:none__json_schema PASS 100% 391 in → 18 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-opus-4.8:xhigh PASS 100% 170 in → 266 out (116 reasoning)
Status: Failed
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 391 in → 111 out (44 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-sonnet-4.6:high PASS 100% 115 in → 126 out (103 reasoning)
Status: Failed
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 284 in → 12 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-sonnet-4.6:low PASS 100% 117 in → 6 out (0 reasoning)
Status: Failed
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 285 in → 12 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-sonnet-4.6:max PASS 100% 117 in → 127 out (114 reasoning)
Status: Failed
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 284 in → 166 out (139 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 117 in → 6 out (0 reasoning)
Status: Failed
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 283 in → 12 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-sonnet-4.6:none PASS 100% 115 in → 6 out (0 reasoning)
Status: Failed
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 282 in → 12 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 115 in → 109 out (92 reasoning)
Status: Failed
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 284 in → 12 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-sonnet-5:high PASS 100% 174 in → 111 out (72 reasoning)
Status: Failed
anthropic/claude-sonnet-5:high__json_schema PASS 100% 392 in → 84 out (58 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
anthropic/claude-sonnet-5:low PASS 100% 170 in → 54 out (45 reasoning)
Status: Failed
anthropic/claude-sonnet-5:low__json_schema PASS 100% 392 in → 62 out (49 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-sonnet-5:medium PASS 100% 171 in → 169 out (76 reasoning)
Status: Failed
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 391 in → 65 out (47 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-sonnet-5:none PASS 100% 173 in → 8 out (0 reasoning)
Status: Failed
anthropic/claude-sonnet-5:none__json_schema PASS 100% 389 in → 18 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
anthropic/claude-sonnet-5:xhigh PASS 100% 171 in → 341 out (127 reasoning)
Status: Failed
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 390 in → 120 out (64 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
arcee-ai/trinity-large-thinking:high PASS 100% 100 in → 1,276 out (1,311 reasoning)
Status: Failed
arcee-ai/trinity-large-thinking:low PASS 100% 99 in → 314 out (312 reasoning)
Status: Failed
arcee-ai/trinity-large-thinking:medium PASS 100% 99 in → 2,101 out (2,159 reasoning)
Status: Failed
arcee-ai/trinity-large-thinking:xhigh PASS 100% 101 in → 1,089 out (1,110 reasoning)
Status: Failed
arcee-ai/trinity-mini:high PASS 100% 101 in → 435 out (496 reasoning)
Status: Failed
arcee-ai/trinity-mini:high__json_schema PASS 100% 101 in → 537 out (608 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
arcee-ai/trinity-mini:low PASS 100% 100 in → 586 out (655 reasoning)
Status: Failed
arcee-ai/trinity-mini:low__json_schema PASS 100% 100 in → 437 out (499 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
arcee-ai/trinity-mini:medium PASS 100% 99 in → 360 out (417 reasoning)
Status: Failed
arcee-ai/trinity-mini:medium__json_schema PASS 100% 99 in → 357 out (401 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
arcee-ai/trinity-mini:xhigh PASS 100% 100 in → 367 out (416 reasoning)
Status: Failed
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 102 in → 348 out (380 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content":"Status: Failed" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 101 in → 159 out (176 reasoning)
Status: Failed
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 248 in → 95 out (94 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 101 in → 266 out (284 reasoning)
Status: Failed
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 104 in → 131 out (133 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed\n"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 181 in → 226 out (221 reasoning)
Status: Failed
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 180 in → 195 out (204 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 102 in → 250 out (265 reasoning)
Status: Failed
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 108 in → 737 out (797 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 101 in → 4 out (0 reasoning)
Status: Failed
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 102 in → 14 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 180 in → 168 out (186 reasoning)
Status: Failed
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 103 in → 300 out (312 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
deepseek/deepseek-v4-flash:high PASS 100% 102 in → 113 out (109 reasoning)
Status: Failed
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 171 in → 90 out (87 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed\n"}
deepseek/deepseek-v4-flash:low PASS 100% 103 in → 96 out (91 reasoning)
Status: Failed
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 198 in → 12 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed\n"}
deepseek/deepseek-v4-flash:medium PASS 100% 100 in → 257 out (277 reasoning)
Status: Failed
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 102 in → 154 out (148 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
deepseek/deepseek-v4-flash:none PASS 100% 100 in → 4 out (0 reasoning)
Status: Failed
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 101 in → 18 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 182 in → 237 out (258 reasoning)
Status: Failed
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 183 in → 146 out (129 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
deepseek/deepseek-v4-pro:high PASS 100% 103 in → 240 out (235 reasoning)
Status: Failed
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 103 in → 14 out (13 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
deepseek/deepseek-v4-pro:low PASS 100% 104 in → 93 out (89 reasoning)
Status: Failed
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 104 in → 261 out (245 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
deepseek/deepseek-v4-pro:medium PASS 100% 102 in → 175 out (170 reasoning)
Status: Failed
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 105 in → 167 out (151 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
deepseek/deepseek-v4-pro:none PASS 100% 102 in → 3 out (0 reasoning)
Status: Failed
deepseek/deepseek-v4-pro:xhigh PASS 100% 101 in → 4 out (0 reasoning)
Status: Failed
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 180 in → 227 out (237 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
google/gemini-3-flash-preview:high PASS 100% 100 in → 304 out (301 reasoning)
Status: Failed
google/gemini-3-flash-preview:high__json_schema PASS 100% 159 in → 942 out (933 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemini-3-flash-preview:low PASS 100% 98 in → 269 out (266 reasoning)
Status: Failed
google/gemini-3-flash-preview:low__json_schema PASS 100% 159 in → 455 out (446 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemini-3-flash-preview:medium PASS 100% 100 in → 247 out (244 reasoning)
Status: Failed
google/gemini-3-flash-preview:medium__json_schema PASS 100% 156 in → 429 out (419 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
google/gemini-3-flash-preview:xhigh PASS 100% 100 in → 298 out (295 reasoning)
Status: Failed
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 159 in → 842 out (832 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 101 in → 449 out (446 reasoning)
Status: Failed
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 99 in → 786 out (771 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
google/gemini-3.1-flash-lite-preview:low PASS 100% 100 in → 124 out (121 reasoning)
Status: Failed
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 99 in → 124 out (110 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 97 in → 166 out (163 reasoning)
Status: Failed
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 101 in → 200 out (190 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 100 in → 430 out (427 reasoning)
Status: Failed
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 101 in → 494 out (479 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
google/gemini-3.1-pro-preview:high PASS 100% 97 in → 284 out (281 reasoning)
Status: Failed
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 159 in → 1,079 out (1,070 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemini-3.1-pro-preview:low PASS 100% 99 in → 239 out (236 reasoning)
Status: Failed
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 159 in → 157 out (140 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
google/gemini-3.1-pro-preview:medium PASS 100% 99 in → 352 out (349 reasoning)
Status: Failed
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 98 in → 157 out (140 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
google/gemini-3.1-pro-preview:xhigh PASS 100% 99 in → 378 out (375 reasoning)
Status: Failed
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 159 in → 120 out (110 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
google/gemini-3.5-flash-lite:high PASS 100% 99 in → 509 out (506 reasoning)
Status: Failed
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 159 in → 492 out (482 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
google/gemini-3.5-flash-lite:low PASS 100% 101 in → 3 out (0 reasoning)
Status: Failed
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 158 in → 10 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
google/gemini-3.5-flash-lite:max PASS 100% 101 in → 600 out (597 reasoning)
Status: Failed
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 100 in → 580 out (571 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemini-3.5-flash-lite:medium PASS 100% 99 in → 431 out (428 reasoning)
Status: Failed
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 157 in → 358 out (348 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 101 in → 550 out (547 reasoning)
Status: Failed
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 99 in → 687 out (678 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemini-3.5-flash:high PASS 100% 99 in → 344 out (341 reasoning)
Status: Failed
google/gemini-3.5-flash:high__json_schema PASS 100% 159 in → 404 out (395 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemini-3.5-flash:low PASS 100% 99 in → 335 out (332 reasoning)
Status: Failed
google/gemini-3.5-flash:low__json_schema PASS 100% 159 in → 133 out (123 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
google/gemini-3.5-flash:medium PASS 100% 100 in → 463 out (460 reasoning)
Status: Failed
google/gemini-3.5-flash:medium__json_schema PASS 100% 158 in → 418 out (406 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed\n"}
google/gemini-3.5-flash:xhigh PASS 100% 99 in → 420 out (417 reasoning)
Status: Failed
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 160 in → 426 out (417 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemini-3.6-flash:high PASS 100% 99 in → 439 out (436 reasoning)
Status: Failed
google/gemini-3.6-flash:high__json_schema PASS 100% 160 in → 586 out (577 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemini-3.6-flash:low PASS 100% 100 in → 165 out (162 reasoning)
Status: Failed
google/gemini-3.6-flash:low__json_schema PASS 100% 159 in → 116 out (107 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemini-3.6-flash:max PASS 100% 99 in → 456 out (453 reasoning)
Status: Failed
google/gemini-3.6-flash:max__json_schema PASS 100% 160 in → 772 out (761 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
google/gemini-3.6-flash:medium PASS 100% 101 in → 412 out (409 reasoning)
Status: Failed
google/gemini-3.6-flash:medium__json_schema PASS 100% 160 in → 606 out (597 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemini-3.6-flash:xhigh PASS 100% 101 in → 437 out (434 reasoning)
Status: Failed
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 100 in → 1,683 out (1,672 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
google/gemma-4-26b-a4b-it:high PASS 100% 116 in → 417 out (403 reasoning)
Status: Failed
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 116 in → 494 out (465 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
google/gemma-4-26b-a4b-it:low PASS 100% 113 in → 333 out (323 reasoning)
Status: Failed
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 116 in → 291 out (271 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
google/gemma-4-26b-a4b-it:medium PASS 100% 116 in → 702 out (683 reasoning)
Status: Failed
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 116 in → 317 out (288 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
google/gemma-4-26b-a4b-it:none PASS 100% 112 in → 4 out (0 reasoning)
Status: Failed
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 113 in → 15 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 116 in → 381 out (383 reasoning)
Status: Failed
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 114 in → 616 out (585 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
google/gemma-4-31b-it:high PASS 100% 116 in → 256 out (255 reasoning)
Status: Failed
google/gemma-4-31b-it:high__json_schema PASS 100% 119 in → 187 out (176 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemma-4-31b-it:low PASS 100% 116 in → 171 out (147 reasoning)
Status: Failed
google/gemma-4-31b-it:low__json_schema PASS 100% 118 in → 179 out (168 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemma-4-31b-it:medium PASS 100% 112 in → 143 out (130 reasoning)
Status: Failed
google/gemma-4-31b-it:medium__json_schema PASS 100% 118 in → 228 out (217 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
google/gemma-4-31b-it:none PASS 100% 112 in → 4 out (0 reasoning)
Status: Failed
google/gemma-4-31b-it:none__json_schema PASS 100% 113 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
google/gemma-4-31b-it:xhigh PASS 100% 113 in → 221 out (212 reasoning)
Status: Failed
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 114 in → 243 out (218 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
ibm-granite/granite-4.1-8b__json_schema PASS 100% 105 in → 14 out
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
inclusionai/ling-2.6-flash PASS 100% 120 in → 4 out
Status: Failed
inclusionai/ling-2.6-flash__json_schema PASS 100% 122 in → 18 out
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
minimax/minimax-m2.5:high PASS 100% 134 in → 104 out (118 reasoning)
Status: Failed
minimax/minimax-m2.5:high__json_schema PASS 100% 134 in → 98 out (101 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
minimax/minimax-m2.5:low PASS 100% 134 in → 401 out (444 reasoning)
Status: Failed
minimax/minimax-m2.5:low__json_schema PASS 100% 133 in → 116 out (118 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
minimax/minimax-m2.5:medium PASS 100% 134 in → 91 out (98 reasoning)
Status: Failed
minimax/minimax-m2.5:medium__json_schema PASS 100% 140 in → 178 out (177 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
minimax/minimax-m2.5:xhigh PASS 100% 133 in → 147 out (142 reasoning)
Status: Failed
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 132 in → 177 out (165 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
minimax/minimax-m2.7:high PASS 100% 134 in → 220 out (215 reasoning)
Status: Failed
minimax/minimax-m2.7:high__json_schema PASS 100% 131 in → 217 out (236 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
minimax/minimax-m2.7:low PASS 100% 134 in → 232 out (228 reasoning)
Status: Failed
minimax/minimax-m2.7:medium PASS 100% 134 in → 206 out (248 reasoning)
Status: Failed
minimax/minimax-m2.7:medium__json_schema PASS 100% 267 in → 227 out (217 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
minimax/minimax-m2.7:xhigh PASS 100% 132 in → 230 out (224 reasoning)
Status: Failed
minimax/minimax-m3:high PASS 100% 219 in → 88 out (83 reasoning)
Status: Failed
minimax/minimax-m3:high__json_schema PASS 100% 257 in → 11 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed\n"}
minimax/minimax-m3:low PASS 100% 272 in → 211 out (227 reasoning)
Status: Failed
minimax/minimax-m3:medium PASS 100% 272 in → 152 out (159 reasoning)
Status: Failed
minimax/minimax-m3:medium__json_schema PASS 100% 257 in → 13 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
minimax/minimax-m3:xhigh PASS 100% 271 in → 118 out (127 reasoning)
Status: Failed
minimax/minimax-m3:xhigh__json_schema PASS 100% 260 in → 14 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
mistralai/devstral-2512 PASS 100% 103 in → 4 out
Status: Failed
mistralai/devstral-2512__json_schema PASS 100% 103 in → 11 out
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
mistralai/mistral-medium-3-5:high PASS 100% 115 in → 219 out (222 reasoning)
Status: Failed
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 104 in → 358 out (403 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
mistralai/mistral-medium-3-5:low PASS 100% 116 in → 382 out (412 reasoning)
Status: Failed
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 103 in → 241 out (249 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
mistralai/mistral-medium-3-5:medium PASS 100% 113 in → 278 out (302 reasoning)
Status: Failed
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 103 in → 207 out (223 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
mistralai/mistral-medium-3-5:none PASS 100% 116 in → 4 out (0 reasoning)
Status: Failed
mistralai/mistral-medium-3-5:xhigh PASS 100% 115 in → 255 out (271 reasoning)
Status: Failed
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 103 in → 290 out (296 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
moonshotai/kimi-k2.6:high PASS 100% 98 in → 430 out (425 reasoning)
Status: Failed
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 283 in → 565 out (554 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 101 in → 537 out (629 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
moonshotai/kimi-k2.7-code:high PASS 100% 97 in → 106 out (101 reasoning)
Status: Failed
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 184 in → 210 out (198 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed\n"}
moonshotai/kimi-k2.7-code:low PASS 100% 101 in → 215 out (210 reasoning)
Status: Failed
moonshotai/kimi-k2.7-code:medium PASS 100% 99 in → 135 out (130 reasoning)
Status: Failed
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 99 in → 159 out (166 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 100 in → 212 out (207 reasoning)
Status: Failed
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 184 in → 76 out (66 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
moonshotai/kimi-k3:high PASS 100% 175 in → 126 out (108 reasoning)
Status: Failed
moonshotai/kimi-k3:high__json_schema PASS 100% 177 in → 158 out (140 reasoning)
Status: Failed
moonshotai/kimi-k3:low PASS 100% 177 in → 147 out (129 reasoning)
Status: Failed
moonshotai/kimi-k3:low__json_schema PASS 100% 178 in → 271 out (253 reasoning)
Status: Failed
moonshotai/kimi-k3:max PASS 100% 176 in → 233 out (212 reasoning)
``` Status: Failed ```
moonshotai/kimi-k3:max__json_schema PASS 100% 178 in → 282 out (264 reasoning)
Status: Failed
moonshotai/kimi-k3:medium PASS 100% 176 in → 195 out (177 reasoning)
Status: Failed
moonshotai/kimi-k3:medium__json_schema PASS 100% 179 in → 274 out (256 reasoning)
Status: Failed
moonshotai/kimi-k3:xhigh PASS 100% 177 in → 168 out (150 reasoning)
Status: Failed
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 176 in → 162 out (144 reasoning)
Status: Failed
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 117 in → 85 out (93 reasoning)
Status: Failed
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 118 in → 121 out (123 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 118 in → 167 out (183 reasoning)
Status: Failed
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 118 in → 58 out (59 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 118 in → 97 out (112 reasoning)
Status: Failed
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 117 in → 102 out (114 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 116 in → 4 out (0 reasoning)
Status: Failed
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 118 in → 14 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 117 in → 48 out (53 reasoning)
Status: Failed
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 118 in → 137 out (144 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 118 in → 160 out (153 reasoning)
Status: Failed
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 115 in → 90 out (75 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 116 in → 314 out (295 reasoning)
Status: Failed
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 117 in → 141 out (126 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 117 in → 58 out (61 reasoning)
Status: Failed
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 116 in → 54 out (39 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 114 in → 4 out (0 reasoning)
Status: Failed
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 117 in → 14 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 117 in → 78 out (87 reasoning)
Status: Failed
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 117 in → 82 out (67 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
openai/gpt-5.4-mini:high PASS 100% 98 in → 142 out (133 reasoning)
Status: Failed
openai/gpt-5.4-mini:high__json_schema PASS 100% 133 in → 122 out (104 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4-mini:low PASS 100% 99 in → 27 out (18 reasoning)
Status: Failed
openai/gpt-5.4-mini:low__json_schema PASS 100% 135 in → 38 out (20 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4-mini:medium PASS 100% 98 in → 42 out (33 reasoning)
Status: Failed
openai/gpt-5.4-mini:medium__json_schema PASS 100% 133 in → 101 out (83 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4-mini:none PASS 100% 97 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.4-mini:none__json_schema PASS 100% 134 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4-mini:xhigh PASS 100% 100 in → 85 out (76 reasoning)
Status: Failed
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 135 in → 416 out (398 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4-nano:high PASS 100% 99 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.4-nano:high__json_schema PASS 100% 134 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4-nano:low PASS 100% 98 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.4-nano:low__json_schema PASS 100% 134 in → 17 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
openai/gpt-5.4-nano:medium PASS 100% 99 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.4-nano:medium__json_schema PASS 100% 135 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4-nano:none PASS 100% 97 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.4-nano:none__json_schema PASS 100% 137 in → 17 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 98 in → 99 out (90 reasoning)
Status: Failed
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 135 in → 107 out (88 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
openai/gpt-5.4:high PASS 100% 98 in → 79 out (70 reasoning)
Status: Failed
openai/gpt-5.4:high__json_schema PASS 100% 134 in → 107 out (89 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4:low PASS 100% 100 in → 18 out (9 reasoning)
Status: Failed
openai/gpt-5.4:low__json_schema PASS 100% 134 in → 27 out (9 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4:medium PASS 100% 98 in → 66 out (57 reasoning)
Status: Failed
openai/gpt-5.4:medium__json_schema PASS 100% 133 in → 82 out (64 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4:none PASS 100% 97 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.4:none__json_schema PASS 100% 133 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.4:xhigh PASS 100% 97 in → 93 out (84 reasoning)
Status: Failed
openai/gpt-5.4:xhigh__json_schema PASS 100% 134 in → 443 out (425 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.5:high PASS 100% 98 in → 92 out (83 reasoning)
Status: Failed
openai/gpt-5.5:high__json_schema PASS 100% 132 in → 196 out (178 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.5:low PASS 100% 97 in → 49 out (40 reasoning)
Status: Failed
openai/gpt-5.5:low__json_schema PASS 100% 134 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.5:medium PASS 100% 98 in → 52 out (43 reasoning)
Status: Failed
openai/gpt-5.5:medium__json_schema PASS 100% 133 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.5:none PASS 100% 96 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.5:none__json_schema PASS 100% 133 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.5:xhigh PASS 100% 97 in → 149 out (140 reasoning)
Status: Failed
openai/gpt-5.5:xhigh__json_schema PASS 100% 135 in → 534 out (516 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-luna:high PASS 100% 100 in → 28 out (19 reasoning)
Status: Failed
openai/gpt-5.6-luna:high__json_schema PASS 100% 134 in → 137 out (119 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-luna:low PASS 100% 98 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-luna:low__json_schema PASS 100% 134 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-luna:max PASS 100% 97 in → 37 out (28 reasoning)
Status: Failed
openai/gpt-5.6-luna:max__json_schema PASS 100% 134 in → 270 out (252 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-luna:medium PASS 100% 98 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-luna:medium__json_schema PASS 100% 136 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-luna:none PASS 100% 97 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-luna:none__json_schema PASS 100% 135 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-luna:xhigh PASS 100% 100 in → 35 out (26 reasoning)
Status: Failed
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 133 in → 117 out (99 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-sol:high PASS 100% 98 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-sol:high__json_schema PASS 100% 134 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-sol:low PASS 100% 97 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-sol:low__json_schema PASS 100% 133 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-sol:max PASS 100% 98 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-sol:max__json_schema PASS 100% 134 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-sol:medium PASS 100% 98 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-sol:medium__json_schema PASS 100% 133 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-sol:none PASS 100% 97 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-sol:none__json_schema PASS 100% 134 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-sol:xhigh PASS 100% 97 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 135 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-terra:high PASS 100% 99 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-terra:high__json_schema PASS 100% 135 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-terra:low PASS 100% 99 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-terra:low__json_schema PASS 100% 135 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-terra:max PASS 100% 99 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-terra:max__json_schema PASS 100% 134 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-terra:medium PASS 100% 98 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-terra:medium__json_schema PASS 100% 135 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-terra:none PASS 100% 98 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-terra:none__json_schema PASS 100% 134 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-5.6-terra:xhigh PASS 100% 98 in → 7 out (0 reasoning)
Status: Failed
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 134 in → 16 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-oss-120b:high PASS 100% 158 in → 127 out (131 reasoning)
Status: Failed
openai/gpt-oss-120b:high__json_schema PASS 100% 144 in → 778 out (896 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
openai/gpt-oss-120b:low PASS 100% 170 in → 30 out (0 reasoning)
Status: Failed
openai/gpt-oss-120b:low__json_schema PASS 100% 159 in → 99 out (86 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
openai/gpt-oss-120b:medium PASS 100% 159 in → 119 out (128 reasoning)
Status: Failed
openai/gpt-oss-120b:medium__json_schema PASS 100% 159 in → 104 out (100 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed\n"}
openai/gpt-oss-120b:xhigh PASS 100% 159 in → 395 out (465 reasoning)
Status: Failed
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 146 in → 184 out (192 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
openai/gpt-oss-20b:high PASS 100% 161 in → 234 out (260 reasoning)
Status: Failed
openai/gpt-oss-20b:high__json_schema PASS 100% 159 in → 206 out (188 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-oss-20b:low PASS 100% 158 in → 31 out (18 reasoning)
Status: Failed
openai/gpt-oss-20b:low__json_schema PASS 100% 156 in → 39 out (25 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-oss-20b:medium PASS 100% 158 in → 161 out (164 reasoning)
Status: Failed
openai/gpt-oss-20b:medium__json_schema PASS 100% 146 in → 173 out (188 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
openai/gpt-oss-20b:xhigh PASS 100% 159 in → 389 out (378 reasoning)
Status: Failed
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 158 in → 369 out (423 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
poolside/laguna-m.1:high PASS 100% 110 in → 1,855 out (1,857 reasoning)
Status: Failed
poolside/laguna-m.1:high__json_schema PASS 100% 106 in → 378 out (369 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
poolside/laguna-m.1:low PASS 100% 109 in → 318 out (313 reasoning)
Status: Failed
poolside/laguna-m.1:low__json_schema PASS 100% 109 in → 257 out (242 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
poolside/laguna-m.1:medium PASS 100% 110 in → 251 out (245 reasoning)
Status: Failed
poolside/laguna-m.1:medium__json_schema PASS 100% 109 in → 252 out (238 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
poolside/laguna-m.1:none PASS 100% 108 in → 5 out (0 reasoning)
Status: Failed
poolside/laguna-m.1:xhigh PASS 100% 110 in → 266 out (261 reasoning)
Status: Failed
poolside/laguna-m.1:xhigh__json_schema PASS 100% 108 in → 525 out (515 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
poolside/laguna-xs-2.1:high PASS 100% 109 in → 251 out (249 reasoning)
Status: Failed
poolside/laguna-xs-2.1:high__json_schema PASS 100% 110 in → 1,262 out (1,250 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
poolside/laguna-xs-2.1:low PASS 100% 110 in → 201 out (197 reasoning)
Status: Failed
poolside/laguna-xs-2.1:low__json_schema PASS 100% 110 in → 233 out (218 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
poolside/laguna-xs-2.1:medium PASS 100% 109 in → 2,319 out (2,319 reasoning)
Status: Failed
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 110 in → 401 out (386 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
poolside/laguna-xs-2.1:none PASS 100% 110 in → 3 out (0 reasoning)
Status: Failed
poolside/laguna-xs-2.1:none__json_schema PASS 100% 107 in → 15 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
poolside/laguna-xs-2.1:xhigh PASS 100% 110 in → 202 out (198 reasoning)
Status: Failed
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 109 in → 1,737 out (1,730 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
poolside/laguna-xs.2:high PASS 100% 147 in → 440 out (437 reasoning)
Status: Failed
poolside/laguna-xs.2:high__json_schema PASS 100% 147 in → 164 out (153 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
poolside/laguna-xs.2:low PASS 100% 147 in → 325 out (320 reasoning)
Status: Failed
poolside/laguna-xs.2:low__json_schema PASS 100% 146 in → 187 out (176 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
poolside/laguna-xs.2:medium PASS 100% 146 in → 351 out (346 reasoning)
Status: Failed
poolside/laguna-xs.2:medium__json_schema PASS 100% 145 in → 368 out (357 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
poolside/laguna-xs.2:none PASS 100% 144 in → 9 out (0 reasoning)
``` Status: Failed ```
poolside/laguna-xs.2:none__json_schema PASS 100% 147 in → 10 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
poolside/laguna-xs.2:xhigh PASS 100% 146 in → 766 out (762 reasoning)
Status: Failed
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 145 in → 521 out (512 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
qwen/qwen3.6-27b:high PASS 100% 107 in → 1,242 out (1,277 reasoning)
Status: Failed
qwen/qwen3.6-27b:high__json_schema PASS 100% 108 in → 987 out (986 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
qwen/qwen3.6-27b:low PASS 100% 108 in → 1,299 out (1,300 reasoning)
Status: Failed
qwen/qwen3.6-27b:low__json_schema PASS 100% 105 in → 1,042 out (1,045 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
qwen/qwen3.6-27b:medium PASS 100% 108 in → 1,006 out (998 reasoning)
Status: Failed
qwen/qwen3.6-27b:medium__json_schema PASS 100% 106 in → 15 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
qwen/qwen3.6-27b:none PASS 100% 107 in → 4 out (0 reasoning)
Status: Failed
qwen/qwen3.6-27b:xhigh PASS 100% 107 in → 1,259 out (1,276 reasoning)
Status: Failed
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 108 in → 1,191 out (1 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
qwen/qwen3.6-35b-a3b:high PASS 100% 106 in → 861 out (885 reasoning)
Status: Failed
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 106 in → 686 out (715 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
qwen/qwen3.6-35b-a3b:low PASS 100% 108 in → 1,159 out (1,222 reasoning)
Status: Failed
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 108 in → 1,280 out (1,331 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 108 in → 921 out (957 reasoning)
Status: Failed
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 107 in → 708 out (734 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
qwen/qwen3.6-35b-a3b:none PASS 100% 108 in → 4 out (0 reasoning)
Status: Failed
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 106 in → 1,155 out (1,194 reasoning)
Status: Failed
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 108 in → 1,135 out (1,163 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
qwen/qwen3.6-flash:high PASS 100% 107 in → 1,132 out (1,124 reasoning)
Status: Failed
qwen/qwen3.6-flash:low PASS 100% 106 in → 1,137 out (1,129 reasoning)
Status: Failed
qwen/qwen3.6-flash:medium PASS 100% 106 in → 1,304 out (1,296 reasoning)
Status: Failed
qwen/qwen3.6-flash:none PASS 100% 109 in → 3 out (0 reasoning)
Status: Failed
qwen/qwen3.6-flash:xhigh PASS 100% 108 in → 964 out (956 reasoning)
Status: Failed
qwen/qwen3.7-flash:high PASS 100% 107 in → 1,066 out (1,058 reasoning)
Status: Failed
qwen/qwen3.7-flash:low PASS 100% 108 in → 1,085 out (1,077 reasoning)
Status: Failed
qwen/qwen3.7-flash:max PASS 100% 105 in → 1,072 out (1,064 reasoning)
Status: Failed
qwen/qwen3.7-flash:medium PASS 100% 106 in → 1,099 out (1,091 reasoning)
Status: Failed
qwen/qwen3.7-flash:none PASS 100% 109 in → 3 out (0 reasoning)
Status: Failed
qwen/qwen3.7-flash:xhigh PASS 100% 107 in → 957 out (949 reasoning)
Status: Failed
qwen/qwen3.7-max:high PASS 100% 107 in → 325 out (317 reasoning)
Status: Failed
qwen/qwen3.7-max:low PASS 100% 108 in → 334 out (326 reasoning)
Status: Failed
qwen/qwen3.7-max:medium PASS 100% 107 in → 321 out (313 reasoning)
Status: Failed
qwen/qwen3.7-max:none PASS 100% 109 in → 3 out (0 reasoning)
Status: Failed
qwen/qwen3.7-max:xhigh PASS 100% 107 in → 281 out (273 reasoning)
Status: Failed
qwen/qwen3.7-plus:high PASS 100% 108 in → 483 out (475 reasoning)
Status: Failed
qwen/qwen3.7-plus:low PASS 100% 108 in → 152 out (144 reasoning)
Status: Failed
qwen/qwen3.7-plus:medium PASS 100% 107 in → 583 out (575 reasoning)
Status: Failed
qwen/qwen3.7-plus:none PASS 100% 109 in → 3 out (0 reasoning)
Status: Failed
qwen/qwen3.7-plus:xhigh PASS 100% 106 in → 367 out (359 reasoning)
Status: Failed
tencent/hy3:high PASS 100% 108 in → 695 out (690 reasoning)
Status: Failed
tencent/hy3:high__json_schema PASS 100% 107 in → 535 out (561 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
tencent/hy3:low PASS 100% 107 in → 435 out (466 reasoning)
Status: Failed
tencent/hy3:low__json_schema PASS 100% 108 in → 486 out (505 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
tencent/hy3:medium PASS 100% 106 in → 502 out (497 reasoning)
Status: Failed
tencent/hy3:medium__json_schema PASS 100% 110 in → 388 out (387 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
tencent/hy3:none PASS 100% 112 in → 4 out (0 reasoning)
Status: Failed
tencent/hy3:none__json_schema PASS 100% 111 in → 12 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
tencent/hy3:xhigh PASS 100% 110 in → 748 out (743 reasoning)
Status: Failed
tencent/hy3:xhigh__json_schema PASS 100% 107 in → 510 out (548 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
thinkingmachines/inkling-small:high PASS 100% 107 in → 270 out (261 reasoning)
Status: Failed
thinkingmachines/inkling-small:high__json_schema PASS 100% 108 in → 491 out (474 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
thinkingmachines/inkling-small:low PASS 100% 107 in → 58 out (59 reasoning)
Status: Failed
thinkingmachines/inkling-small:low__json_schema PASS 100% 108 in → 36 out (18 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
thinkingmachines/inkling-small:max PASS 100% 107 in → 406 out (453 reasoning)
Status: Failed
thinkingmachines/inkling-small:max__json_schema PASS 100% 109 in → 270 out (252 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
thinkingmachines/inkling-small:medium PASS 100% 110 in → 221 out (233 reasoning)
Status: Failed
thinkingmachines/inkling-small:medium__json_schema PASS 100% 106 in → 96 out (78 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
thinkingmachines/inkling-small:xhigh PASS 100% 109 in → 534 out (591 reasoning)
Status: Failed
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 106 in → 397 out (383 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
thinkingmachines/inkling:high PASS 100% 107 in → 99 out (89 reasoning)
Status: Failed
thinkingmachines/inkling:high__json_schema PASS 100% 106 in → 166 out (149 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
thinkingmachines/inkling:low PASS 100% 109 in → 29 out (20 reasoning)
Status: Failed
thinkingmachines/inkling:low__json_schema PASS 100% 106 in → 28 out (15 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
thinkingmachines/inkling:max PASS 100% 106 in → 385 out (376 reasoning)
Status: Failed
thinkingmachines/inkling:medium PASS 100% 108 in → 94 out (99 reasoning)
Status: Failed
thinkingmachines/inkling:medium__json_schema PASS 100% 108 in → 50 out (33 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
thinkingmachines/inkling:none PASS 100% 105 in → 6 out (0 reasoning)
Status: Failed
thinkingmachines/inkling:none__json_schema PASS 100% 105 in → 13 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
thinkingmachines/inkling:xhigh PASS 100% 107 in → 95 out (101 reasoning)
Status: Failed
thinkingmachines/inkling:xhigh__json_schema PASS 100% 106 in → 283 out (265 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
x-ai/grok-4.3:high PASS 100% 275 in → 620 out (617 reasoning)
Status: Failed
x-ai/grok-4.3:high__json_schema PASS 100% 331 in → 724 out (712 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
x-ai/grok-4.3:low PASS 100% 281 in → 348 out (345 reasoning)
Status: Failed
x-ai/grok-4.3:low__json_schema PASS 100% 340 in → 600 out (588 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
x-ai/grok-4.3:max PASS 100% 274 in → 635 out (632 reasoning)
Status: Failed
x-ai/grok-4.3:max__json_schema PASS 100% 332 in → 541 out (533 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
x-ai/grok-4.3:medium PASS 100% 280 in → 388 out (385 reasoning)
Status: Failed
x-ai/grok-4.3:medium__json_schema PASS 100% 337 in → 478 out (466 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
x-ai/grok-4.3:none PASS 100% 272 in → 3 out (0 reasoning)
Status: Failed
x-ai/grok-4.3:none__json_schema PASS 100% 329 in → 8 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
x-ai/grok-4.3:xhigh PASS 100% 274 in → 956 out (953 reasoning)
Status: Failed
x-ai/grok-4.3:xhigh__json_schema PASS 100% 332 in → 840 out (832 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
x-ai/grok-4.5:high PASS 100% 296 in → 209 out (206 reasoning)
Status: Failed
x-ai/grok-4.5:high__json_schema PASS 100% 366 in → 433 out (420 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
x-ai/grok-4.5:low PASS 100% 296 in → 172 out (169 reasoning)
Status: Failed
x-ai/grok-4.5:low__json_schema PASS 100% 366 in → 239 out (224 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed\n" }
x-ai/grok-4.5:max PASS 100% 295 in → 211 out (208 reasoning)
Status: Failed
x-ai/grok-4.5:max__json_schema PASS 100% 364 in → 527 out (517 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
x-ai/grok-4.5:medium PASS 100% 296 in → 213 out (210 reasoning)
Status: Failed
x-ai/grok-4.5:medium__json_schema PASS 100% 364 in → 641 out (628 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
x-ai/grok-4.5:xhigh PASS 100% 296 in → 286 out (283 reasoning)
Status: Failed
x-ai/grok-4.5:xhigh__json_schema PASS 100% 367 in → 330 out (320 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content":"Status: Failed"}
z-ai/glm-4.7-flash:high PASS 100% 97 in → 323 out (320 reasoning)
Status: Failed
z-ai/glm-4.7-flash:low PASS 100% 97 in → 1,831 out (1,968 reasoning)
Status: Failed
z-ai/glm-4.7-flash:none PASS 100% 98 in → 3 out (0 reasoning)
Status: Failed
z-ai/glm-4.7-flash:none__json_schema PASS 100% 97 in → 13 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
z-ai/glm-4.7-flash:xhigh PASS 100% 98 in → 325 out (325 reasoning)
Status: Failed
z-ai/glm-5.2:high PASS 100% 103 in → 354 out (393 reasoning)
Status: Failed
z-ai/glm-5.2:high__json_schema PASS 100% 97 in → 320 out (307 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
z-ai/glm-5.2:low PASS 100% 105 in → 293 out (289 reasoning)
Status: Failed
z-ai/glm-5.2:low__json_schema PASS 100% 98 in → 13 out (0 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
z-ai/glm-5.2:medium PASS 100% 105 in → 581 out (586 reasoning)
Status: Failed
z-ai/glm-5.2:medium__json_schema PASS 100% 107 in → 238 out (224 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) { "resolved_content": "Status: Failed" }
z-ai/glm-5.2:xhigh PASS 100% 104 in → 318 out (313 reasoning)
Status: Failed
z-ai/glm-5.2:xhigh__json_schema PASS 100% 281 in → 197 out (186 reasoning)
Status: Failed
JSON Schema Structured Output
(raw) {"resolved_content": "Status: Failed"}
anthropic/claude-fable-5:max FAIL 0% 172 in → 10 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['status.txt']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
ibm-granite/granite-4.1-8b FAIL 0% 108 in → 8 out
Resolved file content: Status: Failed
Failure: Content mismatch: ['status.txt']
liquid/lfm-2-24b-a2b FAIL 0% 113 in → 47 out
status.txt <<<<<<< HEAD Status: Resolved ======= Status: Failed >>>>>>> fd2035e (Update status) # Keep the current branch value Status: Failed
Failure: Content mismatch: ['status.txt']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: status.txt <<<<<<< HEAD Status: Failed ======= Status: Resolved >>>>>>> edb4406 (Update status) Resolved file content: Status: Failed
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
minimax/minimax-m2.7:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
mistralai/mistral-medium-3-5:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:high__json_schema FAIL 0%
Invalid JSON. Output: Status: Failed
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
moonshotai/kimi-k2.6:low FAIL 0% 99 in → 619 out (697 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
moonshotai/kimi-k2.6:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:medium FAIL 0% 100 in → 422 out (417 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
moonshotai/kimi-k2.6:xhigh FAIL 0% 99 in → 396 out (460 reasoning)
Status: Failed
Failure: Content mismatch: ['status.txt']
moonshotai/kimi-k2.7-code:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
poolside/laguna-m.1:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-27b:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6eb01a5f-295d-9e7d-8caf-3f46702738b0","request_id":"6eb01a5f-295d-9e7d-8caf-3f46702738b0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-deaac94a-376e-9353-a1f9-01f6fea77400","request_id":"deaac94a-376e-9353-a1f9-01f6fea77400"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f90086cc-d5a8-9520-9b52-574e77f610b4","request_id":"f90086cc-d5a8-9520-9b52-574e77f610b4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-13a9dccb-fa2d-995b-bb65-5e7664e3ab75","request_id":"13a9dccb-fa2d-995b-bb65-5e7664e3ab75"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4058c905-f723-9bd5-abcc-e84be960113b","request_id":"4058c905-f723-9bd5-abcc-e84be960113b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0e9ddd3a-39d8-9f60-b5a3-95273f015d51"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3d7318a5-05e1-9178-8140-97f329aff885"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ce170c86-e007-9ee9-b686-81ad0794c4bc"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-99bf5de9-806a-97c6-97a1-4bdeff9887e3"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-18bc59e1-fd62-9ee9-beb1-9e64c56432db"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9248f7ed-ce71-9256-bf19-e524463e7124"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-077b2aca-b333-9689-a23b-2f1887fe20d3","request_id":"077b2aca-b333-9689-a23b-2f1887fe20d3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d3eae9fc-5da7-9148-9e6c-a38d47e5557a","request_id":"d3eae9fc-5da7-9148-9e6c-a38d47e5557a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8b790eb1-2a76-9d18-ba66-ddfee52a3ef1","request_id":"8b790eb1-2a76-9d18-ba66-ddfee52a3ef1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e6b7a3fa-030b-92ba-8a33-ca2d7dbffe77","request_id":"e6b7a3fa-030b-92ba-8a33-ca2d7dbffe77"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0df02e04-ec88-98d7-bc6e-757cb9ad8b30","request_id":"0df02e04-ec88-98d7-bc6e-757cb9ad8b30"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f3b09d43-4d3f-9a04-924d-983395bcaf6c","request_id":"f3b09d43-4d3f-9a04-924d-983395bcaf6c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d6b0684a-bc77-905e-b945-5d4dfdad45c5","request_id":"d6b0684a-bc77-905e-b945-5d4dfdad45c5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a949cffc-a014-9a9a-b42b-7f57144d5d2f","request_id":"a949cffc-a014-9a9a-b42b-7f57144d5d2f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 107 in → 6 out (0 reasoning)
Status: Failed<|end_message|>
Failure: Content mismatch: ['status.txt']
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: Status: Failed{ "resolved_content": "Status: Failed" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:max__json_schema FAIL 0%
Invalid JSON. Output: Status: Failed
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium FAIL 0% 97 in → 409 out (420 reasoning)
Status: Resolved
Failure: Content mismatch: ['status.txt']
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:none FAIL 0%
(empty output)
Failure: Model 'z-ai/glm-5.2:none' violated the reasoning_level=none invariant: provider reported 254 reasoning token(s)
z-ai/glm-5.2:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.