Three-way: author line with three different values
Tests ability to resolve a three-way author-name conflict where all three values differ. Evaluates complex merge decisions.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Author: John' > author.txt
  5. 05 git add author.txt
  6. 06 git commit -m 'Initial author'
  7. 07 git checkout -b fullname
  8. 08 echo 'Author: John Smith' > author.txt
  9. 09 git add author.txt
  10. 10 git commit -m 'Use full name'
  11. 11 git checkout main
  12. 12 git checkout -b initials
  13. 13 echo 'Author: J. Doe' > author.txt
  14. 14 git add author.txt
  15. 15 git commit -m 'Use initials'
  16. 16 git checkout main
  17. 17 git merge fullname
  18. 18 git merge initials
Prompt
Resolve the merge conflict in author.txt. Three different authors were set: base 'Author: John', branch A 'Author: John Smith', branch B 'Author: J. Doe'. Both branches were merged into main (fullname first, then initials). Keep the value from the second merge side. Provide ONLY the resolved file content.
Expected
Author: J. Doe

Scoped model quality, cost, API time, and token usage for merge_conflicts/f009.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 169 in → 37 out (23 reasoning)
Author: J. Doe
anthropic/claude-fable-5:high__json_schema PASS 100% 389 in → 43 out (23 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-fable-5:low PASS 100% 169 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-fable-5:low__json_schema PASS 100% 389 in → 21 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-fable-5:max PASS 100% 169 in → 215 out (62 reasoning)
Author: J. Doe
anthropic/claude-fable-5:max__json_schema PASS 100% 389 in → 144 out (41 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-fable-5:medium PASS 100% 169 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-fable-5:medium__json_schema PASS 100% 389 in → 21 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-fable-5:xhigh PASS 100% 169 in → 91 out (40 reasoning)
Author: J. Doe
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 389 in → 47 out (19 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-haiku-4.5:high PASS 100% 149 in → 183 out (153 reasoning)
Author: J. Doe
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 317 in → 230 out (189 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-haiku-4.5:low PASS 100% 149 in → 251 out (218 reasoning)
``` Author: J. Doe ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 317 in → 274 out (243 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-haiku-4.5:medium PASS 100% 149 in → 406 out (372 reasoning)
``` Author: J. Doe ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 317 in → 142 out (121 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-haiku-4.5:none PASS 100% 120 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 287 in → 16 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 149 in → 178 out (152 reasoning)
``` Author: J. Doe ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 317 in → 278 out (249 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-opus-4.6:high PASS 100% 120 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:high__json_schema PASS 100% 288 in → 68 out (37 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.6:low PASS 100% 120 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:low__json_schema PASS 100% 288 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.6:max PASS 100% 120 in → 175 out (144 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:max__json_schema PASS 100% 288 in → 175 out (136 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.6:medium PASS 100% 120 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 288 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.6:none PASS 100% 120 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:none__json_schema PASS 100% 288 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 120 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 288 in → 68 out (37 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:high PASS 100% 174 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:high__json_schema PASS 100% 394 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:low PASS 100% 174 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:low__json_schema PASS 100% 394 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:max PASS 100% 174 in → 304 out (74 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:max__json_schema PASS 100% 394 in → 129 out (45 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:medium PASS 100% 174 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 394 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:none PASS 100% 174 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:none__json_schema PASS 100% 394 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 174 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 394 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.8:high PASS 100% 169 in → 84 out (36 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:high__json_schema PASS 100% 389 in → 50 out (22 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.8:low PASS 100% 169 in → 49 out (25 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:low__json_schema PASS 100% 389 in → 21 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-opus-4.8:max PASS 100% 169 in → 685 out (133 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:max__json_schema PASS 100% 389 in → 479 out (126 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-opus-4.8:medium PASS 100% 169 in → 40 out (30 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 389 in → 69 out (38 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.8:none PASS 100% 169 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:none__json_schema PASS 100% 389 in → 22 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 169 in → 299 out (83 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 389 in → 70 out (28 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-sonnet-4.6:high PASS 100% 120 in → 207 out (168 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 288 in → 146 out (113 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 120 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 288 in → 15 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
anthropic/claude-sonnet-4.6:max PASS 100% 120 in → 293 out (160 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 288 in → 227 out (175 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 120 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 288 in → 15 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
anthropic/claude-sonnet-4.6:none PASS 100% 120 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 288 in → 15 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 120 in → 235 out (188 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 288 in → 131 out (99 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-5:high PASS 100% 169 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-5:high__json_schema PASS 100% 389 in → 40 out (15 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-sonnet-5:low PASS 100% 169 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-5:low__json_schema PASS 100% 389 in → 23 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-5:medium PASS 100% 169 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 389 in → 23 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-sonnet-5:none PASS 100% 169 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-5:none__json_schema PASS 100% 389 in → 22 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 169 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 389 in → 63 out (33 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 104 in → 542 out (538 reasoning)
Author: J. Doe
arcee-ai/trinity-large-thinking:low PASS 100% 104 in → 605 out (602 reasoning)
Author: J. Doe
arcee-ai/trinity-large-thinking:medium PASS 100% 104 in → 978 out (974 reasoning)
Author: J. Doe
arcee-ai/trinity-large-thinking:xhigh PASS 100% 104 in → 507 out (502 reasoning)
Author: J. Doe
arcee-ai/trinity-mini:low PASS 100% 104 in → 400 out (422 reasoning)
Author: J. Doe
arcee-ai/trinity-mini:medium PASS 100% 104 in → 147 out (153 reasoning)
Author: J. Doe
arcee-ai/trinity-mini:medium__json_schema PASS 100% 104 in → 318 out (343 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
arcee-ai/trinity-mini:xhigh PASS 100% 104 in → 356 out (380 reasoning)
Author: J. Doe
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 104 in → 751 out (808 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 189 in → 519 out (562 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 183 in → 174 out (184 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 104 in → 308 out (319 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 104 in → 73 out (85 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 183 in → 572 out (566 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 185 in → 519 out (503 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 104 in → 239 out (242 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 104 in → 293 out (297 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 104 in → 6 out (0 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 104 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 104 in → 199 out (214 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 106 in → 196 out (179 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash:high PASS 100% 103 in → 327 out (320 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 106 in → 80 out (61 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
deepseek/deepseek-v4-flash:low PASS 100% 104 in → 249 out (273 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 296 in → 446 out (433 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
deepseek/deepseek-v4-flash:medium PASS 100% 104 in → 241 out (261 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 104 in → 105 out (89 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
deepseek/deepseek-v4-flash:none PASS 100% 104 in → 6 out (0 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash:xhigh PASS 100% 104 in → 214 out (209 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 201 in → 13 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
deepseek/deepseek-v4-pro:high PASS 100% 104 in → 293 out (286 reasoning)
Author: J. Doe
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 296 in → 190 out (176 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
deepseek/deepseek-v4-pro:low PASS 100% 104 in → 196 out (189 reasoning)
Author: J. Doe
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 296 in → 167 out (153 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
deepseek/deepseek-v4-pro:medium PASS 100% 104 in → 349 out (342 reasoning)
Author: J. Doe
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 296 in → 202 out (188 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
deepseek/deepseek-v4-pro:none PASS 100% 104 in → 10 out (0 reasoning)
```text Author: J. Doe ```
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 296 in → 12 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 183 in → 174 out (167 reasoning)
Author: J. Doe
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 374 in → 533 out (519 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3-flash-preview:high PASS 100% 99 in → 290 out (285 reasoning)
Author: J. Doe
google/gemini-3-flash-preview:high__json_schema PASS 100% 159 in → 233 out (217 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3-flash-preview:low PASS 100% 99 in → 248 out (243 reasoning)
Author: J. Doe
google/gemini-3-flash-preview:low__json_schema PASS 100% 100 in → 271 out (260 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3-flash-preview:medium PASS 100% 99 in → 247 out (242 reasoning)
Author: J. Doe
google/gemini-3-flash-preview:medium__json_schema PASS 100% 159 in → 797 out (786 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3-flash-preview:xhigh PASS 100% 99 in → 278 out (273 reasoning)
Author: J. Doe
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 159 in → 316 out (305 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 99 in → 1,226 out (1,221 reasoning)
Author: J. Doe
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 100 in → 1,334 out (1,322 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 100 in → 139 out (134 reasoning)
Author: J. Doe
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 100 in → 135 out (123 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 99 in → 221 out (216 reasoning)
Author: J. Doe
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 100 in → 181 out (164 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 100 in → 413 out (408 reasoning)
Author: J. Doe
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 100 in → 810 out (798 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.1-pro-preview:high PASS 100% 99 in → 309 out (304 reasoning)
Author: J. Doe
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 159 in → 552 out (540 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.1-pro-preview:low PASS 100% 99 in → 272 out (267 reasoning)
Author: J. Doe
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 159 in → 196 out (177 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
google/gemini-3.1-pro-preview:medium PASS 100% 99 in → 251 out (246 reasoning)
Author: J. Doe
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 159 in → 176 out (162 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 99 in → 516 out (511 reasoning)
Author: J. Doe
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 159 in → 304 out (293 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.5-flash-lite:high PASS 100% 99 in → 406 out (401 reasoning)
Author: J. Doe
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 159 in → 647 out (636 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.5-flash-lite:low PASS 100% 99 in → 5 out (0 reasoning)
Author: J. Doe
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 159 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.5-flash-lite:max PASS 100% 99 in → 345 out (340 reasoning)
Author: J. Doe
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 159 in → 653 out (642 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.5-flash-lite:medium PASS 100% 100 in → 424 out (419 reasoning)
Author: J. Doe
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 100 in → 470 out (459 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 99 in → 305 out (300 reasoning)
Author: J. Doe
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 159 in → 324 out (312 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.5-flash:high PASS 100% 99 in → 413 out (408 reasoning)
Author: J. Doe
google/gemini-3.5-flash:high__json_schema PASS 100% 159 in → 1,348 out (1,329 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
google/gemini-3.5-flash:low PASS 100% 99 in → 236 out (231 reasoning)
Author: J. Doe
google/gemini-3.5-flash:low__json_schema PASS 100% 159 in → 228 out (216 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.5-flash:medium PASS 100% 99 in → 491 out (486 reasoning)
Author: J. Doe
google/gemini-3.5-flash:medium__json_schema PASS 100% 159 in → 517 out (500 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.5-flash:xhigh PASS 100% 99 in → 592 out (587 reasoning)
Author: J. Doe
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 159 in → 678 out (661 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.6-flash:high PASS 100% 100 in → 339 out (334 reasoning)
Author: J. Doe
google/gemini-3.6-flash:high__json_schema PASS 100% 100 in → 784 out (767 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.6-flash:low PASS 100% 100 in → 88 out (83 reasoning)
Author: J. Doe
google/gemini-3.6-flash:low__json_schema PASS 100% 100 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.6-flash:max PASS 100% 100 in → 406 out (401 reasoning)
Author: J. Doe
google/gemini-3.6-flash:max__json_schema PASS 100% 100 in → 722 out (711 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.6-flash:medium PASS 100% 100 in → 321 out (316 reasoning)
Author: J. Doe
google/gemini-3.6-flash:medium__json_schema PASS 100% 100 in → 609 out (598 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.6-flash:xhigh PASS 100% 99 in → 477 out (472 reasoning)
Author: J. Doe
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 159 in → 1,403 out (1,389 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
google/gemma-4-26b-a4b-it:high PASS 100% 114 in → 446 out (417 reasoning)
Author: J. Doe
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 114 in → 307 out (281 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-26b-a4b-it:low PASS 100% 115 in → 303 out (279 reasoning)
Author: J. Doe
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 115 in → 298 out (275 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-26b-a4b-it:medium PASS 100% 115 in → 267 out (245 reasoning)
Author: J. Doe
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 115 in → 371 out (337 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-26b-a4b-it:none PASS 100% 112 in → 6 out (0 reasoning)
Author: J. Doe
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 112 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 115 in → 284 out (266 reasoning)
Author: J. Doe
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 115 in → 251 out (223 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-31b-it:high PASS 100% 115 in → 194 out (164 reasoning)
Author: J. Doe
google/gemma-4-31b-it:high__json_schema PASS 100% 114 in → 214 out (168 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-31b-it:low PASS 100% 115 in → 228 out (194 reasoning)
Author: J. Doe
google/gemma-4-31b-it:low__json_schema PASS 100% 115 in → 233 out (200 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-31b-it:medium PASS 100% 115 in → 245 out (219 reasoning)
Author: J. Doe
google/gemma-4-31b-it:medium__json_schema PASS 100% 114 in → 237 out (208 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-31b-it:none PASS 100% 112 in → 6 out (0 reasoning)
Author: J. Doe
google/gemma-4-31b-it:xhigh PASS 100% 115 in → 264 out (1 reasoning)
Author: J. Doe
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 118 in → 257 out (244 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
ibm-granite/granite-4.1-8b PASS 100% 107 in → 6 out
Author: J. Doe
ibm-granite/granite-4.1-8b__json_schema PASS 100% 107 in → 16 out
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
inclusionai/ling-2.6-flash PASS 100% 120 in → 6 out
Author: J. Doe
inclusionai/ling-2.6-flash__json_schema PASS 100% 120 in → 17 out
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
liquid/lfm-2-24b-a2b PASS 100% 118 in → 7 out
Author: J. Doe
minimax/minimax-m2.5:high PASS 100% 135 in → 313 out (347 reasoning)
Author: J. Doe
minimax/minimax-m2.5:high__json_schema PASS 100% 142 in → 222 out (229 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m2.5:low PASS 100% 135 in → 360 out (393 reasoning)
Author: J. Doe
minimax/minimax-m2.5:low__json_schema PASS 100% 142 in → 500 out (485 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m2.5:medium PASS 100% 135 in → 1,544 out (1,717 reasoning)
Author: J. Doe
minimax/minimax-m2.5:medium__json_schema PASS 100% 135 in → 380 out (366 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m2.5:xhigh PASS 100% 135 in → 416 out (466 reasoning)
Author: J. Doe
minimax/minimax-m2.7:high PASS 100% 135 in → 368 out (360 reasoning)
Author: J. Doe
minimax/minimax-m2.7:high__json_schema PASS 100% 134 in → 479 out (534 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m2.7:low PASS 100% 134 in → 499 out (556 reasoning)
Author: J. Doe
minimax/minimax-m2.7:low__json_schema PASS 100% 268 in → 302 out (290 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
minimax/minimax-m2.7:medium PASS 100% 135 in → 395 out (429 reasoning)
Author: J. Doe
minimax/minimax-m2.7:medium__json_schema PASS 100% 268 in → 359 out (347 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 268 in → 984 out (968 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m3:high PASS 100% 273 in → 143 out (157 reasoning)
Author: J. Doe
minimax/minimax-m3:high__json_schema PASS 100% 260 in → 12 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
minimax/minimax-m3:low PASS 100% 273 in → 103 out (102 reasoning)
Author: J. Doe
minimax/minimax-m3:low__json_schema PASS 100% 260 in → 15 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m3:medium PASS 100% 273 in → 96 out (96 reasoning)
Author: J. Doe
minimax/minimax-m3:medium__json_schema PASS 100% 260 in → 15 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m3:xhigh PASS 100% 273 in → 76 out (77 reasoning)
Author: J. Doe
minimax/minimax-m3:xhigh__json_schema PASS 100% 260 in → 15 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
mistralai/devstral-2512 PASS 100% 106 in → 6 out
Author: J. Doe
mistralai/devstral-2512__json_schema PASS 100% 106 in → 13 out
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
mistralai/mistral-medium-3-5:high PASS 100% 118 in → 429 out (444 reasoning)
Author: J. Doe
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 106 in → 224 out (230 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
mistralai/mistral-medium-3-5:low PASS 100% 118 in → 333 out (337 reasoning)
Author: J. Doe
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 106 in → 281 out (282 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
mistralai/mistral-medium-3-5:medium PASS 100% 118 in → 525 out (552 reasoning)
Author: J. Doe
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 106 in → 411 out (423 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
mistralai/mistral-medium-3-5:none PASS 100% 118 in → 6 out (0 reasoning)
Author: J. Doe
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 106 in → 13 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 118 in → 357 out (369 reasoning)
Author: J. Doe
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 106 in → 190 out (194 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
moonshotai/kimi-k2.6:high PASS 100% 103 in → 541 out (535 reasoning)
Author: J. Doe
moonshotai/kimi-k2.6:low PASS 100% 103 in → 1,070 out (1,063 reasoning)
Author: J. Doe
moonshotai/kimi-k2.6:medium PASS 100% 103 in → 597 out (590 reasoning)
Author: J. Doe
moonshotai/kimi-k2.6:xhigh PASS 100% 103 in → 1,058 out (1,051 reasoning)
Author: J. Doe
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 103 in → 255 out (240 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
moonshotai/kimi-k2.7-code:high PASS 100% 102 in → 179 out (172 reasoning)
Author: J. Doe
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 103 in → 271 out (255 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
moonshotai/kimi-k2.7-code:low PASS 100% 103 in → 216 out (209 reasoning)
Author: J. Doe
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 103 in → 563 out (616 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
moonshotai/kimi-k2.7-code:medium PASS 100% 102 in → 231 out (224 reasoning)
Author: J. Doe
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 103 in → 371 out (417 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 103 in → 176 out (160 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
moonshotai/kimi-k3:high PASS 100% 180 in → 369 out (349 reasoning)
Author: J. Doe
moonshotai/kimi-k3:high__json_schema PASS 100% 180 in → 257 out (237 reasoning)
Author: J. Doe
moonshotai/kimi-k3:low PASS 100% 180 in → 265 out (245 reasoning)
Author: J. Doe
moonshotai/kimi-k3:low__json_schema PASS 100% 180 in → 201 out (181 reasoning)
Author: J. Doe
moonshotai/kimi-k3:max PASS 100% 180 in → 152 out (132 reasoning)
Author: J. Doe
moonshotai/kimi-k3:max__json_schema PASS 100% 180 in → 151 out (131 reasoning)
Author: J. Doe
moonshotai/kimi-k3:medium PASS 100% 180 in → 203 out (183 reasoning)
Author: J. Doe
moonshotai/kimi-k3:medium__json_schema PASS 100% 180 in → 272 out (252 reasoning)
Author: J. Doe
moonshotai/kimi-k3:xhigh PASS 100% 180 in → 148 out (128 reasoning)
Author: J. Doe
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 180 in → 192 out (172 reasoning)
Author: J. Doe
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 119 in → 342 out (354 reasoning)
Author: J. Doe
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 119 in → 809 out (859 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 119 in → 389 out (422 reasoning)
Author: J. Doe
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 119 in → 2,081 out (2,231 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 119 in → 661 out (701 reasoning)
Author: J. Doe
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 119 in → 707 out (761 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 119 in → 1,625 out (1,761 reasoning)
Author: J. Doe
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 119 in → 219 out (217 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 119 in → 274 out (259 reasoning)
Author: J. Doe
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 119 in → 124 out (104 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 119 in → 262 out (243 reasoning)
Author: J. Doe
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 119 in → 215 out (192 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 119 in → 298 out (274 reasoning)
Author: J. Doe
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 119 in → 138 out (119 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 119 in → 6 out (0 reasoning)
Author: J. Doe
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 119 in → 16 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 119 in → 185 out (190 reasoning)
Author: J. Doe
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 119 in → 219 out (193 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
openai/gpt-5.4-mini:high PASS 100% 99 in → 137 out (126 reasoning)
Author: J. Doe
openai/gpt-5.4-mini:high__json_schema PASS 100% 135 in → 156 out (136 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4-mini:low PASS 100% 99 in → 55 out (44 reasoning)
Author: J. Doe
openai/gpt-5.4-mini:low__json_schema PASS 100% 135 in → 66 out (46 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4-mini:medium PASS 100% 99 in → 112 out (101 reasoning)
Author: J. Doe
openai/gpt-5.4-mini:medium__json_schema PASS 100% 135 in → 112 out (92 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4-mini:none PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.4-mini:none__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4-mini:xhigh PASS 100% 99 in → 212 out (201 reasoning)
Author: J. Doe
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 135 in → 300 out (280 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4-nano:high PASS 100% 99 in → 123 out (112 reasoning)
Author: J. Doe
openai/gpt-5.4-nano:high__json_schema PASS 100% 135 in → 160 out (139 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4-nano:low PASS 100% 99 in → 40 out (29 reasoning)
Author: J. Doe
openai/gpt-5.4-nano:low__json_schema PASS 100% 135 in → 97 out (76 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4-nano:medium PASS 100% 99 in → 66 out (55 reasoning)
Author: J. Doe
openai/gpt-5.4-nano:medium__json_schema PASS 100% 135 in → 189 out (168 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4-nano:none PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.4-nano:none__json_schema PASS 100% 135 in → 19 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 99 in → 902 out (891 reasoning)
Author: J. Doe
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 135 in → 429 out (408 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4:high PASS 100% 99 in → 168 out (157 reasoning)
Author: J. Doe
openai/gpt-5.4:high__json_schema PASS 100% 135 in → 93 out (73 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4:low PASS 100% 99 in → 28 out (17 reasoning)
Author: J. Doe
openai/gpt-5.4:low__json_schema PASS 100% 135 in → 49 out (29 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4:medium PASS 100% 99 in → 80 out (69 reasoning)
Author: J. Doe
openai/gpt-5.4:medium__json_schema PASS 100% 135 in → 116 out (96 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4:none PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.4:none__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4:xhigh PASS 100% 99 in → 323 out (312 reasoning)
Author: J. Doe
openai/gpt-5.4:xhigh__json_schema PASS 100% 135 in → 193 out (173 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.5:high PASS 100% 99 in → 132 out (121 reasoning)
Author: J. Doe
openai/gpt-5.5:high__json_schema PASS 100% 135 in → 132 out (112 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.5:low PASS 100% 99 in → 48 out (37 reasoning)
Author: J. Doe
openai/gpt-5.5:low__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.5:medium PASS 100% 99 in → 88 out (77 reasoning)
Author: J. Doe
openai/gpt-5.5:medium__json_schema PASS 100% 135 in → 99 out (78 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.5:none PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.5:none__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.5:xhigh PASS 100% 99 in → 189 out (178 reasoning)
Author: J. Doe
openai/gpt-5.5:xhigh__json_schema PASS 100% 135 in → 479 out (459 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:high PASS 100% 99 in → 45 out (34 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:high__json_schema PASS 100% 135 in → 52 out (32 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:low PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:low__json_schema PASS 100% 135 in → 37 out (17 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:max PASS 100% 99 in → 91 out (80 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:max__json_schema PASS 100% 135 in → 73 out (53 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:medium PASS 100% 99 in → 28 out (17 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:medium__json_schema PASS 100% 135 in → 37 out (17 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:none PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:none__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:xhigh PASS 100% 99 in → 49 out (38 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 135 in → 51 out (31 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:high PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:high__json_schema PASS 100% 135 in → 19 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.6-sol:low PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:low__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:max PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:max__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:medium PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:medium__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:none PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:none__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:xhigh PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:high PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:high__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:low PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:low__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:max PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:max__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:medium PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:medium__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:none PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:none__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:xhigh PASS 100% 99 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-oss-120b:high PASS 100% 160 in → 159 out (145 reasoning)
Author: J. Doe
openai/gpt-oss-120b:high__json_schema PASS 100% 147 in → 468 out (515 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
openai/gpt-oss-120b:low PASS 100% 147 in → 58 out (49 reasoning)
Author: J. Doe
openai/gpt-oss-120b:low__json_schema PASS 100% 229 in → 53 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
openai/gpt-oss-120b:medium PASS 100% 162 in → 158 out (164 reasoning)
Author: J. Doe
openai/gpt-oss-120b:medium__json_schema PASS 100% 147 in → 259 out (277 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
openai/gpt-oss-120b:xhigh PASS 100% 147 in → 638 out (711 reasoning)
Author: J. Doe
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 147 in → 535 out (593 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-oss-20b:high PASS 100% 160 in → 371 out (366 reasoning)
Author: J. Doe
openai/gpt-oss-20b:high__json_schema PASS 100% 160 in → 6,346 out (7,410 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
openai/gpt-oss-20b:low PASS 100% 160 in → 65 out (54 reasoning)
Author: J. Doe
openai/gpt-oss-20b:low__json_schema PASS 100% 162 in → 52 out (29 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
openai/gpt-oss-20b:medium PASS 100% 158 in → 460 out (514 reasoning)
Author: J. Doe
openai/gpt-oss-20b:xhigh PASS 100% 162 in → 403 out (438 reasoning)
Author: J. Doe
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 158 in → 2,545 out (2,918 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
poolside/laguna-m.1:high PASS 100% 110 in → 424 out (417 reasoning)
Author: J. Doe
poolside/laguna-m.1:high__json_schema PASS 100% 110 in → 296 out (284 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
poolside/laguna-m.1:low PASS 100% 110 in → 262 out (254 reasoning)
Author: J. Doe
poolside/laguna-m.1:low__json_schema PASS 100% 110 in → 1,175 out (1,158 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-m.1:medium PASS 100% 110 in → 313 out (305 reasoning)
Author: J. Doe
poolside/laguna-m.1:medium__json_schema PASS 100% 110 in → 219 out (202 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
poolside/laguna-m.1:none PASS 100% 110 in → 7 out (0 reasoning)
Author: J. Doe
poolside/laguna-m.1:none__json_schema PASS 100% 110 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
poolside/laguna-m.1:xhigh PASS 100% 110 in → 237 out (229 reasoning)
Author: J. Doe
poolside/laguna-m.1:xhigh__json_schema PASS 100% 110 in → 285 out (267 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-xs-2.1:high PASS 100% 110 in → 330 out (324 reasoning)
Author: J. Doe
poolside/laguna-xs-2.1:high__json_schema PASS 100% 110 in → 463 out (446 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-xs-2.1:low PASS 100% 110 in → 759 out (754 reasoning)
Author: J. Doe
poolside/laguna-xs-2.1:medium PASS 100% 110 in → 516 out (510 reasoning)
Author: J. Doe
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 110 in → 1,229 out (1,212 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-xs-2.1:xhigh PASS 100% 110 in → 380 out (375 reasoning)
Author: J. Doe
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 110 in → 370 out (353 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-xs.2:high PASS 100% 147 in → 195 out (187 reasoning)
Author: J. Doe
poolside/laguna-xs.2:high__json_schema PASS 100% 147 in → 287 out (274 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
poolside/laguna-xs.2:low PASS 100% 147 in → 159 out (151 reasoning)
Author: J. Doe
poolside/laguna-xs.2:low__json_schema PASS 100% 147 in → 410 out (397 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
poolside/laguna-xs.2:medium PASS 100% 147 in → 214 out (206 reasoning)
Author: J. Doe
poolside/laguna-xs.2:medium__json_schema PASS 100% 147 in → 299 out (285 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
poolside/laguna-xs.2:none PASS 100% 147 in → 8 out (0 reasoning)
Author: J. Doe
poolside/laguna-xs.2:none__json_schema PASS 100% 147 in → 12 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
poolside/laguna-xs.2:xhigh PASS 100% 147 in → 194 out (186 reasoning)
Author: J. Doe
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 147 in → 251 out (238 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
qwen/qwen3.6-27b:high PASS 100% 105 in → 1,157 out (1,147 reasoning)
Author: J. Doe
qwen/qwen3.6-27b:high__json_schema PASS 100% 105 in → 795 out (793 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-27b:low PASS 100% 105 in → 954 out (926 reasoning)
Author: J. Doe
qwen/qwen3.6-27b:medium PASS 100% 105 in → 1,084 out (1,081 reasoning)
Author: J. Doe
qwen/qwen3.6-27b:medium__json_schema PASS 100% 105 in → 398 out (374 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-27b:none PASS 100% 107 in → 6 out (0 reasoning)
Author: J. Doe
qwen/qwen3.6-27b:none__json_schema PASS 100% 107 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-27b:xhigh PASS 100% 105 in → 981 out (1 reasoning)
Author: J. Doe
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 105 in → 662 out (672 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-35b-a3b:high PASS 100% 105 in → 1,071 out (1,064 reasoning)
Author: J. Doe
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 105 in → 936 out (940 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-35b-a3b:low PASS 100% 105 in → 1,138 out (1,128 reasoning)
Author: J. Doe
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 105 in → 979 out (960 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 105 in → 1,111 out (1,094 reasoning)
Author: J. Doe
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 105 in → 1,147 out (1,147 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-35b-a3b:none PASS 100% 107 in → 6 out (0 reasoning)
Author: J. Doe
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 107 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 105 in → 1,075 out (1,063 reasoning)
Author: J. Doe
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 105 in → 1,544 out (1,487 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-flash:high PASS 100% 105 in → 1,003 out (993 reasoning)
Author: J. Doe
qwen/qwen3.6-flash:low PASS 100% 105 in → 864 out (854 reasoning)
Author: J. Doe
qwen/qwen3.6-flash:medium PASS 100% 105 in → 808 out (798 reasoning)
Author: J. Doe
qwen/qwen3.6-flash:none PASS 100% 107 in → 5 out (0 reasoning)
Author: J. Doe
qwen/qwen3.6-flash:xhigh PASS 100% 105 in → 1,062 out (1,052 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:high PASS 100% 105 in → 1,032 out (1,022 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:low PASS 100% 105 in → 1,081 out (1,071 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:max PASS 100% 105 in → 1,111 out (1,101 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:medium PASS 100% 105 in → 529 out (519 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:none PASS 100% 107 in → 5 out (0 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:xhigh PASS 100% 105 in → 966 out (956 reasoning)
Author: J. Doe
qwen/qwen3.7-max:high PASS 100% 105 in → 231 out (221 reasoning)
Author: J. Doe
qwen/qwen3.7-max:low PASS 100% 105 in → 174 out (164 reasoning)
Author: J. Doe
qwen/qwen3.7-max:medium PASS 100% 105 in → 191 out (181 reasoning)
Author: J. Doe
qwen/qwen3.7-max:none PASS 100% 107 in → 5 out (0 reasoning)
Author: J. Doe
qwen/qwen3.7-max:xhigh PASS 100% 105 in → 165 out (155 reasoning)
Author: J. Doe
qwen/qwen3.7-plus:high PASS 100% 105 in → 254 out (244 reasoning)
Author: J. Doe
qwen/qwen3.7-plus:low PASS 100% 105 in → 195 out (185 reasoning)
Author: J. Doe
qwen/qwen3.7-plus:medium PASS 100% 105 in → 298 out (288 reasoning)
Author: J. Doe
qwen/qwen3.7-plus:none PASS 100% 107 in → 5 out (0 reasoning)
Author: J. Doe
qwen/qwen3.7-plus:xhigh PASS 100% 105 in → 413 out (403 reasoning)
Author: J. Doe
tencent/hy3:high PASS 100% 110 in → 389 out (382 reasoning)
Author: J. Doe
tencent/hy3:high__json_schema PASS 100% 110 in → 925 out (943 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
tencent/hy3:low PASS 100% 110 in → 540 out (533 reasoning)
Author: J. Doe
tencent/hy3:low__json_schema PASS 100% 110 in → 509 out (520 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
tencent/hy3:medium PASS 100% 110 in → 574 out (567 reasoning)
Author: J. Doe
tencent/hy3:medium__json_schema PASS 100% 110 in → 567 out (578 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
tencent/hy3:none PASS 100% 113 in → 6 out (0 reasoning)
Author: J. Doe
tencent/hy3:none__json_schema PASS 100% 113 in → 14 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
tencent/hy3:xhigh PASS 100% 110 in → 491 out (484 reasoning)
Author: J. Doe
tencent/hy3:xhigh__json_schema PASS 100% 110 in → 456 out (448 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
thinkingmachines/inkling-small:high PASS 100% 108 in → 211 out (200 reasoning)
Author: J. Doe
thinkingmachines/inkling-small:low PASS 100% 108 in → 100 out (102 reasoning)
Author: J. Doe
thinkingmachines/inkling-small:low__json_schema PASS 100% 108 in → 72 out (56 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
thinkingmachines/inkling-small:max PASS 100% 108 in → 200 out (189 reasoning)
Author: J. Doe
thinkingmachines/inkling-small:max__json_schema PASS 100% 108 in → 278 out (259 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
thinkingmachines/inkling-small:medium PASS 100% 108 in → 133 out (122 reasoning)
Author: J. Doe
thinkingmachines/inkling-small:medium__json_schema PASS 100% 108 in → 134 out (118 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
thinkingmachines/inkling-small:xhigh PASS 100% 108 in → 343 out (376 reasoning)
Author: J. Doe
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 108 in → 160 out (142 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
thinkingmachines/inkling:high PASS 100% 108 in → 130 out (118 reasoning)
Author: J. Doe
thinkingmachines/inkling:high__json_schema PASS 100% 108 in → 168 out (152 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
thinkingmachines/inkling:low PASS 100% 108 in → 32 out (20 reasoning)
Author: J. Doe
thinkingmachines/inkling:max PASS 100% 108 in → 178 out (166 reasoning)
Author: J. Doe
thinkingmachines/inkling:medium PASS 100% 108 in → 92 out (81 reasoning)
Author: J. Doe
thinkingmachines/inkling:medium__json_schema PASS 100% 108 in → 157 out (139 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
thinkingmachines/inkling:xhigh PASS 100% 108 in → 198 out (187 reasoning)
Author: J. Doe
thinkingmachines/inkling:xhigh__json_schema PASS 100% 108 in → 151 out (135 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
x-ai/grok-4.3:high PASS 100% 278 in → 540 out (535 reasoning)
Author: J. Doe
x-ai/grok-4.3:high__json_schema PASS 100% 335 in → 637 out (623 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
x-ai/grok-4.3:low PASS 100% 284 in → 263 out (258 reasoning)
Author: J. Doe
x-ai/grok-4.3:low__json_schema PASS 100% 341 in → 350 out (336 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
x-ai/grok-4.3:max PASS 100% 278 in → 446 out (441 reasoning)
Author: J. Doe
x-ai/grok-4.3:max__json_schema PASS 100% 335 in → 584 out (574 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
x-ai/grok-4.3:medium PASS 100% 284 in → 468 out (463 reasoning)
Author: J. Doe
x-ai/grok-4.3:medium__json_schema PASS 100% 341 in → 494 out (480 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
x-ai/grok-4.3:none PASS 100% 276 in → 5 out (0 reasoning)
Author: J. Doe
x-ai/grok-4.3:none__json_schema PASS 100% 333 in → 14 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
x-ai/grok-4.3:xhigh PASS 100% 278 in → 415 out (410 reasoning)
Author: J. Doe
x-ai/grok-4.3:xhigh__json_schema PASS 100% 335 in → 579 out (569 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
x-ai/grok-4.5:high PASS 100% 300 in → 332 out (327 reasoning)
Author: J. Doe
x-ai/grok-4.5:high__json_schema PASS 100% 369 in → 287 out (275 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
x-ai/grok-4.5:low PASS 100% 300 in → 168 out (163 reasoning)
Author: J. Doe
x-ai/grok-4.5:low__json_schema PASS 100% 369 in → 210 out (193 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
x-ai/grok-4.5:max PASS 100% 300 in → 184 out (179 reasoning)
Author: J. Doe
x-ai/grok-4.5:max__json_schema PASS 100% 369 in → 460 out (443 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
x-ai/grok-4.5:medium PASS 100% 300 in → 297 out (292 reasoning)
Author: J. Doe
x-ai/grok-4.5:medium__json_schema PASS 100% 369 in → 438 out (421 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
x-ai/grok-4.5:xhigh PASS 100% 300 in → 222 out (217 reasoning)
Author: J. Doe
x-ai/grok-4.5:xhigh__json_schema PASS 100% 369 in → 273 out (261 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
z-ai/glm-4.7-flash:high PASS 100% 98 in → 3,272 out (3,460 reasoning)
Author: J. Doe
z-ai/glm-4.7-flash:high__json_schema PASS 100% 98 in → 997 out (1,039 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
z-ai/glm-4.7-flash:low PASS 100% 98 in → 1,626 out (1,668 reasoning)
Author: J. Doe
z-ai/glm-4.7-flash:medium PASS 100% 98 in → 1,746 out (1,924 reasoning)
Author: J. Doe
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 98 in → 1,233 out (1,341 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
z-ai/glm-4.7-flash:none__json_schema PASS 100% 98 in → 15 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
z-ai/glm-4.7-flash:xhigh PASS 100% 98 in → 1,079 out (1,133 reasoning)
Author: J. Doe
z-ai/glm-5.2:high PASS 100% 105 in → 190 out (183 reasoning)
Author: J. Doe
z-ai/glm-5.2:low PASS 100% 105 in → 236 out (245 reasoning)
Author: J. Doe
z-ai/glm-5.2:low__json_schema PASS 100% 105 in → 12 out (11 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
z-ai/glm-5.2:medium PASS 100% 105 in → 140 out (148 reasoning)
Author: J. Doe
z-ai/glm-5.2:medium__json_schema PASS 100% 105 in → 401 out (430 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
z-ai/glm-5.2:none PASS 100% 99 in → 6 out (0 reasoning)
Author: J. Doe
z-ai/glm-5.2:none__json_schema PASS 100% 99 in → 14 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
z-ai/glm-5.2:xhigh PASS 100% 105 in → 722 out (715 reasoning)
Author: J. Doe
z-ai/glm-5.2:xhigh__json_schema PASS 100% 105 in → 371 out (391 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 104 in → 593 out (640 reasoning)
Author: J.Doe
Failure: Content mismatch: ['author.txt']
arcee-ai/trinity-mini:high__json_schema FAIL 0% 104 in → 224 out (240 reasoning)
Author: J.Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J.Doe"}
Failure: Content mismatch: ['author.txt']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 104 in → 737 out (791 reasoning)
Author:J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author:J. Doe" }
Failure: Content mismatch: ['author.txt']
(empty output)
JSON Schema Structured Output
Failure: Model 'deepseek/deepseek-v4-flash:none' violated the reasoning_level=none invariant: provider reported 121 reasoning token(s)
google/gemma-4-31b-it:none__json_schema FAIL 0% 112 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"resolved_content":""}
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:xhigh__json_schema FAIL 0% 135 in → 2,056 out (2,037 reasoning)
Author: John Smith (J. Doe)
JSON Schema Structured Output
(raw) { "resolved_content": "Author: John Smith (J. Doe)\n" }
Failure: Content mismatch: ['author.txt']
minimax/minimax-m2.7:xhigh FAIL 0% 135 in → 1,167 out (1,155 reasoning)
Author: John Smith (J. Doe)
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
moonshotai/kimi-k2.6:low__json_schema FAIL 0% 103 in → 374 out (367 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"resolved_content":""}
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:xhigh FAIL 0% 103 in → 230 out (263 reasoning)
Author: J. Doe
Failure: Content mismatch: ['author.txt']
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 119 in → 10 out (0 reasoning)
Author: John Smith Author: John Smith
Failure: Content mismatch: ['author.txt']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 119 in → 15 out (0 reasoning)
Author: John Smith
JSON Schema Structured Output
(raw) { "resolved_content": "Author: John Smith" }
Failure: Content mismatch: ['author.txt']
openai/gpt-oss-20b:medium__json_schema FAIL 0% 158 in → 532 out (567 reasoning)
Author: John Smith J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: John Smith J. Doe"}
Failure: Content mismatch: ['author.txt']
poolside/laguna-xs-2.1:low__json_schema FAIL 0% 110 in → 2,286 out (2,266 reasoning)
Author: John Smith J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: John Smith J. Doe" }
Failure: Content mismatch: ['author.txt']
poolside/laguna-xs-2.1:none FAIL 0% 110 in → 22 out (0 reasoning)
Author: John Smith Author: J. Doe </think>Author: John Smith Author: J. Doe
Failure: Content mismatch: ['author.txt']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 110 in → 17 out (0 reasoning)
Author: John Smith
JSON Schema Structured Output
(raw) { "resolved_content": "Author: John Smith\n" }
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-95e4177c-985d-992e-b2b0-6b940b89924b","request_id":"95e4177c-985d-992e-b2b0-6b940b89924b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-56381c15-2758-998a-a972-dd914cd70080","request_id":"56381c15-2758-998a-a972-dd914cd70080"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bf31b25a-6a1c-9558-8f56-311acd200298","request_id":"bf31b25a-6a1c-9558-8f56-311acd200298"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7f980ca2-206f-9b39-b5d8-9e621ca68f6c","request_id":"7f980ca2-206f-9b39-b5d8-9e621ca68f6c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0907df39-342c-97d2-bf78-738535126eac","request_id":"0907df39-342c-97d2-bf78-738535126eac"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b2ea267a-ff1c-91c7-8515-463e5a6dd524"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6992f528-05f0-99f1-a414-5fa36b077da1"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-fb759d28-4c87-9d41-8dfa-faabaec8a82d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-bfffeb51-8e3f-988e-ad86-b3f2a0457638"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-68ed75d8-d019-919f-b4c6-4c075eef8774"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5a870c7b-4163-9900-a450-cd9f06f48834"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b0d287dd-209f-918f-b037-8376d447ae4f","request_id":"b0d287dd-209f-918f-b037-8376d447ae4f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5c85a526-d8ac-9117-9070-20c9bd5dc539","request_id":"5c85a526-d8ac-9117-9070-20c9bd5dc539"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bb01ac05-47be-90bd-8d46-975560ba2fdd","request_id":"bb01ac05-47be-90bd-8d46-975560ba2fdd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a0a24d13-3de8-9c94-9018-f4f8951f5142","request_id":"a0a24d13-3de8-9c94-9018-f4f8951f5142"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0d7dd1d5-08e4-9138-af54-05caaee951d5","request_id":"0d7dd1d5-08e4-9138-af54-05caaee951d5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-83391a9c-5f55-90ec-a27f-5c8228177c82","request_id":"83391a9c-5f55-90ec-a27f-5c8228177c82"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-07b829c4-c9da-935d-8e08-5474679e3613","request_id":"07b829c4-c9da-935d-8e08-5474679e3613"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fbf88930-3ab2-96c4-be32-c8f1e7a3e7ac","request_id":"fbf88930-3ab2-96c4-be32-c8f1e7a3e7ac"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-eda2b0d2-65e3-9e35-b709-0def3b9d3abe","request_id":"eda2b0d2-65e3-9e35-b709-0def3b9d3abe"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-214531d5-996f-9889-a304-f9c90b83a91c","request_id":"214531d5-996f-9889-a304-f9c90b83a91c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: OpenAI-compatible response contained provider error: json_schema response format is not supported for model: thinkingmachines/Inkling-Small
thinkingmachines/inkling-small:none FAIL 0% 108 in → 8 out (0 reasoning)
Author: J. Doe<|end_message|>
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output: Author: J. Doe{ "resolved_content": "Author: J. Doe" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 108 in → 8 out (0 reasoning)
Author: J. Doe<|end_message|>
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output: Author: J. Doe{ "resolved_content": "Author: J. Doe" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 98 in → 5 out (0 reasoning)
Author: J. Smith
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)