Three-way: author line with three different values
Tests ability to resolve a three-way author-name conflict where base, incoming, and current all differ. Evaluates complex multi-value merge decisions.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Author: John' > author.txt
  5. 05 git add author.txt
  6. 06 git commit -m 'Initial author'
  7. 07 git checkout -b fullname
  8. 08 echo 'Author: John Smith' > author.txt
  9. 09 git add author.txt
  10. 10 git commit -m 'Use full name'
  11. 11 git checkout main
  12. 12 echo 'Author: J. Doe' > author.txt
  13. 13 git add author.txt
  14. 14 git commit -m 'Use initials'
  15. 15 git cherry-pick $(git rev-parse fullname)
Prompt
Resolve the cherry-pick conflict in author.txt. The base was 'Author: John'. The incoming commit (fullname branch) changed it to 'Author: John Smith'. The current branch (main) has 'Author: J. Doe'. Keep the current branch value. Provide ONLY the resolved file content.
Expected
Author: J. Doe

Scoped model quality, cost, API time, and token usage for cherry_pick/f009.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 177 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-fable-5:low PASS 100% 175 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-fable-5:low__json_schema PASS 100% 395 in → 21 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-fable-5:max PASS 100% 175 in → 90 out (38 reasoning)
Author: J. Doe
anthropic/claude-fable-5:max__json_schema PASS 100% 395 in → 116 out (39 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-fable-5:medium PASS 100% 174 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-fable-5:medium__json_schema PASS 100% 395 in → 21 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-fable-5:xhigh PASS 100% 177 in → 37 out (26 reasoning)
Author: J. Doe
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 396 in → 43 out (15 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-haiku-4.5:high PASS 100% 150 in → 231 out (204 reasoning)
``` Author: J. Doe ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 318 in → 215 out (187 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-haiku-4.5:low PASS 100% 151 in → 204 out (173 reasoning)
``` Author: J. Doe ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 319 in → 230 out (202 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-haiku-4.5:medium PASS 100% 150 in → 177 out (154 reasoning)
``` Author: J. Doe ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 321 in → 157 out (134 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-haiku-4.5:none PASS 100% 123 in → 13 out (0 reasoning)
``` Author: J. Doe ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 290 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 150 in → 172 out (160 reasoning)
Author: J. Doe
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 320 in → 245 out (221 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
anthropic/claude-opus-4.6:high PASS 100% 123 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:high__json_schema PASS 100% 291 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.6:low PASS 100% 121 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:low__json_schema PASS 100% 289 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.6:max PASS 100% 122 in → 66 out (38 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:max__json_schema PASS 100% 290 in → 72 out (37 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.6:medium PASS 100% 123 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 290 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.6:none PASS 100% 121 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:none__json_schema PASS 100% 291 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 123 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 291 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:high PASS 100% 180 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:high__json_schema PASS 100% 400 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:low PASS 100% 179 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:low__json_schema PASS 100% 403 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:max PASS 100% 182 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:max__json_schema PASS 100% 399 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:medium PASS 100% 183 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 401 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:none PASS 100% 183 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:none__json_schema PASS 100% 401 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 180 in → 12 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 401 in → 25 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.8:high PASS 100% 176 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:high__json_schema PASS 100% 395 in → 23 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.8:low PASS 100% 175 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:low__json_schema PASS 100% 395 in → 23 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.8:max PASS 100% 176 in → 298 out (62 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:max__json_schema PASS 100% 397 in → 104 out (28 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-opus-4.8:medium PASS 100% 179 in → 43 out (27 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 396 in → 23 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-opus-4.8:none PASS 100% 178 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:none__json_schema PASS 100% 397 in → 22 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 176 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 396 in → 23 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-4.6:high PASS 100% 122 in → 59 out (34 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 290 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 121 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 290 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-4.6:max PASS 100% 124 in → 79 out (54 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 292 in → 87 out (53 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 121 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 290 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 123 in → 9 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 290 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 123 in → 55 out (31 reasoning)
Author: J. Doe
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 291 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
anthropic/claude-sonnet-5:high PASS 100% 175 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-5:high__json_schema PASS 100% 396 in → 23 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-sonnet-5:low PASS 100% 176 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-5:low__json_schema PASS 100% 398 in → 23 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-sonnet-5:medium PASS 100% 178 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 395 in → 21 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
anthropic/claude-sonnet-5:none PASS 100% 177 in → 14 out (0 reasoning)
``` Author: J. Doe ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 397 in → 22 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 175 in → 10 out (0 reasoning)
Author: J. Doe
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 396 in → 23 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 105 in → 374 out (367 reasoning)
Author: J. Doe
arcee-ai/trinity-large-thinking:low PASS 100% 106 in → 287 out (280 reasoning)
Author: J. Doe
arcee-ai/trinity-large-thinking:medium PASS 100% 105 in → 381 out (375 reasoning)
Author: J. Doe
arcee-ai/trinity-large-thinking:xhigh PASS 100% 106 in → 301 out (294 reasoning)
Author: J. Doe
arcee-ai/trinity-mini:high PASS 100% 104 in → 327 out (361 reasoning)
Author: J. Doe
arcee-ai/trinity-mini:high__json_schema PASS 100% 106 in → 339 out (365 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
arcee-ai/trinity-mini:low PASS 100% 104 in → 264 out (296 reasoning)
Author: J. Doe
arcee-ai/trinity-mini:low__json_schema PASS 100% 107 in → 328 out (361 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
arcee-ai/trinity-mini:medium PASS 100% 106 in → 344 out (373 reasoning)
Author: J. Doe
arcee-ai/trinity-mini:medium__json_schema PASS 100% 107 in → 315 out (331 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 105 in → 298 out (322 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 186 in → 37 out (31 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 188 in → 53 out (43 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 107 in → 85 out (87 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 105 in → 75 out (74 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 205 in → 277 out (287 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 187 in → 63 out (57 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 106 in → 44 out (38 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 109 in → 96 out (92 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 107 in → 5 out (0 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 106 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 186 in → 80 out (74 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 109 in → 89 out (80 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash:high PASS 100% 108 in → 95 out (89 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 205 in → 12 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
deepseek/deepseek-v4-flash:low PASS 100% 108 in → 93 out (83 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 108 in → 87 out (68 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
deepseek/deepseek-v4-flash:medium PASS 100% 105 in → 67 out (71 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 175 in → 177 out (190 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
deepseek/deepseek-v4-flash:none PASS 100% 106 in → 5 out (0 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 111 in → 14 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 187 in → 198 out (192 reasoning)
Author: J. Doe
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 108 in → 106 out (90 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-pro:high PASS 100% 107 in → 100 out (93 reasoning)
Author: J. Doe
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 106 in → 16 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
deepseek/deepseek-v4-pro:low PASS 100% 105 in → 131 out (125 reasoning)
Author: J. Doe
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 299 in → 100 out (86 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
deepseek/deepseek-v4-pro:medium PASS 100% 107 in → 188 out (181 reasoning)
Author: J. Doe
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 107 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
deepseek/deepseek-v4-pro:none PASS 100% 109 in → 6 out (0 reasoning)
Author: J. Doe
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 297 in → 14 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 186 in → 297 out (290 reasoning)
Author: J. Doe
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 187 in → 186 out (169 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3-flash-preview:high PASS 100% 105 in → 196 out (191 reasoning)
Author: J. Doe
google/gemini-3-flash-preview:high__json_schema PASS 100% 164 in → 285 out (273 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3-flash-preview:low PASS 100% 104 in → 191 out (186 reasoning)
Author: J. Doe
google/gemini-3-flash-preview:low__json_schema PASS 100% 166 in → 336 out (325 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3-flash-preview:medium PASS 100% 106 in → 219 out (214 reasoning)
Author: J. Doe
google/gemini-3-flash-preview:medium__json_schema PASS 100% 106 in → 1,381 out (1,369 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3-flash-preview:xhigh PASS 100% 104 in → 177 out (172 reasoning)
Author: J. Doe
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 106 in → 411 out (399 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 107 in → 346 out (341 reasoning)
Author: J. Doe
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 106 in → 450 out (438 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 105 in → 121 out (116 reasoning)
Author: J. Doe
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 165 in → 137 out (120 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 106 in → 176 out (171 reasoning)
Author: J. Doe
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 106 in → 185 out (168 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 106 in → 248 out (243 reasoning)
Author: J. Doe
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 166 in → 481 out (470 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.1-pro-preview:high PASS 100% 103 in → 241 out (236 reasoning)
Author: J. Doe
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 165 in → 387 out (374 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
google/gemini-3.1-pro-preview:low PASS 100% 105 in → 259 out (254 reasoning)
Author: J. Doe
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 165 in → 148 out (131 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.1-pro-preview:medium PASS 100% 104 in → 294 out (289 reasoning)
Author: J. Doe
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 164 in → 108 out (96 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 106 in → 323 out (318 reasoning)
Author: J. Doe
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 166 in → 950 out (939 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.5-flash-lite:high PASS 100% 105 in → 338 out (333 reasoning)
Author: J. Doe
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 165 in → 575 out (563 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.5-flash-lite:low PASS 100% 105 in → 5 out (0 reasoning)
Author: J. Doe
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 164 in → 12 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.5-flash-lite:max PASS 100% 107 in → 299 out (294 reasoning)
Author: J. Doe
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 105 in → 499 out (482 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.5-flash-lite:medium PASS 100% 105 in → 262 out (257 reasoning)
Author: J. Doe
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 107 in → 405 out (393 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 105 in → 201 out (196 reasoning)
Author: J. Doe
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 165 in → 220 out (203 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.5-flash:high PASS 100% 105 in → 350 out (345 reasoning)
Author: J. Doe
google/gemini-3.5-flash:high__json_schema PASS 100% 164 in → 244 out (233 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.5-flash:low PASS 100% 104 in → 170 out (165 reasoning)
Author: J. Doe
google/gemini-3.5-flash:low__json_schema PASS 100% 164 in → 106 out (94 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.5-flash:medium PASS 100% 105 in → 352 out (347 reasoning)
Author: J. Doe
google/gemini-3.5-flash:medium__json_schema PASS 100% 164 in → 371 out (354 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.5-flash:xhigh PASS 100% 105 in → 337 out (332 reasoning)
Author: J. Doe
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 166 in → 455 out (443 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
google/gemini-3.6-flash:high PASS 100% 105 in → 311 out (306 reasoning)
Author: J. Doe
google/gemini-3.6-flash:high__json_schema PASS 100% 105 in → 600 out (589 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemini-3.6-flash:low PASS 100% 107 in → 87 out (82 reasoning)
Author: J. Doe
google/gemini-3.6-flash:low__json_schema PASS 100% 166 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.6-flash:max PASS 100% 105 in → 315 out (310 reasoning)
Author: J. Doe
google/gemini-3.6-flash:max__json_schema PASS 100% 165 in → 449 out (432 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.6-flash:medium PASS 100% 107 in → 270 out (265 reasoning)
Author: J. Doe
google/gemini-3.6-flash:medium__json_schema PASS 100% 106 in → 168 out (151 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemini-3.6-flash:xhigh PASS 100% 107 in → 308 out (303 reasoning)
Author: J. Doe
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 166 in → 481 out (470 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemma-4-26b-a4b-it:high PASS 100% 120 in → 169 out (137 reasoning)
Author: J. Doe
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 122 in → 246 out (204 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-26b-a4b-it:low PASS 100% 119 in → 248 out (232 reasoning)
Author: J. Doe
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 120 in → 257 out (216 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-26b-a4b-it:medium PASS 100% 120 in → 199 out (175 reasoning)
Author: J. Doe
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 119 in → 258 out (232 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-26b-a4b-it:none PASS 100% 115 in → 6 out (0 reasoning)
Author: J. Doe
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 118 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 122 in → 264 out (241 reasoning)
Author: J. Doe
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 121 in → 256 out (228 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-31b-it:high PASS 100% 121 in → 178 out (142 reasoning)
Author: J. Doe
google/gemma-4-31b-it:high__json_schema PASS 100% 120 in → 202 out (1 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-31b-it:low__json_schema PASS 100% 125 in → 171 out (158 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
google/gemma-4-31b-it:medium PASS 100% 122 in → 206 out (173 reasoning)
Author: J. Doe
google/gemma-4-31b-it:medium__json_schema PASS 100% 119 in → 167 out (130 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-31b-it:none PASS 100% 119 in → 6 out (0 reasoning)
Author: J. Doe
google/gemma-4-31b-it:none__json_schema PASS 100% 118 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
google/gemma-4-31b-it:xhigh PASS 100% 119 in → 194 out (1 reasoning)
Author: J. Doe
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 125 in → 178 out (165 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
ibm-granite/granite-4.1-8b PASS 100% 111 in → 6 out
Author: J. Doe
ibm-granite/granite-4.1-8b__json_schema PASS 100% 110 in → 16 out
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
inclusionai/ling-2.6-flash PASS 100% 126 in → 6 out
Author: J. Doe
inclusionai/ling-2.6-flash__json_schema PASS 100% 126 in → 17 out
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
liquid/lfm-2-24b-a2b PASS 100% 120 in → 12 out
```txt Author: J. Doe ```
minimax/minimax-m2.5:high PASS 100% 138 in → 182 out (189 reasoning)
Author: J. Doe
minimax/minimax-m2.5:high__json_schema PASS 100% 139 in → 196 out (199 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m2.5:low PASS 100% 138 in → 236 out (229 reasoning)
Author: J. Doe
minimax/minimax-m2.5:low__json_schema PASS 100% 138 in → 145 out (127 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m2.5:medium PASS 100% 138 in → 661 out (751 reasoning)
Author: J. Doe
minimax/minimax-m2.5:medium__json_schema PASS 100% 138 in → 209 out (194 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m2.5:xhigh PASS 100% 136 in → 179 out (172 reasoning)
Author: J. Doe
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 138 in → 210 out (193 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m2.7:high PASS 100% 137 in → 106 out (98 reasoning)
Author: J. Doe
minimax/minimax-m2.7:high__json_schema PASS 100% 272 in → 437 out (421 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m2.7:low PASS 100% 139 in → 192 out (202 reasoning)
Author: J. Doe
minimax/minimax-m2.7:low__json_schema PASS 100% 273 in → 240 out (228 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
minimax/minimax-m2.7:medium PASS 100% 139 in → 143 out (136 reasoning)
Author: J. Doe
minimax/minimax-m2.7:medium__json_schema PASS 100% 270 in → 344 out (332 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
minimax/minimax-m2.7:xhigh PASS 100% 136 in → 316 out (340 reasoning)
Author: J. Doe
minimax/minimax-m3:high PASS 100% 275 in → 75 out (74 reasoning)
Author: J. Doe
minimax/minimax-m3:high__json_schema PASS 100% 263 in → 15 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m3:low PASS 100% 276 in → 55 out (53 reasoning)
Author: J. Doe
minimax/minimax-m3:low__json_schema PASS 100% 224 in → 15 out (14 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
minimax/minimax-m3:medium PASS 100% 275 in → 98 out (99 reasoning)
Author: J. Doe
minimax/minimax-m3:medium__json_schema PASS 100% 264 in → 16 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
minimax/minimax-m3:xhigh PASS 100% 276 in → 95 out (98 reasoning)
Author: J. Doe
minimax/minimax-m3:xhigh__json_schema PASS 100% 262 in → 14 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
mistralai/devstral-2512 PASS 100% 110 in → 6 out
Author: J. Doe
mistralai/devstral-2512__json_schema PASS 100% 109 in → 13 out
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
mistralai/mistral-medium-3-5:high PASS 100% 119 in → 215 out (231 reasoning)
Author: J. Doe
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 110 in → 253 out (248 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
mistralai/mistral-medium-3-5:low PASS 100% 121 in → 265 out (266 reasoning)
Author: J. Doe
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 107 in → 191 out (185 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
mistralai/mistral-medium-3-5:medium PASS 100% 121 in → 341 out (341 reasoning)
Author: J. Doe
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 109 in → 215 out (213 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
mistralai/mistral-medium-3-5:none PASS 100% 120 in → 6 out (0 reasoning)
Author: J. Doe
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 107 in → 13 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 122 in → 282 out (286 reasoning)
Author: J. Doe
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 110 in → 194 out (204 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
moonshotai/kimi-k2.6:high__json_schema PASS 100% 288 in → 784 out (772 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
moonshotai/kimi-k2.6:low PASS 100% 107 in → 425 out (418 reasoning)
Author: J. Doe
moonshotai/kimi-k2.6:low__json_schema PASS 100% 288 in → 304 out (292 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 105 in → 281 out (265 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
moonshotai/kimi-k2.6:xhigh PASS 100% 106 in → 150 out (143 reasoning)
Author: J. Doe
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 191 in → 111 out (98 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
moonshotai/kimi-k2.7-code:low PASS 100% 105 in → 54 out (47 reasoning)
Author: J. Doe
moonshotai/kimi-k2.7-code:xhigh PASS 100% 106 in → 63 out (56 reasoning)
Author: J. Doe
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 106 in → 52 out (40 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
moonshotai/kimi-k3:high PASS 100% 181 in → 88 out (68 reasoning)
Author: J. Doe
moonshotai/kimi-k3:high__json_schema PASS 100% 184 in → 82 out (62 reasoning)
Author: J. Doe
moonshotai/kimi-k3:low PASS 100% 184 in → 81 out (61 reasoning)
Author: J. Doe
moonshotai/kimi-k3:low__json_schema PASS 100% 184 in → 83 out (60 reasoning)
``` Author: J. Doe ```
moonshotai/kimi-k3:max PASS 100% 183 in → 158 out (138 reasoning)
Author: J. Doe
moonshotai/kimi-k3:max__json_schema PASS 100% 183 in → 90 out (70 reasoning)
Author: J. Doe
moonshotai/kimi-k3:medium PASS 100% 182 in → 82 out (62 reasoning)
Author: J. Doe
moonshotai/kimi-k3:medium__json_schema PASS 100% 182 in → 85 out (65 reasoning)
Author: J. Doe
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 182 in → 66 out (46 reasoning)
Author: J. Doe
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 123 in → 174 out (182 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 123 in → 126 out (127 reasoning)
Author: J. Doe
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 122 in → 140 out (134 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 122 in → 191 out (190 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 122 in → 6 out (0 reasoning)
Author: J. Doe
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 123 in → 16 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 123 in → 147 out (157 reasoning)
Author: J. Doe
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 123 in → 102 out (98 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 119 in → 37 out (32 reasoning)
Author: J. Doe
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 122 in → 65 out (48 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 121 in → 46 out (43 reasoning)
Author: J. Doe
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 123 in → 65 out (48 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 120 in → 46 out (44 reasoning)
Author: J. Doe
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 122 in → 51 out (34 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 123 in → 6 out (0 reasoning)
Author: J. Doe
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 121 in → 16 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 123 in → 44 out (44 reasoning)
Author: J. Doe
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 123 in → 47 out (30 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
openai/gpt-5.4-mini:high PASS 100% 102 in → 51 out (40 reasoning)
Author: J. Doe
openai/gpt-5.4-mini:high__json_schema PASS 100% 139 in → 114 out (94 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4-mini:low PASS 100% 104 in → 22 out (11 reasoning)
Author: J. Doe
openai/gpt-5.4-mini:low__json_schema PASS 100% 136 in → 31 out (11 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4-mini:medium PASS 100% 101 in → 39 out (28 reasoning)
Author: J. Doe
openai/gpt-5.4-mini:medium__json_schema PASS 100% 136 in → 92 out (72 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4-mini:none PASS 100% 103 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.4-mini:none__json_schema PASS 100% 137 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4-mini:xhigh PASS 100% 103 in → 69 out (58 reasoning)
Author: J. Doe
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 139 in → 201 out (181 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4-nano:high PASS 100% 101 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.4-nano:high__json_schema PASS 100% 136 in → 19 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4-nano:low PASS 100% 104 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.4-nano:low__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4-nano:medium PASS 100% 102 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.4-nano:medium__json_schema PASS 100% 137 in → 19 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4-nano:none PASS 100% 103 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.4-nano:none__json_schema PASS 100% 136 in → 19 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 100 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 138 in → 89 out (68 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4:high PASS 100% 101 in → 42 out (31 reasoning)
Author: J. Doe
openai/gpt-5.4:high__json_schema PASS 100% 137 in → 101 out (80 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.4:low PASS 100% 102 in → 21 out (10 reasoning)
Author: J. Doe
openai/gpt-5.4:low__json_schema PASS 100% 137 in → 29 out (9 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4:medium PASS 100% 103 in → 36 out (25 reasoning)
Author: J. Doe
openai/gpt-5.4:medium__json_schema PASS 100% 139 in → 32 out (12 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4:none PASS 100% 102 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.4:none__json_schema PASS 100% 137 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.4:xhigh PASS 100% 102 in → 50 out (39 reasoning)
Author: J. Doe
openai/gpt-5.4:xhigh__json_schema PASS 100% 138 in → 127 out (107 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.5:high PASS 100% 101 in → 35 out (24 reasoning)
Author: J. Doe
openai/gpt-5.5:high__json_schema PASS 100% 138 in → 536 out (516 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.5:low PASS 100% 102 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.5:low__json_schema PASS 100% 137 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.5:medium PASS 100% 100 in → 22 out (11 reasoning)
Author: J. Doe
openai/gpt-5.5:medium__json_schema PASS 100% 137 in → 19 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-5.5:none PASS 100% 103 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.5:none__json_schema PASS 100% 137 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.5:xhigh PASS 100% 101 in → 80 out (69 reasoning)
Author: J. Doe
openai/gpt-5.5:xhigh__json_schema PASS 100% 138 in → 1,054 out (1,034 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:high PASS 100% 103 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:high__json_schema PASS 100% 137 in → 67 out (47 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:low PASS 100% 101 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:low__json_schema PASS 100% 138 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:max PASS 100% 102 in → 39 out (28 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:max__json_schema PASS 100% 137 in → 187 out (167 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:medium PASS 100% 101 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:medium__json_schema PASS 100% 137 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:none PASS 100% 100 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:none__json_schema PASS 100% 139 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-luna:xhigh PASS 100% 103 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 138 in → 97 out (77 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:high PASS 100% 101 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:high__json_schema PASS 100% 138 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:low PASS 100% 102 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:low__json_schema PASS 100% 137 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:max PASS 100% 102 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:max__json_schema PASS 100% 139 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:medium PASS 100% 101 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:medium__json_schema PASS 100% 137 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:none PASS 100% 103 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:none__json_schema PASS 100% 138 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-sol:xhigh PASS 100% 101 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 138 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:high PASS 100% 102 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:high__json_schema PASS 100% 138 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:low PASS 100% 102 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:low__json_schema PASS 100% 138 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:max PASS 100% 100 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:max__json_schema PASS 100% 138 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:medium PASS 100% 101 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:medium__json_schema PASS 100% 138 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:none PASS 100% 103 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:none__json_schema PASS 100% 138 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-5.6-terra:xhigh PASS 100% 101 in → 9 out (0 reasoning)
Author: J. Doe
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 138 in → 18 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-oss-120b:high PASS 100% 166 in → 206 out (225 reasoning)
Author: J. Doe
openai/gpt-oss-120b:high__json_schema PASS 100% 163 in → 206 out (182 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
openai/gpt-oss-120b:low PASS 100% 150 in → 45 out (32 reasoning)
Author: J. Doe
openai/gpt-oss-120b:low__json_schema PASS 100% 149 in → 52 out (32 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe\n"}
openai/gpt-oss-120b:medium PASS 100% 162 in → 112 out (99 reasoning)
Author: J. Doe
openai/gpt-oss-120b:medium__json_schema PASS 100% 164 in → 104 out (79 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
openai/gpt-oss-120b:xhigh PASS 100% 164 in → 180 out (166 reasoning)
Author: J. Doe
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 162 in → 131 out (124 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe\n"}
openai/gpt-oss-20b:high PASS 100% 151 in → 180 out (185 reasoning)
Author: J. Doe
openai/gpt-oss-20b:high__json_schema PASS 100% 266 in → 400 out (377 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-oss-20b:low PASS 100% 150 in → 30 out (15 reasoning)
Author: J. Doe
openai/gpt-oss-20b:low__json_schema PASS 100% 166 in → 29 out (8 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
openai/gpt-oss-20b:medium PASS 100% 151 in → 82 out (73 reasoning)
Author: J. Doe
openai/gpt-oss-20b:xhigh PASS 100% 149 in → 207 out (223 reasoning)
Author: J. Doe
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 149 in → 162 out (158 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
poolside/laguna-m.1:high PASS 100% 116 in → 377 out (370 reasoning)
Author: J. Doe
poolside/laguna-m.1:high__json_schema PASS 100% 115 in → 321 out (305 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
poolside/laguna-m.1:low PASS 100% 114 in → 439 out (433 reasoning)
Author: J. Doe
poolside/laguna-m.1:low__json_schema PASS 100% 114 in → 320 out (308 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
poolside/laguna-m.1:medium PASS 100% 115 in → 200 out (193 reasoning)
Author: J. Doe
poolside/laguna-m.1:medium__json_schema PASS 100% 113 in → 391 out (375 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-m.1:none PASS 100% 114 in → 7 out (0 reasoning)
Author: J. Doe
poolside/laguna-m.1:none__json_schema PASS 100% 115 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-m.1:xhigh PASS 100% 112 in → 176 out (168 reasoning)
Author: J. Doe
poolside/laguna-m.1:xhigh__json_schema PASS 100% 116 in → 277 out (261 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-xs-2.1:high PASS 100% 113 in → 335 out (329 reasoning)
Author: J. Doe
poolside/laguna-xs-2.1:high__json_schema PASS 100% 115 in → 361 out (345 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-xs-2.1:low PASS 100% 115 in → 265 out (260 reasoning)
Author: J. Doe
poolside/laguna-xs-2.1:low__json_schema PASS 100% 115 in → 345 out (329 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-xs-2.1:medium PASS 100% 114 in → 364 out (359 reasoning)
Author: J. Doe
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 115 in → 242 out (224 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
poolside/laguna-xs-2.1:none PASS 100% 115 in → 5 out (0 reasoning)
Author: J. Doe
poolside/laguna-xs-2.1:none__json_schema PASS 100% 115 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-xs-2.1:xhigh PASS 100% 114 in → 308 out (303 reasoning)
Author: J. Doe
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 114 in → 213 out (196 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
poolside/laguna-xs.2:high PASS 100% 153 in → 193 out (186 reasoning)
Author: J. Doe
poolside/laguna-xs.2:high__json_schema PASS 100% 152 in → 232 out (221 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
poolside/laguna-xs.2:low PASS 100% 153 in → 201 out (194 reasoning)
Author: J. Doe
poolside/laguna-xs.2:low__json_schema PASS 100% 148 in → 203 out (191 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
poolside/laguna-xs.2:medium PASS 100% 151 in → 191 out (184 reasoning)
Author: J. Doe
poolside/laguna-xs.2:medium__json_schema PASS 100% 152 in → 177 out (164 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
poolside/laguna-xs.2:none PASS 100% 152 in → 7 out (0 reasoning)
Author: J. Doe
poolside/laguna-xs.2:none__json_schema PASS 100% 153 in → 12 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
poolside/laguna-xs.2:xhigh PASS 100% 152 in → 198 out (190 reasoning)
Author: J. Doe
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 151 in → 205 out (193 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
qwen/qwen3.6-27b:high PASS 100% 112 in → 684 out (688 reasoning)
Author: J. Doe
qwen/qwen3.6-27b:high__json_schema PASS 100% 112 in → 391 out (383 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-27b:low PASS 100% 109 in → 891 out (881 reasoning)
Author: J. Doe
qwen/qwen3.6-27b:low__json_schema PASS 100% 108 in → 425 out (1 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-27b:medium PASS 100% 112 in → 903 out (909 reasoning)
Author: J. Doe
qwen/qwen3.6-27b:none PASS 100% 113 in → 6 out (0 reasoning)
Author: J. Doe
qwen/qwen3.6-27b:none__json_schema PASS 100% 113 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-27b:xhigh PASS 100% 111 in → 603 out (600 reasoning)
Author: J. Doe
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 112 in → 952 out (1 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-35b-a3b:high PASS 100% 111 in → 840 out (830 reasoning)
Author: J. Doe
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 110 in → 854 out (812 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-35b-a3b:low PASS 100% 111 in → 890 out (880 reasoning)
Author: J. Doe
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 111 in → 893 out (874 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 110 in → 910 out (894 reasoning)
Author: J. Doe
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 112 in → 729 out (711 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-35b-a3b:none PASS 100% 112 in → 6 out (0 reasoning)
Author: J. Doe
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 112 in → 17 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 111 in → 862 out (854 reasoning)
Author: J. Doe
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 111 in → 868 out (834 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
qwen/qwen3.6-flash:high PASS 100% 111 in → 696 out (686 reasoning)
Author: J. Doe
qwen/qwen3.6-flash:low PASS 100% 110 in → 883 out (873 reasoning)
Author: J. Doe
qwen/qwen3.6-flash:medium PASS 100% 110 in → 981 out (971 reasoning)
Author: J. Doe
qwen/qwen3.6-flash:none PASS 100% 113 in → 5 out (0 reasoning)
Author: J. Doe
qwen/qwen3.6-flash:xhigh PASS 100% 112 in → 801 out (791 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:high PASS 100% 110 in → 850 out (840 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:low PASS 100% 108 in → 313 out (303 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:max PASS 100% 110 in → 922 out (912 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:medium PASS 100% 111 in → 381 out (371 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:none PASS 100% 112 in → 5 out (0 reasoning)
Author: J. Doe
qwen/qwen3.7-flash:xhigh PASS 100% 109 in → 288 out (278 reasoning)
Author: J. Doe
qwen/qwen3.7-max:high PASS 100% 111 in → 418 out (408 reasoning)
Author: J. Doe
qwen/qwen3.7-max:low PASS 100% 112 in → 263 out (253 reasoning)
Author: J. Doe
qwen/qwen3.7-max:medium PASS 100% 111 in → 105 out (95 reasoning)
Author: J. Doe
qwen/qwen3.7-max:none PASS 100% 114 in → 5 out (0 reasoning)
Author: J. Doe
qwen/qwen3.7-max:xhigh PASS 100% 110 in → 324 out (314 reasoning)
Author: J. Doe
qwen/qwen3.7-plus:high PASS 100% 112 in → 420 out (410 reasoning)
Author: J. Doe
qwen/qwen3.7-plus:low PASS 100% 110 in → 630 out (620 reasoning)
Author: J. Doe
qwen/qwen3.7-plus:medium PASS 100% 111 in → 478 out (468 reasoning)
Author: J. Doe
qwen/qwen3.7-plus:none PASS 100% 113 in → 5 out (0 reasoning)
Author: J. Doe
qwen/qwen3.7-plus:xhigh PASS 100% 111 in → 405 out (395 reasoning)
Author: J. Doe
tencent/hy3:high PASS 100% 113 in → 375 out (368 reasoning)
Author: J. Doe
tencent/hy3:high__json_schema PASS 100% 114 in → 456 out (455 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
tencent/hy3:low PASS 100% 114 in → 279 out (272 reasoning)
Author: J. Doe
tencent/hy3:low__json_schema PASS 100% 112 in → 272 out (270 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
tencent/hy3:medium PASS 100% 114 in → 367 out (386 reasoning)
Author: J. Doe
tencent/hy3:medium__json_schema PASS 100% 114 in → 388 out (390 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
tencent/hy3:none PASS 100% 116 in → 6 out (0 reasoning)
Author: J. Doe
tencent/hy3:none__json_schema PASS 100% 116 in → 16 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
tencent/hy3:xhigh PASS 100% 114 in → 618 out (611 reasoning)
Author: J. Doe
tencent/hy3:xhigh__json_schema PASS 100% 113 in → 508 out (517 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n" }
thinkingmachines/inkling-small:high PASS 100% 112 in → 181 out (187 reasoning)
Author: J. Doe
thinkingmachines/inkling-small:high__json_schema PASS 100% 109 in → 121 out (105 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
thinkingmachines/inkling-small:low PASS 100% 112 in → 23 out (12 reasoning)
Author: J. Doe
thinkingmachines/inkling-small:low__json_schema PASS 100% 109 in → 36 out (17 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
thinkingmachines/inkling-small:max PASS 100% 111 in → 160 out (162 reasoning)
Author: J. Doe
thinkingmachines/inkling-small:max__json_schema PASS 100% 112 in → 83 out (64 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
thinkingmachines/inkling-small:medium PASS 100% 110 in → 90 out (79 reasoning)
Author: J. Doe
thinkingmachines/inkling-small:medium__json_schema PASS 100% 110 in → 58 out (39 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
thinkingmachines/inkling-small:xhigh PASS 100% 111 in → 164 out (168 reasoning)
Author: J. Doe
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 110 in → 146 out (128 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
thinkingmachines/inkling:high PASS 100% 111 in → 158 out (159 reasoning)
Author: J. Doe
thinkingmachines/inkling:low PASS 100% 111 in → 30 out (18 reasoning)
Author: J. Doe
thinkingmachines/inkling:max PASS 100% 110 in → 146 out (135 reasoning)
Author: J. Doe
thinkingmachines/inkling:max__json_schema PASS 100% 113 in → 154 out (138 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content": "Author: J. Doe"}
thinkingmachines/inkling:medium PASS 100% 110 in → 35 out (23 reasoning)
Author: J. Doe
thinkingmachines/inkling:none PASS 100% 108 in → 8 out (0 reasoning)
Author: J. Doe
thinkingmachines/inkling:xhigh PASS 100% 111 in → 112 out (100 reasoning)
Author: J. Doe
x-ai/grok-4.3:high PASS 100% 278 in → 184 out (179 reasoning)
Author: J. Doe
x-ai/grok-4.3:high__json_schema PASS 100% 336 in → 495 out (481 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
x-ai/grok-4.3:low PASS 100% 288 in → 244 out (239 reasoning)
Author: J. Doe
x-ai/grok-4.3:low__json_schema PASS 100% 344 in → 292 out (278 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
x-ai/grok-4.3:max PASS 100% 279 in → 147 out (142 reasoning)
Author: J. Doe
x-ai/grok-4.3:max__json_schema PASS 100% 338 in → 308 out (298 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
x-ai/grok-4.3:medium PASS 100% 284 in → 309 out (304 reasoning)
Author: J. Doe
x-ai/grok-4.3:medium__json_schema PASS 100% 344 in → 545 out (531 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
x-ai/grok-4.3:none PASS 100% 280 in → 5 out (0 reasoning)
Author: J. Doe
x-ai/grok-4.3:none__json_schema PASS 100% 334 in → 10 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
x-ai/grok-4.3:xhigh PASS 100% 281 in → 251 out (246 reasoning)
Author: J. Doe
x-ai/grok-4.3:xhigh__json_schema PASS 100% 337 in → 320 out (310 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
x-ai/grok-4.5:high PASS 100% 302 in → 163 out (158 reasoning)
Author: J. Doe
x-ai/grok-4.5:high__json_schema PASS 100% 370 in → 455 out (443 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
x-ai/grok-4.5:low PASS 100% 304 in → 152 out (147 reasoning)
Author: J. Doe
x-ai/grok-4.5:low__json_schema PASS 100% 371 in → 232 out (220 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
x-ai/grok-4.5:max PASS 100% 304 in → 175 out (170 reasoning)
Author: J. Doe
x-ai/grok-4.5:max__json_schema PASS 100% 372 in → 210 out (198 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
x-ai/grok-4.5:medium PASS 100% 304 in → 201 out (196 reasoning)
Author: J. Doe
x-ai/grok-4.5:medium__json_schema PASS 100% 371 in → 400 out (388 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
x-ai/grok-4.5:xhigh PASS 100% 303 in → 158 out (153 reasoning)
Author: J. Doe
x-ai/grok-4.5:xhigh__json_schema PASS 100% 372 in → 360 out (348 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
z-ai/glm-4.7-flash:high PASS 100% 101 in → 346 out (351 reasoning)
Author: J. Doe
z-ai/glm-4.7-flash:low PASS 100% 102 in → 362 out (367 reasoning)
Author: J. Doe
z-ai/glm-4.7-flash:medium PASS 100% 102 in → 440 out (447 reasoning)
Author: J. Doe
z-ai/glm-4.7-flash:none PASS 100% 102 in → 5 out (0 reasoning)
Author: J. Doe
z-ai/glm-4.7-flash:none__json_schema PASS 100% 101 in → 15 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
z-ai/glm-4.7-flash:xhigh PASS 100% 102 in → 383 out (376 reasoning)
Author: J. Doe
z-ai/glm-5.2:high PASS 100% 108 in → 294 out (310 reasoning)
Author: J. Doe
z-ai/glm-5.2:high__json_schema PASS 100% 108 in → 198 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe\n"}
z-ai/glm-5.2:low PASS 100% 110 in → 91 out (84 reasoning)
Author: J. Doe
z-ai/glm-5.2:low__json_schema PASS 100% 286 in → 218 out (201 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
z-ai/glm-5.2:medium PASS 100% 109 in → 271 out (256 reasoning)
Author: J. Doe
z-ai/glm-5.2:medium__json_schema PASS 100% 109 in → 227 out (211 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
z-ai/glm-5.2:none PASS 100% 103 in → 6 out (0 reasoning)
Author: J. Doe
z-ai/glm-5.2:none__json_schema PASS 100% 279 in → 11 out (0 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) {"resolved_content":"Author: J. Doe"}
z-ai/glm-5.2:xhigh PASS 100% 106 in → 291 out (284 reasoning)
Author: J. Doe
z-ai/glm-5.2:xhigh__json_schema PASS 100% 109 in → 100 out (97 reasoning)
Author: J. Doe
JSON Schema Structured Output
(raw) { "resolved_content": "Author: J. Doe" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:xhigh FAIL 0% 106 in → 375 out (423 reasoning)
Author:J. Doe
Failure: Content mismatch: ['author.txt']
google/gemma-4-31b-it:low FAIL 0% 119 in → 153 out (1 reasoning)
C Author: J. Doe ``` Wait, the user requested "ONLY the resolved file content." Let me re-read. Input: Base: Author: John Incoming (fullname): Author: John Smith Current (main): Author: J. Doe Instruction: Keep current branch value. Resolved content: Author: J. Doe Wait, I started the response with "C\nC\n". That was a typo/brain-glitch. I must provide ONLY the content. Final check: Current branch (HEAD) = Author: J. Doe Incoming = Author: John Smith Keep current. Result: Author: J. DoeAuthor: J. Doe
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output: ```txt Author: J. Doe ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:high FAIL 0% 105 in → 214 out (207 reasoning)
Author: J. Doe
Failure: Content mismatch: ['author.txt']
moonshotai/kimi-k2.6:medium FAIL 0% 107 in → 195 out (188 reasoning)
Author: J. Doe
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
moonshotai/kimi-k2.7-code:high FAIL 0% 106 in → 100 out (93 reasoning)
Author: J. Doe
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:medium FAIL 0% 107 in → 86 out (79 reasoning)
Author: J. Doe
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
Failure: OpenAI-compatible response contained provider error: Provider returned error
nvidia/nemotron-3-nano-30b-a3b:high FAIL 0% 123 in → 71 out (71 reasoning)
Author:J. Doe
Failure: Content mismatch: ['author.txt']
nvidia/nemotron-3-nano-30b-a3b:medium FAIL 0% 123 in → 93 out (93 reasoning)
Author:J. Doe
Failure: Content mismatch: ['author.txt']
openai/gpt-oss-20b:medium__json_schema FAIL 0% 149 in → 202 out (165 reasoning)
*** Removed --- --- author.txt <<<<<<< HEAD Author: J. Doe ======= Author: John Smith >>>>>>> 1285439 (Use full name)
JSON Schema Structured Output
(raw) { "resolved_content": "*** Removed ---\n--- author.txt\n<<<<<<< HEAD\nAuthor: J. Doe\n=======\nAuthor: John Smith\n>>>>>>> 1285439 (Use full name)" }
Failure: Expected file heading before file content
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c881ba0b-70e6-9e81-9ffc-5c6ee3ddbdac","request_id":"c881ba0b-70e6-9e81-9ffc-5c6ee3ddbdac"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2241344b-d9bb-9806-b1cc-d5b5beee4e02","request_id":"2241344b-d9bb-9806-b1cc-d5b5beee4e02"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-81073662-a85b-9311-91b2-e78f07353cb5","request_id":"81073662-a85b-9311-91b2-e78f07353cb5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-633b99f1-d82b-959e-aa2a-b51f56604bdb","request_id":"633b99f1-d82b-959e-aa2a-b51f56604bdb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-557f4a33-5b68-9baa-b3d0-07d59b739c11","request_id":"557f4a33-5b68-9baa-b3d0-07d59b739c11"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-99cb029f-080e-9e0c-bb11-d396b259e73c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-551e8024-1f8e-9992-9d61-18fc600aff95"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a6435480-a768-9898-b8a6-d55f74520201"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-510a8470-b9b2-9569-8c34-3e29cbaf37ff"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5cd64238-5ca9-95c7-93dc-2f177b345f47"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0b36e457-976c-9ce1-b633-f01343c8907c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-13010491-31f9-9b71-bf2d-5cc0ae1e3812","request_id":"13010491-31f9-9b71-bf2d-5cc0ae1e3812"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0db17cfe-ce2e-945c-b714-8b52e18117e4","request_id":"0db17cfe-ce2e-945c-b714-8b52e18117e4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b2409164-213f-92af-962f-a7ffca4dafd8","request_id":"b2409164-213f-92af-962f-a7ffca4dafd8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1f9a8589-9778-9f02-9f7c-fc6fb9e32cd4","request_id":"1f9a8589-9778-9f02-9f7c-fc6fb9e32cd4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-52cacaba-ffef-928b-a5c7-b963a024d54d","request_id":"52cacaba-ffef-928b-a5c7-b963a024d54d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-acfd51e5-dc58-9e4f-9bd8-92ffc02feb4c","request_id":"acfd51e5-dc58-9e4f-9bd8-92ffc02feb4c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a50ed3b3-5a2e-9685-8754-db4aa453b2c2","request_id":"a50ed3b3-5a2e-9685-8754-db4aa453b2c2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b1f5ea99-e929-9eb1-a625-dba731a9f5ee","request_id":"b1f5ea99-e929-9eb1-a625-dba731a9f5ee"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-095a5d43-5174-9b67-a079-d76a6a85ffba","request_id":"095a5d43-5174-9b67-a079-d76a6a85ffba"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-06cd3018-ccf3-9d48-bc1d-ed839f81586c","request_id":"06cd3018-ccf3-9d48-bc1d-ed839f81586c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 110 in → 8 out (0 reasoning)
Author: J. Doe<|end_message|>
Failure: Content mismatch: ['author.txt']
Invalid JSON. Output: Author: J. Doe{ "resolved_content": "Author: J. Doe" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Author: J. Doe{ "resolved_content": "Author: J. Doe" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Author: J. Doe
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)