Three-way: multiple files with conflicting and non-conflicting changes
Tests ability to handle cherry-pick where some files conflict and others merge cleanly. Evaluates understanding of partial-conflict scenarios.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Title: My Project' > project.txt
  5. 05 echo 'File A content' > file_a.txt
  6. 06 echo 'File B content' > file_b.txt
  7. 07 git add project.txt file_a.txt file_b.txt
  8. 08 git commit -m 'Initial project'
  9. 09 git checkout -b branch-a
  10. 10 echo 'Title: My Project v2' > project.txt
  11. 11 echo 'File A updated' > file_a.txt
  12. 12 git add project.txt file_a.txt
  13. 13 git commit -m 'Update title and file A'
  14. 14 git checkout main
  15. 15 echo 'Title: My Project v3' > project.txt
  16. 16 echo 'File B updated' > file_b.txt
  17. 17 git add project.txt file_b.txt
  18. 18 git commit -m 'Update title and file B'
  19. 19 git cherry-pick $(git rev-parse branch-a)
Prompt
Resolve the cherry-pick conflict in project.txt. The incoming commit set Title: My Project v2. The current branch (main) has Title: My Project v3. Both branches also updated different files (file_a and file_b) without conflicts. For the project.txt conflict, the higher version is more appropriate. Provide ONLY the resolved project.txt content.
Expected
Title: My Project v3

Scoped model quality, cost, API time, and token usage for cherry_pick/f008.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high__json_schema PASS 100% 412 in → 31 out (5 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-fable-5:low PASS 100% 192 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-fable-5:low__json_schema PASS 100% 412 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-fable-5:max PASS 100% 192 in → 296 out (50 reasoning)
``` Title: My Project v3 ```
anthropic/claude-fable-5:max__json_schema PASS 100% 412 in → 156 out (66 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-fable-5:medium__json_schema PASS 100% 410 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 411 in → 43 out (16 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-haiku-4.5:high PASS 100% 167 in → 157 out (143 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 334 in → 286 out (263 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-haiku-4.5:low PASS 100% 167 in → 160 out (146 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 335 in → 168 out (153 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-haiku-4.5:medium PASS 100% 166 in → 130 out (109 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 334 in → 192 out (181 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-haiku-4.5:none PASS 100% 135 in → 13 out (0 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 303 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 167 in → 288 out (266 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 333 in → 225 out (200 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-opus-4.6:high PASS 100% 137 in → 14 out (0 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 304 in → 66 out (29 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.6:low PASS 100% 135 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.6:low__json_schema PASS 100% 304 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 305 in → 109 out (79 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.6:medium PASS 100% 137 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 304 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.6:none PASS 100% 136 in → 13 out (0 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 305 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 137 in → 14 out (0 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 305 in → 66 out (29 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:high PASS 100% 196 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:high__json_schema PASS 100% 418 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:low PASS 100% 198 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:low__json_schema PASS 100% 417 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:max PASS 100% 196 in → 159 out (35 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:max__json_schema PASS 100% 417 in → 144 out (52 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:medium PASS 100% 197 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 418 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:none PASS 100% 197 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:none__json_schema PASS 100% 415 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 197 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 417 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.8:high PASS 100% 194 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:high__json_schema PASS 100% 412 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.8:low PASS 100% 193 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:low__json_schema PASS 100% 412 in → 23 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.8:max PASS 100% 192 in → 496 out (67 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 412 in → 188 out (66 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-opus-4.8:medium PASS 100% 193 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 413 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.8:none PASS 100% 195 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:none__json_schema PASS 100% 411 in → 22 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 191 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 412 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 305 in → 89 out (56 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 138 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 304 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 306 in → 132 out (97 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 136 in → 63 out (31 reasoning)
``` Title: My Project v3 ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 306 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 135 in → 13 out (0 reasoning)
``` Title: My Project v3 ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 306 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 136 in → 109 out (73 reasoning)
``` Title: My Project v3 ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 305 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-sonnet-5:high PASS 100% 191 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:high__json_schema PASS 100% 413 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-sonnet-5:low PASS 100% 192 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:low__json_schema PASS 100% 410 in → 23 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-sonnet-5:medium PASS 100% 192 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 410 in → 23 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-5:none PASS 100% 192 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:none__json_schema PASS 100% 410 in → 22 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 191 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 413 in → 23 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 122 in → 418 out (411 reasoning)
Title: My Project v3
arcee-ai/trinity-large-thinking:low PASS 100% 123 in → 429 out (423 reasoning)
Title: My Project v3
arcee-ai/trinity-large-thinking:medium PASS 100% 123 in → 375 out (368 reasoning)
Title: My Project v3
arcee-ai/trinity-large-thinking:xhigh PASS 100% 121 in → 385 out (377 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:high PASS 100% 122 in → 280 out (311 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:high__json_schema PASS 100% 121 in → 295 out (328 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
arcee-ai/trinity-mini:low PASS 100% 121 in → 287 out (325 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:low__json_schema PASS 100% 122 in → 301 out (328 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content":"Title: My Project v3" }
arcee-ai/trinity-mini:medium PASS 100% 122 in → 312 out (352 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:medium__json_schema PASS 100% 122 in → 303 out (344 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
arcee-ai/trinity-mini:xhigh PASS 100% 122 in → 261 out (291 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 123 in → 254 out (285 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 201 in → 87 out (99 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 123 in → 265 out (286 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 128 in → 191 out (200 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 122 in → 418 out (459 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 200 in → 68 out (77 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 214 in → 238 out (244 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 128 in → 97 out (106 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 123 in → 33 out (26 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 121 in → 7 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 123 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 123 in → 249 out (266 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 200 in → 107 out (104 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash:high PASS 100% 123 in → 76 out (68 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 124 in → 313 out (351 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
deepseek/deepseek-v4-flash:low PASS 100% 124 in → 90 out (82 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 125 in → 151 out (131 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
deepseek/deepseek-v4-flash:medium PASS 100% 123 in → 102 out (111 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 126 in → 92 out (72 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
deepseek/deepseek-v4-flash:none PASS 100% 122 in → 7 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 123 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 202 in → 152 out (144 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:high PASS 100% 123 in → 122 out (114 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 123 in → 123 out (113 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-pro:low PASS 100% 121 in → 232 out (225 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 314 in → 139 out (124 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
deepseek/deepseek-v4-pro:medium PASS 100% 121 in → 160 out (152 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 121 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-pro:none PASS 100% 122 in → 7 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 121 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 392 in → 202 out (187 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3-flash-preview:high PASS 100% 125 in → 156 out (150 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:high__json_schema PASS 100% 184 in → 277 out (264 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3-flash-preview:low PASS 100% 125 in → 207 out (201 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:low__json_schema PASS 100% 184 in → 311 out (299 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3-flash-preview:medium PASS 100% 125 in → 231 out (225 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:medium__json_schema PASS 100% 185 in → 590 out (578 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3-flash-preview:xhigh PASS 100% 126 in → 234 out (228 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 185 in → 276 out (263 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 127 in → 341 out (335 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 127 in → 420 out (408 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 124 in → 120 out (114 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 127 in → 150 out (132 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 123 in → 139 out (133 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 123 in → 229 out (216 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 124 in → 241 out (235 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 126 in → 1,180 out (1,167 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-pro-preview:high PASS 100% 125 in → 158 out (152 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 186 in → 225 out (212 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-pro-preview:low PASS 100% 126 in → 148 out (142 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 183 in → 98 out (85 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-pro-preview:medium PASS 100% 126 in → 302 out (296 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 185 in → 95 out (80 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 124 in → 332 out (326 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 126 in → 474 out (461 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.5-flash-lite:high PASS 100% 126 in → 360 out (354 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 126 in → 621 out (609 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.5-flash-lite:low PASS 100% 126 in → 6 out (0 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 127 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.5-flash-lite:max PASS 100% 126 in → 285 out (279 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 184 in → 560 out (547 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.5-flash-lite:medium PASS 100% 123 in → 239 out (233 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 126 in → 437 out (424 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 125 in → 324 out (318 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 186 in → 631 out (618 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.5-flash:high PASS 100% 126 in → 397 out (391 reasoning)
Title: My Project v3
google/gemini-3.5-flash:high__json_schema PASS 100% 186 in → 409 out (397 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.5-flash:low PASS 100% 124 in → 207 out (201 reasoning)
Title: My Project v3
google/gemini-3.5-flash:low__json_schema PASS 100% 183 in → 301 out (286 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
google/gemini-3.5-flash:medium PASS 100% 124 in → 337 out (331 reasoning)
Title: My Project v3
google/gemini-3.5-flash:medium__json_schema PASS 100% 186 in → 658 out (638 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
google/gemini-3.5-flash:xhigh PASS 100% 126 in → 366 out (360 reasoning)
Title: My Project v3
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 185 in → 516 out (503 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.6-flash:high PASS 100% 126 in → 315 out (309 reasoning)
Title: My Project v3
google/gemini-3.6-flash:high__json_schema PASS 100% 185 in → 514 out (502 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.6-flash:low PASS 100% 127 in → 195 out (189 reasoning)
Title: My Project v3
google/gemini-3.6-flash:low__json_schema PASS 100% 127 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.6-flash:max PASS 100% 124 in → 273 out (267 reasoning)
Title: My Project v3
google/gemini-3.6-flash:max__json_schema PASS 100% 126 in → 886 out (874 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.6-flash:medium PASS 100% 127 in → 359 out (353 reasoning)
Title: My Project v3
google/gemini-3.6-flash:medium__json_schema PASS 100% 183 in → 212 out (194 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.6-flash:xhigh PASS 100% 124 in → 272 out (266 reasoning)
Title: My Project v3
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 185 in → 347 out (329 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:high PASS 100% 139 in → 194 out (157 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 139 in → 166 out (133 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:low PASS 100% 139 in → 172 out (145 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 142 in → 182 out (136 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:medium PASS 100% 141 in → 159 out (152 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 140 in → 215 out (169 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemma-4-26b-a4b-it:none PASS 100% 138 in → 7 out (0 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 139 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 141 in → 175 out (168 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 139 in → 201 out (158 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-31b-it:high PASS 100% 142 in → 160 out (126 reasoning)
Title: My Project v3
google/gemma-4-31b-it:high__json_schema PASS 100% 140 in → 182 out (140 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemma-4-31b-it:low PASS 100% 139 in → 162 out (128 reasoning)
Title: My Project v3
google/gemma-4-31b-it:low__json_schema PASS 100% 142 in → 154 out (140 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemma-4-31b-it:medium PASS 100% 143 in → 193 out (186 reasoning)
Title: My Project v3
google/gemma-4-31b-it:medium__json_schema PASS 100% 141 in → 154 out (123 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-31b-it:none PASS 100% 139 in → 7 out (0 reasoning)
Title: My Project v3
google/gemma-4-31b-it:none__json_schema PASS 100% 137 in → 14 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemma-4-31b-it:xhigh PASS 100% 141 in → 145 out (117 reasoning)
Title: My Project v3
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 141 in → 162 out (119 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
ibm-granite/granite-4.1-8b PASS 100% 130 in → 7 out
Title: My Project v3
ibm-granite/granite-4.1-8b__json_schema PASS 100% 131 in → 17 out
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
inclusionai/ling-2.6-flash PASS 100% 141 in → 7 out
Title: My Project v3
inclusionai/ling-2.6-flash__json_schema PASS 100% 140 in → 18 out
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.5:high PASS 100% 159 in → 127 out (122 reasoning)
Title: My Project v3
minimax/minimax-m2.5:high__json_schema PASS 100% 153 in → 182 out (165 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.5:low PASS 100% 154 in → 156 out (149 reasoning)
Title: My Project v3
minimax/minimax-m2.5:low__json_schema PASS 100% 164 in → 225 out (209 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.5:medium PASS 100% 152 in → 230 out (251 reasoning)
Title: My Project v3
minimax/minimax-m2.5:medium__json_schema PASS 100% 152 in → 202 out (213 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.5:xhigh PASS 100% 152 in → 139 out (131 reasoning)
Title: My Project v3
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 152 in → 275 out (261 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.7:high PASS 100% 156 in → 147 out (140 reasoning)
Title: My Project v3
minimax/minimax-m2.7:high__json_schema PASS 100% 286 in → 407 out (390 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.7:low PASS 100% 152 in → 179 out (206 reasoning)
Title: My Project v3
minimax/minimax-m2.7:low__json_schema PASS 100% 287 in → 430 out (417 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
minimax/minimax-m2.7:medium PASS 100% 153 in → 128 out (120 reasoning)
Title: My Project v3
minimax/minimax-m2.7:medium__json_schema PASS 100% 285 in → 441 out (423 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
minimax/minimax-m2.7:xhigh PASS 100% 152 in → 247 out (272 reasoning)
Title: My Project v3
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 286 in → 380 out (367 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
minimax/minimax-m3:high PASS 100% 291 in → 97 out (93 reasoning)
Title: My Project v3
minimax/minimax-m3:high__json_schema PASS 100% 276 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m3:low PASS 100% 292 in → 119 out (118 reasoning)
Title: My Project v3
minimax/minimax-m3:low__json_schema PASS 100% 277 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m3:medium PASS 100% 292 in → 69 out (64 reasoning)
Title: My Project v3
minimax/minimax-m3:medium__json_schema PASS 100% 238 in → 19 out (15 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m3:xhigh PASS 100% 291 in → 99 out (91 reasoning)
Title: My Project v3
minimax/minimax-m3:xhigh__json_schema PASS 100% 239 in → 16 out (14 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
mistralai/devstral-2512__json_schema PASS 100% 124 in → 14 out
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:high PASS 100% 136 in → 192 out (203 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 125 in → 158 out (149 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:low PASS 100% 135 in → 287 out (302 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 125 in → 166 out (165 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:medium PASS 100% 136 in → 194 out (204 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 124 in → 228 out (225 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 125 in → 14 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 134 in → 259 out (270 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 124 in → 198 out (214 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
moonshotai/kimi-k2.6:high PASS 100% 122 in → 288 out (280 reasoning)
Title: My Project v3
moonshotai/kimi-k2.6:low PASS 100% 123 in → 445 out (437 reasoning)
Title: My Project v3
moonshotai/kimi-k2.6:low__json_schema PASS 100% 120 in → 768 out (550 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 122 in → 293 out (306 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.6:xhigh PASS 100% 123 in → 285 out (279 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:high PASS 100% 121 in → 158 out (150 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 120 in → 83 out (81 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.7-code:low PASS 100% 121 in → 142 out (134 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 209 in → 78 out (64 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
moonshotai/kimi-k2.7-code:medium PASS 100% 121 in → 55 out (47 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 122 in → 152 out (135 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 121 in → 51 out (43 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 206 in → 89 out (74 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
moonshotai/kimi-k3:high PASS 100% 200 in → 133 out (109 reasoning)
``` Title: My Project v3 ```
moonshotai/kimi-k3:high__json_schema PASS 100% 200 in → 95 out (74 reasoning)
Title: My Project v3
moonshotai/kimi-k3:low PASS 100% 201 in → 98 out (77 reasoning)
Title: My Project v3
moonshotai/kimi-k3:low__json_schema PASS 100% 198 in → 105 out (84 reasoning)
Title: My Project v3
moonshotai/kimi-k3:max PASS 100% 199 in → 111 out (90 reasoning)
Title: My Project v3
moonshotai/kimi-k3:max__json_schema PASS 100% 198 in → 120 out (96 reasoning)
``` Title: My Project v3 ```
moonshotai/kimi-k3:medium__json_schema PASS 100% 199 in → 103 out (82 reasoning)
Title: My Project v3
moonshotai/kimi-k3:xhigh PASS 100% 201 in → 105 out (84 reasoning)
Title: My Project v3
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 199 in → 139 out (118 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 135 in → 56 out (53 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 137 in → 192 out (210 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 137 in → 108 out (120 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 137 in → 67 out (59 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 136 in → 116 out (135 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 137 in → 164 out (162 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 138 in → 7 out (0 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 137 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 134 in → 249 out (288 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 137 in → 138 out (151 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 137 in → 49 out (40 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 137 in → 48 out (30 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 136 in → 37 out (33 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 137 in → 60 out (42 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 136 in → 39 out (30 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 137 in → 59 out (40 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 137 in → 7 out (0 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 136 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 138 in → 40 out (31 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 136 in → 54 out (36 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-5.4-mini:high PASS 100% 116 in → 46 out (34 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:high__json_schema PASS 100% 153 in → 93 out (71 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.4-mini:low PASS 100% 118 in → 31 out (19 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:low__json_schema PASS 100% 153 in → 38 out (17 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-mini:medium PASS 100% 119 in → 40 out (28 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:medium__json_schema PASS 100% 155 in → 66 out (44 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.4-mini:none PASS 100% 118 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:none__json_schema PASS 100% 153 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-mini:xhigh PASS 100% 119 in → 75 out (63 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 155 in → 181 out (160 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-nano:high PASS 100% 118 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:high__json_schema PASS 100% 152 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-nano:low PASS 100% 118 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:low__json_schema PASS 100% 154 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-nano:medium PASS 100% 116 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:medium__json_schema PASS 100% 154 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.4-nano:none PASS 100% 118 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:none__json_schema PASS 100% 153 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-nano:xhigh PASS 100% 119 in → 54 out (42 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 153 in → 86 out (64 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.4:high PASS 100% 119 in → 52 out (40 reasoning)
Title: My Project v3
openai/gpt-5.4:high__json_schema PASS 100% 156 in → 45 out (24 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:low PASS 100% 120 in → 21 out (9 reasoning)
Title: My Project v3
openai/gpt-5.4:low__json_schema PASS 100% 155 in → 30 out (9 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:medium PASS 100% 118 in → 34 out (22 reasoning)
Title: My Project v3
openai/gpt-5.4:medium__json_schema PASS 100% 154 in → 68 out (47 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:none PASS 100% 121 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4:none__json_schema PASS 100% 154 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:xhigh PASS 100% 117 in → 48 out (36 reasoning)
Title: My Project v3
openai/gpt-5.4:xhigh__json_schema PASS 100% 153 in → 151 out (130 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:high PASS 100% 117 in → 49 out (37 reasoning)
Title: My Project v3
openai/gpt-5.5:high__json_schema PASS 100% 155 in → 537 out (516 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:low PASS 100% 119 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.5:low__json_schema PASS 100% 154 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:medium PASS 100% 117 in → 27 out (15 reasoning)
Title: My Project v3
openai/gpt-5.5:medium__json_schema PASS 100% 153 in → 293 out (272 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:none PASS 100% 119 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.5:none__json_schema PASS 100% 154 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:xhigh PASS 100% 119 in → 49 out (37 reasoning)
Title: My Project v3
openai/gpt-5.5:xhigh__json_schema PASS 100% 156 in → 377 out (356 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:high PASS 100% 119 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:high__json_schema PASS 100% 152 in → 38 out (17 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:low PASS 100% 119 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:low__json_schema PASS 100% 153 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:max PASS 100% 117 in → 51 out (39 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:max__json_schema PASS 100% 154 in → 162 out (141 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:medium PASS 100% 116 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:medium__json_schema PASS 100% 154 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:none PASS 100% 118 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:none__json_schema PASS 100% 153 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:xhigh PASS 100% 117 in → 41 out (29 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 153 in → 43 out (22 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:high PASS 100% 119 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:high__json_schema PASS 100% 155 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:low PASS 100% 118 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:low__json_schema PASS 100% 155 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.6-sol:max PASS 100% 117 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:max__json_schema PASS 100% 155 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.6-sol:medium PASS 100% 117 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:medium__json_schema PASS 100% 154 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.6-sol:none PASS 100% 120 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:none__json_schema PASS 100% 154 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.6-sol:xhigh PASS 100% 117 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 154 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.6-terra:high PASS 100% 118 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:high__json_schema PASS 100% 156 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:low PASS 100% 119 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:low__json_schema PASS 100% 155 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:max PASS 100% 118 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:max__json_schema PASS 100% 155 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:medium PASS 100% 118 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:medium__json_schema PASS 100% 154 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:none PASS 100% 119 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:none__json_schema PASS 100% 153 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:xhigh PASS 100% 118 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 154 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-oss-120b:high PASS 100% 179 in → 311 out (356 reasoning)
Title: My Project v3
openai/gpt-oss-120b:high__json_schema PASS 100% 178 in → 184 out (159 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
openai/gpt-oss-120b:low PASS 100% 177 in → 46 out (34 reasoning)
Title: My Project v3
openai/gpt-oss-120b:low__json_schema PASS 100% 182 in → 122 out (110 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
openai/gpt-oss-120b:medium PASS 100% 176 in → 139 out (143 reasoning)
Title: My Project v3
openai/gpt-oss-120b:medium__json_schema PASS 100% 180 in → 84 out (73 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
openai/gpt-oss-120b:xhigh PASS 100% 179 in → 136 out (130 reasoning)
Title: My Project v3
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 165 in → 249 out (273 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-oss-20b:high PASS 100% 178 in → 184 out (193 reasoning)
Title: My Project v3
openai/gpt-oss-20b:high__json_schema PASS 100% 179 in → 152 out (126 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-oss-20b:low PASS 100% 180 in → 30 out (15 reasoning)
Title: My Project v3
openai/gpt-oss-20b:low__json_schema PASS 100% 177 in → 50 out (26 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-oss-20b:medium PASS 100% 165 in → 54 out (43 reasoning)
Title: My Project v3
openai/gpt-oss-20b:xhigh PASS 100% 181 in → 232 out (217 reasoning)
Title: My Project v3
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 179 in → 181 out (159 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content":"Title: My Project v3" }
poolside/laguna-m.1:high PASS 100% 131 in → 248 out (239 reasoning)
Title: My Project v3
poolside/laguna-m.1:high__json_schema PASS 100% 129 in → 273 out (260 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-m.1:low PASS 100% 131 in → 256 out (247 reasoning)
Title: My Project v3
poolside/laguna-m.1:low__json_schema PASS 100% 130 in → 267 out (249 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-m.1:medium PASS 100% 130 in → 220 out (211 reasoning)
Title: My Project v3
poolside/laguna-m.1:medium__json_schema PASS 100% 129 in → 265 out (252 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-m.1:none PASS 100% 130 in → 8 out (0 reasoning)
Title: My Project v3
poolside/laguna-m.1:none__json_schema PASS 100% 130 in → 13 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-m.1:xhigh PASS 100% 129 in → 245 out (237 reasoning)
Title: My Project v3
poolside/laguna-m.1:xhigh__json_schema PASS 100% 131 in → 301 out (283 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:high PASS 100% 132 in → 219 out (212 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:high__json_schema PASS 100% 130 in → 201 out (182 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:low PASS 100% 131 in → 256 out (249 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:low__json_schema PASS 100% 131 in → 204 out (185 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:medium PASS 100% 130 in → 250 out (243 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 130 in → 219 out (200 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:none PASS 100% 129 in → 6 out (0 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:none__json_schema PASS 100% 130 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:xhigh PASS 100% 130 in → 226 out (219 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 132 in → 162 out (143 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs.2:high PASS 100% 169 in → 130 out (121 reasoning)
Title: My Project v3
poolside/laguna-xs.2:high__json_schema PASS 100% 168 in → 163 out (143 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
poolside/laguna-xs.2:low PASS 100% 167 in → 196 out (187 reasoning)
Title: My Project v3
poolside/laguna-xs.2:low__json_schema PASS 100% 167 in → 153 out (139 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-xs.2:medium PASS 100% 168 in → 175 out (166 reasoning)
Title: My Project v3
poolside/laguna-xs.2:medium__json_schema PASS 100% 168 in → 147 out (127 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
poolside/laguna-xs.2:none PASS 100% 169 in → 8 out (0 reasoning)
Title: My Project v3
poolside/laguna-xs.2:none__json_schema PASS 100% 166 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3"}
poolside/laguna-xs.2:xhigh PASS 100% 167 in → 158 out (149 reasoning)
Title: My Project v3
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 169 in → 136 out (122 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
qwen/qwen3.6-27b:high PASS 100% 128 in → 822 out (832 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:high__json_schema PASS 100% 125 in → 340 out (326 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-27b:low PASS 100% 126 in → 1,028 out (1,017 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:low__json_schema PASS 100% 126 in → 1,031 out (1,024 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-27b:medium PASS 100% 127 in → 1,068 out (1 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:none PASS 100% 129 in → 7 out (0 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:none__json_schema PASS 100% 129 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-27b:xhigh PASS 100% 126 in → 524 out (509 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 127 in → 945 out (1 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-35b-a3b:high PASS 100% 126 in → 869 out (891 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 128 in → 966 out (961 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-35b-a3b:low PASS 100% 126 in → 275 out (261 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 126 in → 840 out (813 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 127 in → 886 out (884 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 126 in → 869 out (842 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-35b-a3b:none PASS 100% 128 in → 7 out (0 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 130 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 128 in → 756 out (745 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 128 in → 1,137 out (1,121 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-flash:high PASS 100% 127 in → 573 out (562 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:low PASS 100% 125 in → 233 out (222 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:medium PASS 100% 126 in → 697 out (686 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:none PASS 100% 127 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:xhigh PASS 100% 125 in → 876 out (865 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:high PASS 100% 127 in → 684 out (673 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:low PASS 100% 128 in → 1,027 out (1,016 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:max PASS 100% 126 in → 304 out (293 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:medium PASS 100% 127 in → 391 out (380 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:none PASS 100% 127 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:xhigh PASS 100% 127 in → 761 out (750 reasoning)
Title: My Project v3
qwen/qwen3.7-max:high PASS 100% 127 in → 119 out (108 reasoning)
Title: My Project v3
qwen/qwen3.7-max:low PASS 100% 126 in → 214 out (203 reasoning)
Title: My Project v3
qwen/qwen3.7-max:medium PASS 100% 127 in → 312 out (301 reasoning)
Title: My Project v3
qwen/qwen3.7-max:none PASS 100% 130 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.7-max:xhigh PASS 100% 125 in → 245 out (234 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:high PASS 100% 127 in → 290 out (279 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:low PASS 100% 128 in → 489 out (478 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:medium PASS 100% 128 in → 10,442 out (10,431 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:none PASS 100% 128 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:xhigh PASS 100% 127 in → 484 out (473 reasoning)
Title: My Project v3
tencent/hy3:high PASS 100% 128 in → 580 out (572 reasoning)
Title: My Project v3
tencent/hy3:high__json_schema PASS 100% 128 in → 434 out (443 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
tencent/hy3:low PASS 100% 129 in → 286 out (278 reasoning)
Title: My Project v3
tencent/hy3:low__json_schema PASS 100% 129 in → 227 out (215 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
tencent/hy3:medium PASS 100% 129 in → 204 out (196 reasoning)
Title: My Project v3
tencent/hy3:medium__json_schema PASS 100% 128 in → 340 out (349 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
tencent/hy3:none PASS 100% 132 in → 7 out (0 reasoning)
Title: My Project v3
tencent/hy3:none__json_schema PASS 100% 132 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
tencent/hy3:xhigh PASS 100% 128 in → 489 out (481 reasoning)
Title: My Project v3
tencent/hy3:xhigh__json_schema PASS 100% 127 in → 268 out (259 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:high PASS 100% 125 in → 150 out (163 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:high__json_schema PASS 100% 127 in → 207 out (187 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:low PASS 100% 126 in → 34 out (23 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:low__json_schema PASS 100% 128 in → 44 out (22 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
thinkingmachines/inkling-small:max PASS 100% 127 in → 185 out (173 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:max__json_schema PASS 100% 127 in → 262 out (243 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:medium PASS 100% 125 in → 72 out (69 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:medium__json_schema PASS 100% 127 in → 100 out (80 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:xhigh PASS 100% 126 in → 175 out (191 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 128 in → 177 out (157 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling:high PASS 100% 127 in → 128 out (115 reasoning)
Title: My Project v3
thinkingmachines/inkling:low PASS 100% 127 in → 23 out (11 reasoning)
Title: My Project v3
thinkingmachines/inkling:max PASS 100% 126 in → 144 out (132 reasoning)
Title: My Project v3
thinkingmachines/inkling:max__json_schema PASS 100% 128 in → 214 out (194 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling:medium PASS 100% 126 in → 58 out (45 reasoning)
Title: My Project v3
thinkingmachines/inkling:none__json_schema PASS 100% 126 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n"}
thinkingmachines/inkling:xhigh PASS 100% 125 in → 113 out (101 reasoning)
Title: My Project v3
x-ai/grok-4.3:high PASS 100% 296 in → 134 out (128 reasoning)
Title: My Project v3
x-ai/grok-4.3:high__json_schema PASS 100% 353 in → 379 out (364 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.3:low PASS 100% 303 in → 210 out (204 reasoning)
Title: My Project v3
x-ai/grok-4.3:low__json_schema PASS 100% 358 in → 261 out (249 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
x-ai/grok-4.3:max PASS 100% 298 in → 185 out (179 reasoning)
Title: My Project v3
x-ai/grok-4.3:max__json_schema PASS 100% 353 in → 378 out (367 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.3:medium PASS 100% 301 in → 369 out (363 reasoning)
Title: My Project v3
x-ai/grok-4.3:medium__json_schema PASS 100% 360 in → 435 out (420 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.3:none PASS 100% 293 in → 6 out (0 reasoning)
Title: My Project v3
x-ai/grok-4.3:none__json_schema PASS 100% 351 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.3:xhigh PASS 100% 295 in → 276 out (270 reasoning)
Title: My Project v3
x-ai/grok-4.3:xhigh__json_schema PASS 100% 354 in → 299 out (284 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.5:high PASS 100% 319 in → 211 out (205 reasoning)
Title: My Project v3
x-ai/grok-4.5:high__json_schema PASS 100% 390 in → 386 out (373 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.5:low PASS 100% 321 in → 115 out (109 reasoning)
Title: My Project v3
x-ai/grok-4.5:low__json_schema PASS 100% 389 in → 191 out (173 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
x-ai/grok-4.5:max PASS 100% 320 in → 168 out (162 reasoning)
Title: My Project v3
x-ai/grok-4.5:max__json_schema PASS 100% 389 in → 343 out (325 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
x-ai/grok-4.5:medium PASS 100% 320 in → 187 out (181 reasoning)
Title: My Project v3
x-ai/grok-4.5:medium__json_schema PASS 100% 391 in → 185 out (167 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
x-ai/grok-4.5:xhigh PASS 100% 320 in → 172 out (166 reasoning)
Title: My Project v3
x-ai/grok-4.5:xhigh__json_schema PASS 100% 389 in → 284 out (266 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
z-ai/glm-4.7-flash:high PASS 100% 116 in → 349 out (349 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:high__json_schema PASS 100% 117 in → 323 out (288 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-4.7-flash:low PASS 100% 118 in → 329 out (296 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:medium PASS 100% 116 in → 452 out (474 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:none PASS 100% 117 in → 7 out (0 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:none__json_schema PASS 100% 118 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-4.7-flash:xhigh PASS 100% 118 in → 358 out (337 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 118 in → 354 out (320 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-5.2:high PASS 100% 125 in → 133 out (125 reasoning)
Title: My Project v3
z-ai/glm-5.2:low PASS 100% 124 in → 266 out (255 reasoning)
Title: My Project v3
z-ai/glm-5.2:low__json_schema PASS 100% 118 in → 274 out (230 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-5.2:medium PASS 100% 125 in → 243 out (235 reasoning)
Title: My Project v3
z-ai/glm-5.2:medium__json_schema PASS 100% 126 in → 213 out (196 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-5.2:none PASS 100% 120 in → 7 out (0 reasoning)
Title: My Project v3
z-ai/glm-5.2:none__json_schema PASS 100% 296 in → 12 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
z-ai/glm-5.2:xhigh PASS 100% 123 in → 242 out (274 reasoning)
Title: My Project v3
z-ai/glm-5.2:xhigh__json_schema PASS 100% 127 in → 160 out (145 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-fable-5:high FAIL 0% 194 in → 109 out (6 reasoning)
Title: My Project v3 **Explanation:** Since the higher version is more appropriate, keep the HEAD (main branch) version `v3` and discard the incoming `v2` change, removing all conflict markers. The changes to file_a and file_b don't conflict, so they merge automatically and require no action.
Failure: Content mismatch: ['project.txt']
anthropic/claude-fable-5:medium FAIL 0% 193 in → 128 out (8 reasoning)
``` Title: My Project v3 ``` **Reasoning:** The conflict is between v3 (HEAD/main) and v2 (incoming cherry-pick). Since the higher version is more appropriate, we keep `Title: My Project v3` and remove all conflict markers. The changes to file_a and file_b merge cleanly and don't affect this file.
Failure: Content mismatch: ['project.txt']
anthropic/claude-fable-5:xhigh FAIL 0% 192 in → 131 out (19 reasoning)
``` Title: My Project v3 ``` The conflict is resolved by keeping the HEAD (main branch) version, since v3 is the higher version. The conflict markers (`<<<<<<< HEAD`, `=======`, `>>>>>>> 3931f44`) and the incoming v2 line are removed entirely.
Failure: Content mismatch: ['project.txt']
anthropic/claude-opus-4.6:max FAIL 0% 134 in → 173 out (80 reasoning)
``` Title: My Project v3 ``` **Reasoning:** The conflict is between `v3` (current `main` branch) and `v2` (incoming cherry-picked commit). Since the higher version (`v3`) is more appropriate, the HEAD content is kept, and the conflict markers and the incoming change are removed.
Failure: Content mismatch: ['project.txt']
anthropic/claude-sonnet-4.6:high FAIL 0% 137 in → 197 out (105 reasoning)
``` Title: My Project v3 ``` **Reasoning:** The current `HEAD` (main branch) contains `v3`, which is the higher version number. Since the instructions specify that the higher version is more appropriate, the conflict is resolved by keeping `Title: My Project v3` and discarding the incoming `v2` from the cherry-picked commit.
Failure: Content mismatch: ['project.txt']
anthropic/claude-sonnet-4.6:max FAIL 0% 137 in → 210 out (106 reasoning)
``` Title: My Project v3 ``` **Reasoning:** In a cherry-pick, `HEAD` is the current branch (`main`) with `v3`, and the incoming commit brings `v2`. Since `v3` is the higher version and the instruction states the higher version is more appropriate, the `HEAD` version is kept and the conflict markers are removed entirely.
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:xhigh FAIL 0% 123 in → 173 out (182 reasoning)
Title: MyProject v3
Failure: Content mismatch: ['project.txt']
liquid/lfm-2-24b-a2b FAIL 0% 133 in → 8 out
project.txt My Project v2
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output: project.txt Title: My Project v2
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
mistralai/devstral-2512 FAIL 0% 122 in → 34 out
The resolved `project.txt` content should keep the higher version number (v3) from the current branch (main): ``` Title: My Project v3 ```
Failure: Content mismatch: ['project.txt']
mistralai/mistral-medium-3-5:none FAIL 0% 136 in → 19 out (0 reasoning)
The resolved `project.txt` content is: ``` Title: My Project v3 ```
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
moonshotai/kimi-k2.6:medium FAIL 0% 121 in → 433 out (498 reasoning)
Title: My Project v3
Failure: Content mismatch: ['project.txt']
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0% 120 in → 249 out (241 reasoning)
.cache
JSON Schema Structured Output
(raw) {"resolved_content":".cache"}
Failure: Content mismatch: ['project.txt']
(empty output)
Failure: OpenAI-compatible response contained provider error: Provider returned error
openai/gpt-oss-20b:medium__json_schema FAIL 0% 179 in → 133 out (0 reasoning)
}?We must output only the resolved project.txt content. The conflict we have is Title: My Project v3 vs v2. Higher version is more appropriate, so use v3. So content is
JSON Schema Structured Output
(raw) { "resolved_content": "}?We must output only the resolved project.txt content. The conflict we have is Title: My Project v3 vs v2. Higher version is more appropriate, so use v3. So content is " }
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c18686a0-d0ce-9296-a585-1c3513452a2e","request_id":"c18686a0-d0ce-9296-a585-1c3513452a2e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-90ab3d64-e7cc-9a6b-8d94-9454f0adb6e2","request_id":"90ab3d64-e7cc-9a6b-8d94-9454f0adb6e2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-249d9a05-d93d-9c2e-9a89-4aeeb0137d4e","request_id":"249d9a05-d93d-9c2e-9a89-4aeeb0137d4e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3e2f1c3f-d5aa-9791-a70f-364103b5c6e1","request_id":"3e2f1c3f-d5aa-9791-a70f-364103b5c6e1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2100f27b-8594-95bd-9fcd-c2795028f7e1","request_id":"2100f27b-8594-95bd-9fcd-c2795028f7e1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-86263536-6b3b-9b3f-b450-7a2afd24e85f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e98734c3-ec5f-99ab-94df-dbf29b163d0c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6fde1f81-9dc1-9190-b24b-ed1183f42489"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b3b9e96a-142b-9938-972d-9fc47a6562ef"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d38d2430-92b2-9d16-a188-494d397365f7"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-49c0ec5f-ef25-9883-8ec1-a4990fae14fb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b03837ca-bf5f-9d33-864e-5799603c95a5","request_id":"b03837ca-bf5f-9d33-864e-5799603c95a5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-abe71166-1884-912d-83bc-75350e80eaa5","request_id":"abe71166-1884-912d-83bc-75350e80eaa5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fb1b8373-d4d6-9f1c-bbc2-f0006cfdebeb","request_id":"fb1b8373-d4d6-9f1c-bbc2-f0006cfdebeb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-48f2a966-8d71-9b95-abcd-7d94ef3ef8de","request_id":"48f2a966-8d71-9b95-abcd-7d94ef3ef8de"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7b11e960-d6d7-9bad-8d92-e178795639f2","request_id":"7b11e960-d6d7-9bad-8d92-e178795639f2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-be4baa0d-de35-9aeb-94a1-8c3b892f5252","request_id":"be4baa0d-de35-9aeb-94a1-8c3b892f5252"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4a9f7ed5-4048-907b-91a9-47edbc28f260","request_id":"4a9f7ed5-4048-907b-91a9-47edbc28f260"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e6bbed81-44c8-9344-917c-9c8c8d54786f","request_id":"e6bbed81-44c8-9344-917c-9c8c8d54786f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7528532c-d180-94d5-a67a-d9b21a0ccca6","request_id":"7528532c-d180-94d5-a67a-d9b21a0ccca6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1dbb43c9-99e6-9037-9b2f-6eb1eb42db44","request_id":"1dbb43c9-99e6-9037-9b2f-6eb1eb42db44"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 128 in → 9 out (0 reasoning)
Title: My Project v3<|end_message|>
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output: Title: My Project v3{ "resolved_content": "Title: My Project v3" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 128 in → 9 out (0 reasoning)
Title: My Project v3<|end_message|>
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)