Three-way: multiple files with conflicting and non-conflicting changes
Tests ability to handle merge where some files conflict and others merge cleanly. Evaluates partial-conflict understanding.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Title: My Project' > project.txt
  5. 05 echo 'File A content' > file_a.txt
  6. 06 echo 'File B content' > file_b.txt
  7. 07 git add project.txt file_a.txt file_b.txt
  8. 08 git commit -m 'Initial project'
  9. 09 git checkout -b branch-a
  10. 10 echo 'Title: My Project v2' > project.txt
  11. 11 echo 'File A updated' > file_a.txt
  12. 12 git add project.txt file_a.txt
  13. 13 git commit -m 'Update title and file A'
  14. 14 git checkout main
  15. 15 echo 'Title: My Project v3' > project.txt
  16. 16 echo 'File B updated' > file_b.txt
  17. 17 git add project.txt file_b.txt
  18. 18 git commit -m 'Update title and file B'
  19. 19 git merge branch-a
Prompt
Resolve the merge conflict in project.txt. Branch A set Title: My Project v2. Branch B set Title: My Project v3. Both branches also updated different files (file_a and file_b) without conflicts. For the project.txt conflict, the higher version is more appropriate. Provide ONLY the resolved project.txt content.
Expected
Title: My Project v3

Scoped model quality, cost, API time, and token usage for merge_conflicts/f008.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 173 in → 20 out (6 reasoning)
Title: My Project v3
anthropic/claude-fable-5:high__json_schema PASS 100% 393 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-fable-5:low PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-fable-5:low__json_schema PASS 100% 393 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-fable-5:max PASS 100% 173 in → 232 out (57 reasoning)
``` Title: My Project v3 ```
anthropic/claude-fable-5:max__json_schema PASS 100% 393 in → 197 out (75 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-fable-5:medium PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-fable-5:medium__json_schema PASS 100% 393 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-fable-5:xhigh PASS 100% 173 in → 36 out (19 reasoning)
Title: My Project v3
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 393 in → 37 out (7 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-haiku-4.5:high PASS 100% 149 in → 175 out (174 reasoning)
Title: My Project v3
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 317 in → 182 out (154 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-haiku-4.5:low PASS 100% 149 in → 158 out (143 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 317 in → 217 out (197 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-haiku-4.5:medium PASS 100% 149 in → 135 out (120 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 317 in → 141 out (120 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-haiku-4.5:none PASS 100% 120 in → 13 out (0 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 287 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 149 in → 269 out (251 reasoning)
Title: My Project v3
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 317 in → 337 out (323 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.6:high PASS 100% 120 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.6:high__json_schema PASS 100% 288 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.6:low PASS 100% 120 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.6:low__json_schema PASS 100% 288 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 288 in → 137 out (105 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.6:medium PASS 100% 120 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 288 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.6:none PASS 100% 120 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.6:none__json_schema PASS 100% 288 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.6:xhigh PASS 100% 120 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 288 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.7:high PASS 100% 178 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:high__json_schema PASS 100% 398 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:low PASS 100% 178 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:low__json_schema PASS 100% 398 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:max PASS 100% 178 in → 124 out (48 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:max__json_schema PASS 100% 398 in → 188 out (56 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:medium PASS 100% 178 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 398 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:none PASS 100% 178 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:none__json_schema PASS 100% 398 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 178 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 398 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.8:high PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:high__json_schema PASS 100% 393 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.8:low PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:low__json_schema PASS 100% 393 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.8:max PASS 100% 173 in → 178 out (58 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:max__json_schema PASS 100% 393 in → 191 out (52 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-opus-4.8:medium PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 393 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.8:none PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:none__json_schema PASS 100% 393 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-opus-4.8:xhigh PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 393 in → 37 out (11 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 288 in → 95 out (58 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-sonnet-4.6:low PASS 100% 120 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 288 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 288 in → 152 out (117 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 120 in → 46 out (19 reasoning)
Title: My Project v3
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 288 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 288 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 288 in → 82 out (52 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-sonnet-5:high PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:high__json_schema PASS 100% 393 in → 63 out (26 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-sonnet-5:low PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:low__json_schema PASS 100% 393 in → 23 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-sonnet-5:medium PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 393 in → 23 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-5:none PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:none__json_schema PASS 100% 393 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-sonnet-5:xhigh PASS 100% 173 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 393 in → 66 out (22 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
arcee-ai/trinity-large-thinking:high PASS 100% 107 in → 339 out (331 reasoning)
Title: My Project v3
arcee-ai/trinity-large-thinking:low PASS 100% 107 in → 153 out (144 reasoning)
Title: My Project v3
arcee-ai/trinity-large-thinking:medium PASS 100% 107 in → 434 out (425 reasoning)
Title: My Project v3
arcee-ai/trinity-large-thinking:xhigh PASS 100% 107 in → 130 out (121 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:high__json_schema PASS 100% 107 in → 393 out (436 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
arcee-ai/trinity-mini:low__json_schema PASS 100% 107 in → 370 out (398 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 107 in → 380 out (417 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
arcee-ai/trinity-mini:xhigh PASS 100% 107 in → 290 out (311 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 107 in → 361 out (380 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content":"Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 106 in → 142 out (151 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 185 in → 100 out (101 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 106 in → 131 out (147 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 106 in → 212 out (210 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 198 in → 169 out (162 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 198 in → 266 out (270 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 106 in → 59 out (52 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 106 in → 228 out (211 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 106 in → 7 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 106 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 106 in → 225 out (261 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 106 in → 155 out (146 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash:high PASS 100% 106 in → 82 out (76 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 108 in → 104 out (86 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash:low PASS 100% 106 in → 101 out (93 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 108 in → 231 out (213 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash:medium PASS 100% 106 in → 109 out (101 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 106 in → 71 out (53 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash:none PASS 100% 106 in → 6 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 108 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 185 in → 343 out (335 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 185 in → 237 out (238 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3"}
deepseek/deepseek-v4-pro:high PASS 100% 106 in → 121 out (113 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 108 in → 199 out (181 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-pro:low PASS 100% 106 in → 218 out (210 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 298 in → 299 out (284 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
deepseek/deepseek-v4-pro:medium PASS 100% 106 in → 7 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 108 in → 261 out (242 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
deepseek/deepseek-v4-pro:none PASS 100% 106 in → 7 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 298 in → 14 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 185 in → 605 out (597 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 187 in → 439 out (421 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3-flash-preview:high PASS 100% 107 in → 186 out (180 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:high__json_schema PASS 100% 167 in → 364 out (352 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3-flash-preview:low PASS 100% 107 in → 209 out (203 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:low__json_schema PASS 100% 167 in → 267 out (255 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3-flash-preview:medium PASS 100% 107 in → 162 out (156 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:medium__json_schema PASS 100% 167 in → 280 out (263 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3-flash-preview:xhigh PASS 100% 107 in → 212 out (206 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 167 in → 308 out (296 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 107 in → 652 out (646 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 167 in → 906 out (888 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.1-flash-lite-preview:low PASS 100% 108 in → 133 out (127 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 167 in → 131 out (113 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 107 in → 182 out (176 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 167 in → 131 out (113 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 108 in → 357 out (351 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 108 in → 744 out (726 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.1-pro-preview:high PASS 100% 107 in → 297 out (291 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 167 in → 825 out (812 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-pro-preview:low PASS 100% 107 in → 263 out (257 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 167 in → 99 out (81 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.1-pro-preview:medium PASS 100% 107 in → 255 out (249 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 167 in → 119 out (106 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 107 in → 291 out (285 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 167 in → 167 out (155 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.5-flash-lite:high PASS 100% 108 in → 400 out (394 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 167 in → 638 out (625 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.5-flash-lite:low PASS 100% 108 in → 6 out (0 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 167 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.5-flash-lite:max PASS 100% 108 in → 368 out (362 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 167 in → 676 out (664 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.5-flash-lite:medium PASS 100% 108 in → 220 out (214 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 167 in → 421 out (408 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 108 in → 363 out (357 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 167 in → 589 out (576 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.5-flash:high PASS 100% 107 in → 331 out (325 reasoning)
Title: My Project v3
google/gemini-3.5-flash:high__json_schema PASS 100% 167 in → 557 out (544 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.5-flash:low PASS 100% 107 in → 244 out (238 reasoning)
Title: My Project v3
google/gemini-3.5-flash:low__json_schema PASS 100% 167 in → 209 out (197 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.5-flash:medium PASS 100% 107 in → 333 out (327 reasoning)
Title: My Project v3
google/gemini-3.5-flash:medium__json_schema PASS 100% 167 in → 731 out (716 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
google/gemini-3.5-flash:xhigh PASS 100% 107 in → 381 out (375 reasoning)
Title: My Project v3
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 167 in → 747 out (729 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.6-flash:high PASS 100% 107 in → 226 out (220 reasoning)
Title: My Project v3
google/gemini-3.6-flash:high__json_schema PASS 100% 167 in → 425 out (413 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.6-flash:low PASS 100% 108 in → 79 out (73 reasoning)
Title: My Project v3
google/gemini-3.6-flash:low__json_schema PASS 100% 108 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.6-flash:max PASS 100% 108 in → 338 out (332 reasoning)
Title: My Project v3
google/gemini-3.6-flash:max__json_schema PASS 100% 167 in → 417 out (405 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.6-flash:medium PASS 100% 108 in → 255 out (249 reasoning)
Title: My Project v3
google/gemini-3.6-flash:medium__json_schema PASS 100% 167 in → 420 out (408 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.6-flash:xhigh PASS 100% 108 in → 253 out (247 reasoning)
Title: My Project v3
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 167 in → 490 out (473 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:high PASS 100% 123 in → 203 out (185 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 123 in → 192 out (147 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemma-4-26b-a4b-it:low PASS 100% 123 in → 137 out (130 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 123 in → 189 out (145 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:medium PASS 100% 123 in → 145 out (138 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 123 in → 177 out (134 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:none PASS 100% 120 in → 7 out (0 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 120 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 123 in → 164 out (129 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 122 in → 181 out (136 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemma-4-31b-it:high PASS 100% 123 in → 142 out (113 reasoning)
Title: My Project v3
google/gemma-4-31b-it:high__json_schema PASS 100% 122 in → 132 out (91 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-31b-it:low PASS 100% 123 in → 149 out (116 reasoning)
Title: My Project v3
google/gemma-4-31b-it:low__json_schema PASS 100% 123 in → 150 out (109 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-31b-it:medium PASS 100% 123 in → 116 out (90 reasoning)
Title: My Project v3
google/gemma-4-31b-it:medium__json_schema PASS 100% 123 in → 142 out (1 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-31b-it:none PASS 100% 120 in → 7 out (0 reasoning)
Title: My Project v3
google/gemma-4-31b-it:none__json_schema PASS 100% 120 in → 14 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemma-4-31b-it:xhigh PASS 100% 123 in → 129 out (1 reasoning)
Title: My Project v3
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 122 in → 166 out (131 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
ibm-granite/granite-4.1-8b PASS 100% 115 in → 7 out
Title: My Project v3
ibm-granite/granite-4.1-8b__json_schema PASS 100% 115 in → 17 out
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
inclusionai/ling-2.6-flash PASS 100% 122 in → 7 out
Title: My Project v3
inclusionai/ling-2.6-flash__json_schema PASS 100% 122 in → 19 out
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
liquid/lfm-2-24b-a2b PASS 100% 116 in → 7 out
Title: My Project v3
minimax/minimax-m2.5:high PASS 100% 139 in → 228 out (221 reasoning)
Title: My Project v3
minimax/minimax-m2.5:high__json_schema PASS 100% 137 in → 112 out (104 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.5:low PASS 100% 137 in → 499 out (573 reasoning)
Title: My Project v3
minimax/minimax-m2.5:low__json_schema PASS 100% 137 in → 452 out (465 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.5:medium PASS 100% 137 in → 555 out (547 reasoning)
Title: My Project v3
minimax/minimax-m2.5:medium__json_schema PASS 100% 137 in → 373 out (393 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.5:xhigh PASS 100% 137 in → 424 out (417 reasoning)
Title: My Project v3
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 137 in → 270 out (296 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.7:high PASS 100% 139 in → 175 out (168 reasoning)
Title: My Project v3
minimax/minimax-m2.7:high__json_schema PASS 100% 270 in → 399 out (386 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
minimax/minimax-m2.7:low PASS 100% 136 in → 227 out (256 reasoning)
Title: My Project v3
minimax/minimax-m2.7:low__json_schema PASS 100% 270 in → 400 out (387 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
minimax/minimax-m2.7:medium PASS 100% 137 in → 233 out (257 reasoning)
Title: My Project v3
minimax/minimax-m2.7:medium__json_schema PASS 100% 270 in → 343 out (326 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.7:xhigh PASS 100% 137 in → 271 out (309 reasoning)
Title: My Project v3
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 270 in → 372 out (359 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
minimax/minimax-m3:high PASS 100% 275 in → 105 out (114 reasoning)
Title: My Project v3
minimax/minimax-m3:high__json_schema PASS 100% 262 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m3:low PASS 100% 275 in → 32 out (26 reasoning)
Title: My Project v3
minimax/minimax-m3:low__json_schema PASS 100% 262 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
minimax/minimax-m3:medium PASS 100% 275 in → 79 out (82 reasoning)
Title: My Project v3
minimax/minimax-m3:medium__json_schema PASS 100% 262 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m3:xhigh PASS 100% 275 in → 105 out (108 reasoning)
Title: My Project v3
minimax/minimax-m3:xhigh__json_schema PASS 100% 223 in → 16 out (12 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/devstral-2512__json_schema PASS 100% 106 in → 14 out
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:high PASS 100% 118 in → 306 out (310 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 106 in → 206 out (217 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:low PASS 100% 118 in → 229 out (232 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 106 in → 243 out (256 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:medium PASS 100% 118 in → 296 out (292 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 106 in → 259 out (262 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:none PASS 100% 118 in → 7 out (0 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 106 in → 14 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 118 in → 329 out (340 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 106 in → 244 out (247 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
moonshotai/kimi-k2.6:high PASS 100% 106 in → 451 out (443 reasoning)
Title: My Project v3
moonshotai/kimi-k2.6:high__json_schema PASS 100% 106 in → 256 out (260 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.6:low PASS 100% 106 in → 336 out (328 reasoning)
Title: My Project v3
moonshotai/kimi-k2.6:low__json_schema PASS 100% 105 in → 378 out (344 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 105 in → 595 out (334 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 289 in → 509 out (496 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 181 in → 178 out (160 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 106 in → 145 out (127 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.7-code:medium PASS 100% 105 in → 104 out (96 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 106 in → 116 out (111 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 106 in → 148 out (140 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 106 in → 108 out (92 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k3:high PASS 100% 183 in → 133 out (109 reasoning)
``` Title: My Project v3 ```
moonshotai/kimi-k3:high__json_schema PASS 100% 183 in → 154 out (133 reasoning)
Title: My Project v3
moonshotai/kimi-k3:low PASS 100% 183 in → 116 out (95 reasoning)
Title: My Project v3
moonshotai/kimi-k3:low__json_schema PASS 100% 183 in → 123 out (102 reasoning)
Title: My Project v3
moonshotai/kimi-k3:max PASS 100% 183 in → 141 out (120 reasoning)
Title: My Project v3
moonshotai/kimi-k3:max__json_schema PASS 100% 183 in → 103 out (82 reasoning)
Title: My Project v3
moonshotai/kimi-k3:medium PASS 100% 183 in → 182 out (161 reasoning)
Title: My Project v3
moonshotai/kimi-k3:medium__json_schema PASS 100% 183 in → 247 out (226 reasoning)
Title: My Project v3
moonshotai/kimi-k3:xhigh PASS 100% 183 in → 215 out (194 reasoning)
Title: My Project v3
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 183 in → 149 out (128 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 119 in → 116 out (112 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 119 in → 182 out (185 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 119 in → 181 out (201 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 119 in → 68 out (57 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 119 in → 164 out (180 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 119 in → 185 out (180 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 119 in → 7 out (0 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 119 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 119 in → 141 out (145 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 119 in → 208 out (216 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 119 in → 142 out (127 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 119 in → 92 out (74 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 119 in → 55 out (46 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 119 in → 60 out (41 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 119 in → 58 out (49 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 119 in → 109 out (91 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 119 in → 7 out (0 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 119 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 119 in → 104 out (101 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 119 in → 52 out (34 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-5.4-mini:high PASS 100% 102 in → 87 out (75 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:high__json_schema PASS 100% 138 in → 192 out (171 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-mini:low PASS 100% 102 in → 28 out (16 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:low__json_schema PASS 100% 138 in → 40 out (19 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-mini:medium PASS 100% 102 in → 67 out (55 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:medium__json_schema PASS 100% 138 in → 85 out (64 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-mini:none PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:none__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-mini:xhigh PASS 100% 102 in → 115 out (103 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 138 in → 153 out (132 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-nano:high PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:high__json_schema PASS 100% 138 in → 112 out (91 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-nano:low PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:low__json_schema PASS 100% 138 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.4-nano:medium PASS 100% 102 in → 36 out (24 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:medium__json_schema PASS 100% 138 in → 72 out (50 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.4-nano:none PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:none__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-nano:xhigh PASS 100% 102 in → 174 out (162 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 138 in → 168 out (147 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:high PASS 100% 102 in → 53 out (41 reasoning)
Title: My Project v3
openai/gpt-5.4:high__json_schema PASS 100% 138 in → 48 out (27 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:low PASS 100% 102 in → 22 out (10 reasoning)
Title: My Project v3
openai/gpt-5.4:low__json_schema PASS 100% 138 in → 30 out (9 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:medium PASS 100% 102 in → 28 out (16 reasoning)
Title: My Project v3
openai/gpt-5.4:medium__json_schema PASS 100% 138 in → 38 out (17 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:none PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4:none__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:xhigh PASS 100% 102 in → 66 out (54 reasoning)
Title: My Project v3
openai/gpt-5.4:xhigh__json_schema PASS 100% 138 in → 214 out (193 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:high PASS 100% 102 in → 64 out (52 reasoning)
Title: My Project v3
openai/gpt-5.5:high__json_schema PASS 100% 138 in → 280 out (259 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:low PASS 100% 102 in → 27 out (15 reasoning)
Title: My Project v3
openai/gpt-5.5:low__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:medium PASS 100% 102 in → 28 out (16 reasoning)
Title: My Project v3
openai/gpt-5.5:medium__json_schema PASS 100% 138 in → 262 out (241 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:none PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.5:none__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:xhigh PASS 100% 102 in → 97 out (85 reasoning)
Title: My Project v3
openai/gpt-5.5:xhigh__json_schema PASS 100% 138 in → 537 out (516 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:high PASS 100% 102 in → 33 out (21 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:high__json_schema PASS 100% 138 in → 46 out (25 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:low PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:low__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:max PASS 100% 102 in → 50 out (38 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:max__json_schema PASS 100% 138 in → 151 out (130 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:medium PASS 100% 102 in → 26 out (14 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:medium__json_schema PASS 100% 138 in → 35 out (14 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:none PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:none__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:xhigh PASS 100% 102 in → 42 out (30 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 138 in → 106 out (85 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:high PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:high__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:low PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:low__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:max PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:max__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:medium PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:medium__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:none PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:none__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:xhigh PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:high PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:high__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:low PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:low__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:max PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:max__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:medium PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:medium__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:none PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:none__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:xhigh PASS 100% 102 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 138 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-oss-120b:high PASS 100% 165 in → 179 out (201 reasoning)
Title: My Project v3
openai/gpt-oss-120b:high__json_schema PASS 100% 302 in → 299 out (278 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-oss-120b:low PASS 100% 161 in → 47 out (35 reasoning)
Title: My Project v3
openai/gpt-oss-120b:low__json_schema PASS 100% 165 in → 40 out (18 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-oss-120b:medium PASS 100% 163 in → 51 out (45 reasoning)
Title: My Project v3
openai/gpt-oss-120b:medium__json_schema PASS 100% 150 in → 86 out (71 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
openai/gpt-oss-120b:xhigh PASS 100% 163 in → 308 out (354 reasoning)
Title: My Project v3
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 161 in → 251 out (267 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
openai/gpt-oss-20b:high PASS 100% 161 in → 179 out (187 reasoning)
Title: My Project v3
openai/gpt-oss-20b:high__json_schema PASS 100% 214 in → 314 out (362 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-oss-20b:low PASS 100% 163 in → 43 out (27 reasoning)
Title: My Project v3
openai/gpt-oss-20b:low__json_schema PASS 100% 150 in → 47 out (23 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-oss-20b:medium PASS 100% 165 in → 89 out (80 reasoning)
Title: My Project v3
openai/gpt-oss-20b:medium__json_schema PASS 100% 150 in → 162 out (150 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-oss-20b:xhigh PASS 100% 165 in → 164 out (171 reasoning)
Title: My Project v3
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 161 in → 258 out (274 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
poolside/laguna-m.1:high PASS 100% 113 in → 235 out (226 reasoning)
Title: My Project v3
poolside/laguna-m.1:high__json_schema PASS 100% 113 in → 242 out (229 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-m.1:low PASS 100% 113 in → 254 out (246 reasoning)
Title: My Project v3
poolside/laguna-m.1:low__json_schema PASS 100% 113 in → 290 out (276 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-m.1:medium PASS 100% 113 in → 261 out (252 reasoning)
Title: My Project v3
poolside/laguna-m.1:medium__json_schema PASS 100% 113 in → 257 out (243 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-m.1:none__json_schema PASS 100% 113 in → 13 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-m.1:xhigh PASS 100% 113 in → 378 out (370 reasoning)
Title: My Project v3
poolside/laguna-m.1:xhigh__json_schema PASS 100% 113 in → 257 out (244 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-xs-2.1:high PASS 100% 113 in → 356 out (349 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:high__json_schema PASS 100% 113 in → 276 out (259 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:low PASS 100% 113 in → 244 out (238 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:low__json_schema PASS 100% 113 in → 253 out (235 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:medium PASS 100% 113 in → 160 out (153 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 113 in → 211 out (192 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:none PASS 100% 113 in → 6 out (0 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:none__json_schema PASS 100% 113 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:xhigh PASS 100% 113 in → 207 out (200 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 113 in → 234 out (216 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs.2:high PASS 100% 150 in → 175 out (166 reasoning)
Title: My Project v3
poolside/laguna-xs.2:high__json_schema PASS 100% 150 in → 190 out (170 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
poolside/laguna-xs.2:low PASS 100% 150 in → 201 out (192 reasoning)
Title: My Project v3
poolside/laguna-xs.2:low__json_schema PASS 100% 150 in → 198 out (184 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-xs.2:medium PASS 100% 150 in → 181 out (172 reasoning)
Title: My Project v3
poolside/laguna-xs.2:medium__json_schema PASS 100% 150 in → 188 out (174 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-xs.2:none PASS 100% 150 in → 8 out (0 reasoning)
Title: My Project v3
poolside/laguna-xs.2:none__json_schema PASS 100% 150 in → 13 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-xs.2:xhigh PASS 100% 150 in → 189 out (180 reasoning)
Title: My Project v3
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 150 in → 155 out (141 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
qwen/qwen3.6-27b:high PASS 100% 108 in → 923 out (1 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:high__json_schema PASS 100% 108 in → 812 out (813 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-27b:low PASS 100% 108 in → 427 out (421 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:low__json_schema PASS 100% 108 in → 951 out (946 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-27b:medium PASS 100% 108 in → 969 out (962 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:medium__json_schema PASS 100% 108 in → 924 out (922 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-27b:none PASS 100% 110 in → 7 out (0 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:none__json_schema PASS 100% 110 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
qwen/qwen3.6-27b:xhigh PASS 100% 108 in → 363 out (340 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 108 in → 1,112 out (1,098 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-35b-a3b:high PASS 100% 108 in → 665 out (672 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 108 in → 725 out (736 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
qwen/qwen3.6-35b-a3b:low PASS 100% 108 in → 788 out (803 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 108 in → 296 out (292 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
qwen/qwen3.6-35b-a3b:medium PASS 100% 108 in → 254 out (243 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 108 in → 383 out (387 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-35b-a3b:none PASS 100% 110 in → 7 out (0 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 110 in → 13 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 108 in → 1,089 out (1,041 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 108 in → 366 out (333 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-flash:high PASS 100% 108 in → 885 out (874 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:low PASS 100% 108 in → 192 out (181 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:medium PASS 100% 108 in → 143 out (132 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:none PASS 100% 110 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:xhigh PASS 100% 108 in → 846 out (835 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:high PASS 100% 108 in → 802 out (791 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:low PASS 100% 108 in → 812 out (801 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:max PASS 100% 108 in → 548 out (537 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:medium PASS 100% 108 in → 835 out (824 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:none PASS 100% 110 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:xhigh PASS 100% 108 in → 812 out (801 reasoning)
Title: My Project v3
qwen/qwen3.7-max:high PASS 100% 108 in → 262 out (251 reasoning)
Title: My Project v3
qwen/qwen3.7-max:low PASS 100% 108 in → 241 out (230 reasoning)
Title: My Project v3
qwen/qwen3.7-max:medium PASS 100% 108 in → 189 out (178 reasoning)
Title: My Project v3
qwen/qwen3.7-max:none PASS 100% 110 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.7-max:xhigh PASS 100% 108 in → 415 out (404 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:high PASS 100% 108 in → 165 out (154 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:low PASS 100% 108 in → 479 out (468 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:none PASS 100% 110 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:xhigh PASS 100% 108 in → 499 out (488 reasoning)
Title: My Project v3
tencent/hy3:high PASS 100% 112 in → 428 out (420 reasoning)
Title: My Project v3
tencent/hy3:high__json_schema PASS 100% 112 in → 597 out (593 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
tencent/hy3:low PASS 100% 112 in → 452 out (444 reasoning)
Title: My Project v3
tencent/hy3:low__json_schema PASS 100% 112 in → 340 out (355 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
tencent/hy3:medium PASS 100% 112 in → 321 out (313 reasoning)
Title: My Project v3
tencent/hy3:medium__json_schema PASS 100% 112 in → 400 out (401 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
tencent/hy3:none__json_schema PASS 100% 115 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
tencent/hy3:xhigh PASS 100% 112 in → 346 out (338 reasoning)
Title: My Project v3
tencent/hy3:xhigh__json_schema PASS 100% 112 in → 404 out (396 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:high PASS 100% 111 in → 318 out (339 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:high__json_schema PASS 100% 111 in → 218 out (206 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:low PASS 100% 111 in → 34 out (22 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:low__json_schema PASS 100% 111 in → 37 out (17 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:max PASS 100% 111 in → 299 out (300 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:max__json_schema PASS 100% 111 in → 284 out (264 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:medium PASS 100% 111 in → 185 out (173 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:medium__json_schema PASS 100% 111 in → 265 out (245 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:none PASS 100% 109 in → 9 out (0 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:xhigh PASS 100% 111 in → 260 out (275 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 111 in → 226 out (207 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling:high PASS 100% 111 in → 230 out (220 reasoning)
Title: My Project v3
thinkingmachines/inkling:low PASS 100% 111 in → 26 out (13 reasoning)
Title: My Project v3
thinkingmachines/inkling:low__json_schema PASS 100% 111 in → 34 out (15 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling:max PASS 100% 111 in → 146 out (134 reasoning)
Title: My Project v3
thinkingmachines/inkling:medium PASS 100% 111 in → 129 out (117 reasoning)
Title: My Project v3
thinkingmachines/inkling:medium__json_schema PASS 100% 111 in → 124 out (106 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
thinkingmachines/inkling:none PASS 100% 109 in → 9 out (0 reasoning)
Title: My Project v3
thinkingmachines/inkling:xhigh PASS 100% 111 in → 179 out (168 reasoning)
Title: My Project v3
x-ai/grok-4.3:high PASS 100% 281 in → 292 out (283 reasoning)
``` Title: My Project v3 ```
x-ai/grok-4.3:high__json_schema PASS 100% 338 in → 474 out (463 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.3:low PASS 100% 287 in → 168 out (162 reasoning)
Title: My Project v3
x-ai/grok-4.3:low__json_schema PASS 100% 344 in → 302 out (291 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.3:max PASS 100% 281 in → 257 out (251 reasoning)
Title: My Project v3
x-ai/grok-4.3:max__json_schema PASS 100% 338 in → 418 out (403 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.3:medium PASS 100% 287 in → 343 out (337 reasoning)
Title: My Project v3
x-ai/grok-4.3:medium__json_schema PASS 100% 344 in → 554 out (539 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.3:none PASS 100% 279 in → 6 out (0 reasoning)
Title: My Project v3
x-ai/grok-4.3:none__json_schema PASS 100% 336 in → 11 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.3:xhigh PASS 100% 281 in → 402 out (396 reasoning)
Title: My Project v3
x-ai/grok-4.3:xhigh__json_schema PASS 100% 338 in → 290 out (279 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.5:high PASS 100% 306 in → 188 out (182 reasoning)
Title: My Project v3
x-ai/grok-4.5:high__json_schema PASS 100% 375 in → 341 out (325 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.5:low PASS 100% 306 in → 186 out (180 reasoning)
Title: My Project v3
x-ai/grok-4.5:low__json_schema PASS 100% 375 in → 369 out (356 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.5:max PASS 100% 306 in → 266 out (260 reasoning)
Title: My Project v3
x-ai/grok-4.5:max__json_schema PASS 100% 375 in → 546 out (530 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.5:medium PASS 100% 306 in → 232 out (226 reasoning)
Title: My Project v3
x-ai/grok-4.5:medium__json_schema PASS 100% 375 in → 484 out (468 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.5:xhigh PASS 100% 306 in → 244 out (238 reasoning)
Title: My Project v3
x-ai/grok-4.5:xhigh__json_schema PASS 100% 375 in → 387 out (371 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-4.7-flash:high PASS 100% 101 in → 427 out (434 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:low PASS 100% 101 in → 270 out (271 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:medium PASS 100% 101 in → 363 out (357 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:none PASS 100% 101 in → 7 out (0 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:none__json_schema PASS 100% 101 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-4.7-flash:xhigh PASS 100% 101 in → 387 out (375 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 101 in → 411 out (405 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-5.2:high PASS 100% 108 in → 232 out (221 reasoning)
Title: My Project v3
z-ai/glm-5.2:high__json_schema PASS 100% 108 in → 357 out (340 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-5.2:low PASS 100% 108 in → 348 out (340 reasoning)
Title: My Project v3
z-ai/glm-5.2:low__json_schema PASS 100% 108 in → 208 out (188 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-5.2:medium PASS 100% 108 in → 271 out (263 reasoning)
Title: My Project v3
z-ai/glm-5.2:medium__json_schema PASS 100% 108 in → 262 out (262 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-5.2:none PASS 100% 102 in → 7 out (0 reasoning)
Title: My Project v3
z-ai/glm-5.2:none__json_schema PASS 100% 104 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-5.2:xhigh PASS 100% 108 in → 391 out (383 reasoning)
Title: My Project v3
z-ai/glm-5.2:xhigh__json_schema PASS 100% 108 in → 434 out (434 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
anthropic/claude-opus-4.6:max FAIL 0% 120 in → 175 out (83 reasoning)
``` Title: My Project v3 ``` **Reasoning:** The conflict is between `v2` (Branch A) and `v3` (Branch B/HEAD). Per the instruction that the higher version is more appropriate, `v3` is selected. The other changes (file_a and file_b) merge cleanly and require no intervention.
Failure: Content mismatch: ['project.txt']
anthropic/claude-sonnet-4.6:high FAIL 0% 120 in → 160 out (86 reasoning)
``` Title: My Project v3 ``` **Reason:** The conflict was resolved by choosing `My Project v3` (Branch B / HEAD), as the higher version number is the more appropriate change per the given instructions. The conflict markers are removed entirely.
Failure: Content mismatch: ['project.txt']
anthropic/claude-sonnet-4.6:max FAIL 0% 120 in → 232 out (132 reasoning)
``` Title: My Project v3 ``` **Reasoning:** Between v2 (Branch A) and v3 (Branch B), v3 is the higher version. In the conflicted file, `HEAD` contains `Title: My Project v3`, so that change is kept and the Branch A (`branch-a`) version is discarded. The conflict markers are removed entirely.
Failure: Content mismatch: ['project.txt']
anthropic/claude-sonnet-4.6:none FAIL 0% 120 in → 23 out (0 reasoning)
Here is the resolved project.txt content: ``` Title: My Project v3 ```
Failure: Content mismatch: ['project.txt']
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 120 in → 183 out (88 reasoning)
``` Title: My Project v3 ``` **Resolution reasoning:** The conflict was between `My Project v2` (Branch A) and `My Project v3` (Branch B/HEAD). Per the requirement that the higher version is more appropriate, `My Project v3` was selected as the resolved value, discarding the v2 change from Branch A.
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 107 in → 294 out (326 reasoning)
Title: My Projectv3
Failure: Content mismatch: ['project.txt']
arcee-ai/trinity-mini:low FAIL 0% 107 in → 344 out (398 reasoning)
Title: My Projectv3
Failure: Content mismatch: ['project.txt']
arcee-ai/trinity-mini:medium FAIL 0% 107 in → 376 out (410 reasoning)
Title:My Project v3
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
mistralai/devstral-2512 FAIL 0% 106 in → 30 out
Here is the resolved `project.txt` content with the higher version (v3) as requested: ``` Title: My Project v3 ```
Failure: Content mismatch: ['project.txt']
moonshotai/kimi-k2.6:medium FAIL 0% 106 in → 233 out (256 reasoning)
Title: My Project v3
Failure: Content mismatch: ['project.txt']
moonshotai/kimi-k2.6:xhigh FAIL 0% 106 in → 211 out (238 reasoning)
Title: My Project v3
Failure: Content mismatch: ['project.txt']
moonshotai/kimi-k2.7-code:high FAIL 0% 106 in → 252 out (244 reasoning)
Title: My Project v3
Failure: Content mismatch: ['project.txt']
moonshotai/kimi-k2.7-code:low FAIL 0% 106 in → 191 out (183 reasoning)
Title: My Project v3
Failure: Content mismatch: ['project.txt']
poolside/laguna-m.1:none FAIL 0% 113 in → 52 out (0 reasoning)
Title: My Project v3 This appears to be the conflicted content. Based on the requirement that the higher version is more appropriate, v3 should be selected. The resolved file content should be: Title: My Project v3
Failure: Content mismatch: ['project.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ef0b6e0c-79e9-9bb3-9e0e-5d7469d8adfc","request_id":"ef0b6e0c-79e9-9bb3-9e0e-5d7469d8adfc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7e9122f7-1dd8-9652-a9f0-c6491e6779b4","request_id":"7e9122f7-1dd8-9652-a9f0-c6491e6779b4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e051d244-571a-9195-ab27-6d20dcb2182d","request_id":"e051d244-571a-9195-ab27-6d20dcb2182d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a2eb888c-b436-96c5-998c-40723dbe47bf","request_id":"a2eb888c-b436-96c5-998c-40723dbe47bf"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2ac0b227-c3a7-91ab-b1ba-fc386aab5bc0","request_id":"2ac0b227-c3a7-91ab-b1ba-fc386aab5bc0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-eeaeaa99-25ee-9127-9cea-cf739785a306"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7afef151-ff85-94e8-8054-27d6ab379ace"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-366f13a5-2b4a-925d-8633-0d708134f931"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c527f634-87b6-93a8-a1d3-6a9494afdd47"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-25531d81-1ad3-91ea-9f5d-e18085d39031"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0b45f6b1-fb5b-941c-82b3-9c6ae945f4bd"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0f72a00c-f10e-9163-a11e-cee1d1e5db39","request_id":"0f72a00c-f10e-9163-a11e-cee1d1e5db39"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-703640d9-5e75-99e0-8891-2ae3e3bbc2e4","request_id":"703640d9-5e75-99e0-8891-2ae3e3bbc2e4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d9fb4b34-8e22-973f-9471-aafa517d6683","request_id":"d9fb4b34-8e22-973f-9471-aafa517d6683"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0d106f2d-ed36-978a-a196-5b8406f31d67","request_id":"0d106f2d-ed36-978a-a196-5b8406f31d67"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bc7f81b6-01a2-9a7a-9dfd-85644ea06eb5","request_id":"bc7f81b6-01a2-9a7a-9dfd-85644ea06eb5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9017d4fb-3c95-93bb-a7ff-dbd1db91275f","request_id":"9017d4fb-3c95-93bb-a7ff-dbd1db91275f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6dd8e897-5455-93a7-a3da-4da0656e7f64","request_id":"6dd8e897-5455-93a7-a3da-4da0656e7f64"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium FAIL 0% 108 in → 316 out (227 reasoning)
**Resolved `project.txt`:** ``` Title: My Project v3 ``` **Reasoning:** The conflict was between `v2` (Branch A) and `v3` (HEAD/Branch B). Since the instruction specifies to choose the higher version, `v3` is the correct resolution. The other files (`file_a` and `file_b`) had no conflicts and were merged cleanly.
Failure: Content mismatch: ['project.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-576c198c-fc9e-9c85-811f-150ef06d36f1","request_id":"576c198c-fc9e-9c85-811f-150ef06d36f1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1212eb99-857b-93ac-a57d-9fc9c62a62a6","request_id":"1212eb99-857b-93ac-a57d-9fc9c62a62a6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-39c0db2c-2f11-98e4-b2b9-b0bd1bcb2b5e","request_id":"39c0db2c-2f11-98e4-b2b9-b0bd1bcb2b5e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
Failure: Model 'tencent/hy3:none' violated the reasoning_level=none invariant: provider reported 425 reasoning token(s)
Invalid JSON. Output: Title: My Project v3{ "resolved_content": "Title: My Project v3" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3{ "resolved_content": "Title: My Project v3" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)