Three-way: multiple files with conflicting and non-conflicting changes
Tests ability to handle rebase with partial conflicts across files. Evaluates understanding of clean vs conflicted files in rebase.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Title: My Project' > project.txt
  5. 05 echo 'File A content' > file_a.txt
  6. 06 echo 'File B content' > file_b.txt
  7. 07 git add project.txt file_a.txt file_b.txt
  8. 08 git commit -m 'Initial project'
  9. 09 git checkout -b branch-a
  10. 10 echo 'Title: My Project v2' > project.txt
  11. 11 echo 'File A updated' > file_a.txt
  12. 12 git add project.txt file_a.txt
  13. 13 git commit -m 'Update title and file A'
  14. 14 git checkout main
  15. 15 echo 'Title: My Project v3' > project.txt
  16. 16 echo 'File B updated' > file_b.txt
  17. 17 git add project.txt file_b.txt
  18. 18 git commit -m 'Update title and file B'
  19. 19 git rebase branch-a
Prompt
Resolve the rebase conflict in project.txt. Branch being rebased set Title: My Project v2. Main branch set Title: My Project v3. Both branches also updated different files (file_a and file_b) without conflicts. For the project.txt conflict, the higher version is more appropriate. Provide ONLY the resolved project.txt content.
Expected
Title: My Project v3

Scoped model quality, cost, API time, and token usage for rebase/f008.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 185 in → 166 out (42 reasoning)
``` Title: My Project v3 ```
anthropic/claude-fable-5:high__json_schema PASS 100% 406 in → 130 out (51 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-fable-5:low PASS 100% 187 in → 48 out (27 reasoning)
Title: My Project v3
anthropic/claude-fable-5:low__json_schema PASS 100% 406 in → 48 out (30 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-fable-5:max PASS 100% 187 in → 540 out (182 reasoning)
Title: My Project v3
anthropic/claude-fable-5:max__json_schema PASS 100% 406 in → 568 out (65 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-fable-5:medium PASS 100% 187 in → 113 out (112 reasoning)
``` Title: My Project v3 ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 405 in → 33 out (8 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 406 in → 143 out (87 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-haiku-4.5:high PASS 100% 161 in → 199 out (188 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 331 in → 235 out (215 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-haiku-4.5:low PASS 100% 161 in → 225 out (199 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 331 in → 207 out (191 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-haiku-4.5:medium PASS 100% 163 in → 202 out (187 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 330 in → 222 out (192 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-haiku-4.5:none PASS 100% 133 in → 13 out (0 reasoning)
``` Title: My Project v3 ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 299 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 161 in → 257 out (261 reasoning)
Title: My Project v3
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 330 in → 250 out (214 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-opus-4.6:high PASS 100% 133 in → 14 out (0 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 302 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.6:low PASS 100% 133 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.6:low__json_schema PASS 100% 301 in → 15 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 300 in → 680 out (362 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.6:medium PASS 100% 131 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 301 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.6:none PASS 100% 132 in → 13 out (0 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 300 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 133 in → 14 out (0 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 300 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:high PASS 100% 191 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:high__json_schema PASS 100% 412 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:low PASS 100% 193 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:low__json_schema PASS 100% 413 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:max PASS 100% 192 in → 2,033 out (253 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:max__json_schema PASS 100% 413 in → 489 out (141 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:medium PASS 100% 192 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 413 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:none PASS 100% 193 in → 12 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:none__json_schema PASS 100% 415 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 192 in → 496 out (131 reasoning)
Title: My Project v3
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 412 in → 25 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.8:high PASS 100% 188 in → 482 out (145 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 407 in → 135 out (75 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-opus-4.8:low PASS 100% 188 in → 216 out (139 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:low__json_schema PASS 100% 407 in → 21 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-opus-4.8:max PASS 100% 187 in → 1,552 out (214 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 407 in → 1,611 out (163 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-opus-4.8:medium PASS 100% 185 in → 137 out (79 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 407 in → 261 out (73 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-opus-4.8:none PASS 100% 188 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-opus-4.8:none__json_schema PASS 100% 406 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
anthropic/claude-opus-4.8:xhigh PASS 100% 188 in → 665 out (88 reasoning)
``` Title: My Project v3 ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 409 in → 388 out (137 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 304 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 132 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 302 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:max PASS 100% 133 in → 816 out (435 reasoning)
``` Title: My Project v3 ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 302 in → 111 out (83 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 132 in → 56 out (29 reasoning)
Title: My Project v3
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 299 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 134 in → 9 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 300 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 132 in → 887 out (406 reasoning)
``` Title: My Project v3 ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 301 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-5:high PASS 100% 186 in → 258 out (60 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:high__json_schema PASS 100% 405 in → 183 out (42 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
anthropic/claude-sonnet-5:low PASS 100% 187 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:low__json_schema PASS 100% 407 in → 23 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-sonnet-5:medium PASS 100% 186 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 406 in → 23 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
anthropic/claude-sonnet-5:none PASS 100% 188 in → 10 out (0 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:none__json_schema PASS 100% 409 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
anthropic/claude-sonnet-5:xhigh PASS 100% 186 in → 1,458 out (189 reasoning)
Title: My Project v3
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 408 in → 1,464 out (374 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
arcee-ai/trinity-large-thinking:high PASS 100% 119 in → 476 out (469 reasoning)
Title: My Project v3
arcee-ai/trinity-large-thinking:low PASS 100% 119 in → 361 out (353 reasoning)
Title: My Project v3
arcee-ai/trinity-large-thinking:medium PASS 100% 117 in → 511 out (505 reasoning)
Title: My Project v3
arcee-ai/trinity-large-thinking:xhigh PASS 100% 118 in → 316 out (308 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:high PASS 100% 119 in → 227 out (246 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:high__json_schema PASS 100% 117 in → 264 out (275 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
arcee-ai/trinity-mini:low PASS 100% 120 in → 302 out (343 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:medium PASS 100% 119 in → 348 out (382 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:medium__json_schema PASS 100% 117 in → 272 out (292 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
arcee-ai/trinity-mini:xhigh PASS 100% 117 in → 301 out (323 reasoning)
Title: My Project v3
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 118 in → 289 out (315 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 197 in → 106 out (114 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 201 in → 287 out (297 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 118 in → 38 out (47 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 118 in → 209 out (216 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 209 in → 587 out (623 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 197 in → 126 out (135 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 117 in → 79 out (72 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:none PASS 100% 118 in → 7 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 120 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 198 in → 158 out (166 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 119 in → 60 out (43 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash:high PASS 100% 119 in → 219 out (216 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 121 in → 137 out (135 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash:low PASS 100% 117 in → 230 out (222 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 215 in → 13 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
deepseek/deepseek-v4-flash:medium PASS 100% 118 in → 67 out (64 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:none PASS 100% 118 in → 6 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 119 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 198 in → 347 out (339 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:high PASS 100% 117 in → 133 out (125 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 309 in → 127 out (112 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
deepseek/deepseek-v4-pro:low PASS 100% 118 in → 7 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 117 in → 133 out (116 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
deepseek/deepseek-v4-pro:medium PASS 100% 119 in → 759 out (751 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 309 in → 45 out (30 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
deepseek/deepseek-v4-pro:none PASS 100% 117 in → 7 out (0 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:xhigh PASS 100% 197 in → 1,557 out (1,549 reasoning)
Title: My Project v3
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 386 in → 262 out (247 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3-flash-preview:high PASS 100% 122 in → 443 out (437 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:high__json_schema PASS 100% 123 in → 320 out (308 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3-flash-preview:low PASS 100% 121 in → 238 out (232 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:low__json_schema PASS 100% 181 in → 276 out (264 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3-flash-preview:medium PASS 100% 120 in → 250 out (244 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:medium__json_schema PASS 100% 182 in → 387 out (374 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3-flash-preview:xhigh PASS 100% 121 in → 135 out (129 reasoning)
Title: My Project v3
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 122 in → 249 out (237 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 123 in → 218 out (212 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 181 in → 463 out (450 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 120 in → 134 out (128 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 182 in → 118 out (105 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 120 in → 208 out (202 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 182 in → 205 out (192 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 122 in → 295 out (289 reasoning)
Title: My Project v3
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 121 in → 506 out (493 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.1-pro-preview:high PASS 100% 122 in → 631 out (625 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 182 in → 732 out (720 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.1-pro-preview:low PASS 100% 120 in → 367 out (361 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 180 in → 99 out (84 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
google/gemini-3.1-pro-preview:medium PASS 100% 121 in → 344 out (338 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 181 in → 412 out (400 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 122 in → 549 out (543 reasoning)
Title: My Project v3
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 179 in → 671 out (659 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.5-flash-lite:high PASS 100% 121 in → 269 out (263 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 122 in → 515 out (497 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.5-flash-lite:low PASS 100% 123 in → 6 out (0 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 182 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.5-flash-lite:max PASS 100% 122 in → 254 out (248 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 182 in → 594 out (576 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.5-flash-lite:medium PASS 100% 122 in → 236 out (230 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 181 in → 353 out (340 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 121 in → 329 out (323 reasoning)
Title: My Project v3
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 123 in → 505 out (493 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.5-flash:high PASS 100% 121 in → 448 out (442 reasoning)
Title: My Project v3
google/gemini-3.5-flash:high__json_schema PASS 100% 180 in → 666 out (646 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
google/gemini-3.5-flash:low PASS 100% 120 in → 467 out (461 reasoning)
Title: My Project v3
google/gemini-3.5-flash:low__json_schema PASS 100% 180 in → 163 out (150 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.5-flash:medium PASS 100% 120 in → 461 out (455 reasoning)
Title: My Project v3
google/gemini-3.5-flash:medium__json_schema PASS 100% 182 in → 385 out (373 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.5-flash:xhigh PASS 100% 122 in → 423 out (417 reasoning)
Title: My Project v3
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 181 in → 516 out (503 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemini-3.6-flash:high PASS 100% 121 in → 240 out (234 reasoning)
Title: My Project v3
google/gemini-3.6-flash:high__json_schema PASS 100% 123 in → 429 out (417 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.6-flash:low PASS 100% 123 in → 202 out (196 reasoning)
Title: My Project v3
google/gemini-3.6-flash:low__json_schema PASS 100% 123 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemini-3.6-flash:max PASS 100% 120 in → 339 out (333 reasoning)
Title: My Project v3
google/gemini-3.6-flash:max__json_schema PASS 100% 182 in → 460 out (448 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.6-flash:medium PASS 100% 123 in → 266 out (260 reasoning)
Title: My Project v3
google/gemini-3.6-flash:medium__json_schema PASS 100% 122 in → 337 out (325 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemini-3.6-flash:xhigh PASS 100% 121 in → 290 out (284 reasoning)
Title: My Project v3
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 123 in → 460 out (448 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
google/gemma-4-26b-a4b-it:high PASS 100% 137 in → 193 out (168 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 135 in → 215 out (182 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:low PASS 100% 135 in → 256 out (222 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 138 in → 291 out (242 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:medium PASS 100% 136 in → 562 out (514 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 137 in → 256 out (213 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:none PASS 100% 135 in → 7 out (0 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 135 in → 18 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 136 in → 232 out (198 reasoning)
Title: My Project v3
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 137 in → 247 out (213 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-31b-it:high PASS 100% 136 in → 131 out (110 reasoning)
Title: My Project v3
google/gemma-4-31b-it:high__json_schema PASS 100% 138 in → 293 out (1 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemma-4-31b-it:low PASS 100% 138 in → 196 out (158 reasoning)
Title: My Project v3
google/gemma-4-31b-it:low__json_schema PASS 100% 136 in → 208 out (159 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-31b-it:medium PASS 100% 136 in → 139 out (117 reasoning)
Title: My Project v3
google/gemma-4-31b-it:medium__json_schema PASS 100% 137 in → 233 out (182 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
google/gemma-4-31b-it:none PASS 100% 134 in → 7 out (0 reasoning)
Title: My Project v3
google/gemma-4-31b-it:none__json_schema PASS 100% 134 in → 14 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
google/gemma-4-31b-it:xhigh PASS 100% 137 in → 153 out (1 reasoning)
Title: My Project v3
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 136 in → 203 out (168 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
ibm-granite/granite-4.1-8b PASS 100% 129 in → 7 out
Title: My Project v3
ibm-granite/granite-4.1-8b__json_schema PASS 100% 127 in → 17 out
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
inclusionai/ling-2.6-flash PASS 100% 135 in → 7 out
Title: My Project v3
inclusionai/ling-2.6-flash__json_schema PASS 100% 137 in → 18 out
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
liquid/lfm-2-24b-a2b PASS 100% 126 in → 10 out
project.txt Title: My Project v3
minimax/minimax-m2.5:high PASS 100% 150 in → 331 out (384 reasoning)
Title: My Project v3
minimax/minimax-m2.5:high__json_schema PASS 100% 150 in → 189 out (193 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.5:low__json_schema PASS 100% 150 in → 279 out (307 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
minimax/minimax-m2.5:medium PASS 100% 149 in → 223 out (253 reasoning)
Title: My Project v3
minimax/minimax-m2.5:medium__json_schema PASS 100% 149 in → 351 out (335 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.5:xhigh PASS 100% 148 in → 158 out (148 reasoning)
Title: My Project v3
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 161 in → 142 out (149 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.7:high PASS 100% 148 in → 385 out (376 reasoning)
Title: My Project v3
minimax/minimax-m2.7:high__json_schema PASS 100% 148 in → 266 out (310 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m2.7:low PASS 100% 149 in → 246 out (288 reasoning)
Title: My Project v3
minimax/minimax-m2.7:medium PASS 100% 149 in → 223 out (214 reasoning)
Title: My Project v3
minimax/minimax-m2.7:medium__json_schema PASS 100% 282 in → 257 out (244 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
minimax/minimax-m2.7:xhigh PASS 100% 151 in → 224 out (217 reasoning)
Title: My Project v3
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 283 in → 402 out (389 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
minimax/minimax-m3:high PASS 100% 286 in → 83 out (82 reasoning)
Title: My Project v3
minimax/minimax-m3:high__json_schema PASS 100% 235 in → 19 out (14 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
minimax/minimax-m3:low PASS 100% 289 in → 84 out (86 reasoning)
Title: My Project v3
minimax/minimax-m3:medium PASS 100% 287 in → 42 out (38 reasoning)
Title: My Project v3
minimax/minimax-m3:medium__json_schema PASS 100% 273 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
minimax/minimax-m3:xhigh PASS 100% 286 in → 74 out (66 reasoning)
Title: My Project v3
minimax/minimax-m3:xhigh__json_schema PASS 100% 275 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
mistralai/devstral-2512__json_schema PASS 100% 119 in → 14 out
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:high PASS 100% 133 in → 292 out (299 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 120 in → 140 out (137 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:low PASS 100% 133 in → 348 out (363 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 119 in → 252 out (254 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:medium PASS 100% 131 in → 341 out (360 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 121 in → 198 out (218 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
mistralai/mistral-medium-3-5:none PASS 100% 131 in → 7 out (0 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:xhigh PASS 100% 132 in → 257 out (260 reasoning)
Title: My Project v3
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 120 in → 214 out (219 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
moonshotai/kimi-k2.6:high__json_schema PASS 100% 116 in → 643 out (418 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 117 in → 2,247 out (2,483 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 117 in → 905 out (761 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.7-code:high PASS 100% 117 in → 76 out (68 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 201 in → 216 out (201 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
moonshotai/kimi-k2.7-code:low PASS 100% 120 in → 213 out (205 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:medium PASS 100% 116 in → 178 out (170 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 118 in → 159 out (141 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 118 in → 91 out (83 reasoning)
Title: My Project v3
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 117 in → 133 out (119 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
moonshotai/kimi-k3:high PASS 100% 196 in → 225 out (204 reasoning)
Title: My Project v3
moonshotai/kimi-k3:high__json_schema PASS 100% 195 in → 353 out (332 reasoning)
Title: My Project v3
moonshotai/kimi-k3:low PASS 100% 194 in → 639 out (618 reasoning)
Title: My Project v3
moonshotai/kimi-k3:low__json_schema PASS 100% 193 in → 484 out (463 reasoning)
Title: My Project v3
moonshotai/kimi-k3:max PASS 100% 197 in → 739 out (718 reasoning)
Title: My Project v3
moonshotai/kimi-k3:max__json_schema PASS 100% 195 in → 758 out (737 reasoning)
Title: My Project v3
moonshotai/kimi-k3:medium PASS 100% 193 in → 93 out (72 reasoning)
Title: My Project v3
moonshotai/kimi-k3:medium__json_schema PASS 100% 196 in → 253 out (232 reasoning)
Title: My Project v3
moonshotai/kimi-k3:xhigh PASS 100% 196 in → 417 out (393 reasoning)
``` Title: My Project v3 ```
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 195 in → 106 out (85 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 134 in → 95 out (101 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 134 in → 177 out (192 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 134 in → 228 out (241 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 132 in → 1,493 out (1,714 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 134 in → 104 out (98 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 133 in → 7 out (0 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 133 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 132 in → 82 out (87 reasoning)
Title: My Project v3
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 134 in → 86 out (84 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 131 in → 38 out (29 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 132 in → 406 out (374 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 131 in → 38 out (29 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 131 in → 51 out (33 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 134 in → 80 out (81 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 134 in → 44 out (27 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 134 in → 7 out (0 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 132 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 132 in → 37 out (32 reasoning)
Title: My Project v3
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 133 in → 59 out (41 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-5.4-mini:high PASS 100% 116 in → 92 out (80 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:high__json_schema PASS 100% 149 in → 113 out (92 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-mini:low PASS 100% 114 in → 31 out (19 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:low__json_schema PASS 100% 149 in → 47 out (25 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.4-mini:medium PASS 100% 113 in → 38 out (26 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:medium__json_schema PASS 100% 151 in → 83 out (62 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-mini:none PASS 100% 114 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:none__json_schema PASS 100% 150 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-mini:xhigh PASS 100% 114 in → 109 out (97 reasoning)
Title: My Project v3
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 150 in → 146 out (125 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-nano:high PASS 100% 113 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:high__json_schema PASS 100% 149 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.4-nano:low PASS 100% 115 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:low__json_schema PASS 100% 150 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-nano:medium PASS 100% 115 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:medium__json_schema PASS 100% 149 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.4-nano:none PASS 100% 114 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:none__json_schema PASS 100% 151 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4-nano:xhigh PASS 100% 114 in → 68 out (56 reasoning)
Title: My Project v3
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 149 in → 131 out (109 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.4:high PASS 100% 115 in → 42 out (30 reasoning)
Title: My Project v3
openai/gpt-5.4:high__json_schema PASS 100% 149 in → 104 out (83 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:low PASS 100% 113 in → 22 out (10 reasoning)
Title: My Project v3
openai/gpt-5.4:low__json_schema PASS 100% 151 in → 30 out (9 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:medium PASS 100% 115 in → 32 out (20 reasoning)
Title: My Project v3
openai/gpt-5.4:medium__json_schema PASS 100% 150 in → 41 out (20 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:none PASS 100% 115 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.4:none__json_schema PASS 100% 149 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.4:xhigh PASS 100% 113 in → 177 out (165 reasoning)
Title: My Project v3
openai/gpt-5.4:xhigh__json_schema PASS 100% 149 in → 374 out (353 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:high PASS 100% 114 in → 61 out (49 reasoning)
Title: My Project v3
openai/gpt-5.5:high__json_schema PASS 100% 150 in → 537 out (516 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:low PASS 100% 115 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.5:low__json_schema PASS 100% 149 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:medium PASS 100% 114 in → 76 out (64 reasoning)
Title: My Project v3
openai/gpt-5.5:medium__json_schema PASS 100% 148 in → 276 out (255 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:none PASS 100% 114 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.5:none__json_schema PASS 100% 149 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.5:xhigh PASS 100% 114 in → 179 out (167 reasoning)
Title: My Project v3
openai/gpt-5.5:xhigh__json_schema PASS 100% 149 in → 537 out (516 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:high PASS 100% 114 in → 31 out (19 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:high__json_schema PASS 100% 149 in → 197 out (176 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:low PASS 100% 113 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:low__json_schema PASS 100% 150 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:max PASS 100% 114 in → 68 out (56 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:max__json_schema PASS 100% 150 in → 132 out (111 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:medium PASS 100% 113 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:medium__json_schema PASS 100% 150 in → 39 out (18 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:none PASS 100% 114 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:none__json_schema PASS 100% 149 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-luna:xhigh PASS 100% 115 in → 42 out (30 reasoning)
Title: My Project v3
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 150 in → 52 out (31 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:high PASS 100% 113 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:high__json_schema PASS 100% 148 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:low PASS 100% 114 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:low__json_schema PASS 100% 150 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:max PASS 100% 113 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:max__json_schema PASS 100% 151 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.6-sol:medium PASS 100% 114 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:medium__json_schema PASS 100% 149 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.6-sol:none PASS 100% 113 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:none__json_schema PASS 100% 150 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-sol:xhigh PASS 100% 115 in → 64 out (52 reasoning)
Title: My Project v3
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 149 in → 20 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3\n"}
openai/gpt-5.6-terra:high PASS 100% 114 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:high__json_schema PASS 100% 151 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:low PASS 100% 113 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:low__json_schema PASS 100% 151 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:max PASS 100% 114 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:max__json_schema PASS 100% 150 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:medium PASS 100% 113 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:medium__json_schema PASS 100% 150 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:none PASS 100% 114 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:none__json_schema PASS 100% 148 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-5.6-terra:xhigh PASS 100% 114 in → 10 out (0 reasoning)
Title: My Project v3
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 149 in → 19 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-oss-120b:high PASS 100% 175 in → 74 out (68 reasoning)
Title: My Project v3
openai/gpt-oss-120b:high__json_schema PASS 100% 175 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-oss-120b:low PASS 100% 174 in → 89 out (73 reasoning)
Title: My Project v3
openai/gpt-oss-120b:low__json_schema PASS 100% 179 in → 97 out (84 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
openai/gpt-oss-120b:medium PASS 100% 174 in → 104 out (106 reasoning)
Title: My Project v3
openai/gpt-oss-120b:medium__json_schema PASS 100% 172 in → 120 out (111 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
openai/gpt-oss-120b:xhigh PASS 100% 175 in → 49 out (43 reasoning)
Title: My Project v3
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 176 in → 212 out (186 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
openai/gpt-oss-20b:high PASS 100% 176 in → 446 out (507 reasoning)
Title: My Project v3
openai/gpt-oss-20b:high__json_schema PASS 100% 162 in → 177 out (178 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
openai/gpt-oss-20b:low PASS 100% 163 in → 42 out (27 reasoning)
Title: My Project v3
openai/gpt-oss-20b:low__json_schema PASS 100% 162 in → 52 out (29 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
openai/gpt-oss-20b:medium PASS 100% 162 in → 90 out (86 reasoning)
Title: My Project v3
openai/gpt-oss-20b:medium__json_schema PASS 100% 174 in → 72 out (59 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
openai/gpt-oss-20b:xhigh PASS 100% 176 in → 277 out (315 reasoning)
Title: My Project v3
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 161 in → 292 out (304 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
poolside/laguna-m.1:high PASS 100% 127 in → 249 out (240 reasoning)
Title: My Project v3
poolside/laguna-m.1:high__json_schema PASS 100% 127 in → 478 out (459 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
poolside/laguna-m.1:low PASS 100% 126 in → 382 out (374 reasoning)
Title: My Project v3
poolside/laguna-m.1:low__json_schema PASS 100% 127 in → 328 out (310 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-m.1:medium PASS 100% 125 in → 252 out (243 reasoning)
Title: My Project v3
poolside/laguna-m.1:medium__json_schema PASS 100% 126 in → 383 out (365 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-m.1:none PASS 100% 128 in → 8 out (0 reasoning)
Title: My Project v3
poolside/laguna-m.1:xhigh PASS 100% 127 in → 229 out (220 reasoning)
Title: My Project v3
poolside/laguna-m.1:xhigh__json_schema PASS 100% 126 in → 260 out (247 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-xs-2.1:high PASS 100% 126 in → 221 out (214 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:high__json_schema PASS 100% 128 in → 199 out (180 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:low PASS 100% 127 in → 194 out (187 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:low__json_schema PASS 100% 126 in → 228 out (209 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:medium PASS 100% 128 in → 198 out (191 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 126 in → 223 out (204 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:none PASS 100% 126 in → 6 out (0 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:none__json_schema PASS 100% 128 in → 17 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
poolside/laguna-xs-2.1:xhigh PASS 100% 125 in → 219 out (212 reasoning)
Title: My Project v3
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 126 in → 507 out (489 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
poolside/laguna-xs.2:high PASS 100% 164 in → 148 out (139 reasoning)
Title: My Project v3
poolside/laguna-xs.2:high__json_schema PASS 100% 164 in → 124 out (110 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-xs.2:low PASS 100% 163 in → 236 out (228 reasoning)
Title: My Project v3
poolside/laguna-xs.2:low__json_schema PASS 100% 162 in → 142 out (128 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-xs.2:medium PASS 100% 165 in → 162 out (153 reasoning)
Title: My Project v3
poolside/laguna-xs.2:medium__json_schema PASS 100% 164 in → 145 out (131 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-xs.2:none__json_schema PASS 100% 165 in → 13 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
poolside/laguna-xs.2:xhigh PASS 100% 163 in → 163 out (154 reasoning)
Title: My Project v3
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 164 in → 201 out (186 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
qwen/qwen3.6-27b:high PASS 100% 123 in → 1,036 out (1,018 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:high__json_schema PASS 100% 124 in → 1,247 out (1 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-27b:low PASS 100% 121 in → 1,030 out (1,019 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:low__json_schema PASS 100% 123 in → 1,084 out (1,072 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-27b:medium PASS 100% 122 in → 766 out (780 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:medium__json_schema PASS 100% 123 in → 1,170 out (1,102 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-27b:none PASS 100% 123 in → 7 out (0 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:xhigh PASS 100% 122 in → 1,043 out (1,027 reasoning)
Title: My Project v3
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 124 in → 1,093 out (1 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-35b-a3b:high PASS 100% 123 in → 955 out (935 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 124 in → 914 out (927 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-35b-a3b:low PASS 100% 123 in → 939 out (918 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 123 in → 943 out (936 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
qwen/qwen3.6-35b-a3b:medium PASS 100% 124 in → 1,082 out (1,071 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 124 in → 767 out (734 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-35b-a3b:none PASS 100% 123 in → 7 out (0 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 124 in → 985 out (993 reasoning)
Title: My Project v3
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 122 in → 799 out (778 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
qwen/qwen3.6-flash:high PASS 100% 124 in → 1,125 out (1,114 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:low PASS 100% 122 in → 995 out (984 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:medium PASS 100% 122 in → 784 out (773 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:none PASS 100% 125 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.6-flash:xhigh PASS 100% 122 in → 670 out (659 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:high PASS 100% 123 in → 506 out (495 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:low PASS 100% 124 in → 993 out (982 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:max PASS 100% 124 in → 1,010 out (999 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:medium PASS 100% 122 in → 445 out (434 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:none PASS 100% 126 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.7-flash:xhigh PASS 100% 123 in → 404 out (393 reasoning)
Title: My Project v3
qwen/qwen3.7-max:high PASS 100% 122 in → 263 out (252 reasoning)
Title: My Project v3
qwen/qwen3.7-max:low PASS 100% 123 in → 608 out (597 reasoning)
Title: My Project v3
qwen/qwen3.7-max:medium PASS 100% 123 in → 444 out (433 reasoning)
Title: My Project v3
qwen/qwen3.7-max:none PASS 100% 125 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.7-max:xhigh PASS 100% 122 in → 667 out (656 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:high PASS 100% 123 in → 621 out (610 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:low PASS 100% 123 in → 632 out (621 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:medium PASS 100% 124 in → 554 out (543 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:none PASS 100% 126 in → 6 out (0 reasoning)
Title: My Project v3
qwen/qwen3.7-plus:xhigh PASS 100% 122 in → 609 out (598 reasoning)
Title: My Project v3
tencent/hy3:high PASS 100% 124 in → 323 out (315 reasoning)
Title: My Project v3
tencent/hy3:high__json_schema PASS 100% 123 in → 330 out (326 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
tencent/hy3:low PASS 100% 125 in → 285 out (306 reasoning)
Title: My Project v3
tencent/hy3:low__json_schema PASS 100% 123 in → 500 out (501 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
tencent/hy3:medium PASS 100% 124 in → 418 out (410 reasoning)
Title: My Project v3
tencent/hy3:medium__json_schema PASS 100% 124 in → 658 out (664 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
tencent/hy3:none PASS 100% 127 in → 7 out (0 reasoning)
Title: My Project v3
tencent/hy3:xhigh PASS 100% 124 in → 432 out (424 reasoning)
Title: My Project v3
tencent/hy3:xhigh__json_schema PASS 100% 124 in → 442 out (454 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:high PASS 100% 125 in → 196 out (205 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:high__json_schema PASS 100% 122 in → 219 out (199 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:low PASS 100% 123 in → 44 out (32 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:low__json_schema PASS 100% 123 in → 47 out (28 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:max PASS 100% 123 in → 223 out (239 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:max__json_schema PASS 100% 123 in → 739 out (718 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:medium PASS 100% 124 in → 53 out (41 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:medium__json_schema PASS 100% 124 in → 144 out (125 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling-small:xhigh PASS 100% 123 in → 770 out (819 reasoning)
Title: My Project v3
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 123 in → 501 out (483 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3\n"}
thinkingmachines/inkling:high PASS 100% 124 in → 154 out (165 reasoning)
Title: My Project v3
thinkingmachines/inkling:high__json_schema PASS 100% 123 in → 237 out (217 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
thinkingmachines/inkling:low PASS 100% 124 in → 27 out (17 reasoning)
Title: My Project v3
thinkingmachines/inkling:max PASS 100% 122 in → 145 out (132 reasoning)
Title: My Project v3
thinkingmachines/inkling:max__json_schema PASS 100% 122 in → 451 out (432 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling:medium PASS 100% 123 in → 46 out (33 reasoning)
Title: My Project v3
thinkingmachines/inkling:medium__json_schema PASS 100% 122 in → 99 out (80 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
thinkingmachines/inkling:xhigh PASS 100% 121 in → 178 out (167 reasoning)
Title: My Project v3
thinkingmachines/inkling:xhigh__json_schema PASS 100% 124 in → 150 out (133 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.3:high PASS 100% 293 in → 236 out (230 reasoning)
Title: My Project v3
x-ai/grok-4.3:high__json_schema PASS 100% 352 in → 687 out (672 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.3:low PASS 100% 299 in → 386 out (380 reasoning)
Title: My Project v3
x-ai/grok-4.3:low__json_schema PASS 100% 354 in → 290 out (279 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.3:max PASS 100% 293 in → 432 out (426 reasoning)
Title: My Project v3
x-ai/grok-4.3:max__json_schema PASS 100% 351 in → 391 out (380 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.3:medium PASS 100% 299 in → 484 out (478 reasoning)
Title: My Project v3
x-ai/grok-4.3:medium__json_schema PASS 100% 357 in → 588 out (573 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.3:none PASS 100% 289 in → 6 out (0 reasoning)
Title: My Project v3
x-ai/grok-4.3:none__json_schema PASS 100% 347 in → 12 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
x-ai/grok-4.3:xhigh PASS 100% 293 in → 231 out (225 reasoning)
Title: My Project v3
x-ai/grok-4.3:xhigh__json_schema PASS 100% 350 in → 340 out (329 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content":"Title: My Project v3"}
x-ai/grok-4.5:high PASS 100% 315 in → 498 out (492 reasoning)
Title: My Project v3
x-ai/grok-4.5:high__json_schema PASS 100% 386 in → 1,004 out (991 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
x-ai/grok-4.5:low PASS 100% 316 in → 264 out (258 reasoning)
Title: My Project v3
x-ai/grok-4.5:low__json_schema PASS 100% 384 in → 445 out (432 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) {"resolved_content": "Title: My Project v3"}
x-ai/grok-4.5:max PASS 100% 316 in → 837 out (831 reasoning)
Title: My Project v3
x-ai/grok-4.5:max__json_schema PASS 100% 385 in → 634 out (616 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
x-ai/grok-4.5:medium PASS 100% 315 in → 381 out (375 reasoning)
Title: My Project v3
x-ai/grok-4.5:medium__json_schema PASS 100% 384 in → 278 out (262 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
x-ai/grok-4.5:xhigh PASS 100% 316 in → 716 out (710 reasoning)
Title: My Project v3
x-ai/grok-4.5:xhigh__json_schema PASS 100% 385 in → 311 out (293 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3\n" }
z-ai/glm-4.7-flash:high PASS 100% 114 in → 394 out (371 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:low PASS 100% 113 in → 403 out (401 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:low__json_schema PASS 100% 113 in → 392 out (374 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-4.7-flash:medium PASS 100% 114 in → 376 out (376 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:none PASS 100% 115 in → 6 out (0 reasoning)
Title: My Project v3
z-ai/glm-4.7-flash:none__json_schema PASS 100% 114 in → 16 out (0 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-4.7-flash:xhigh PASS 100% 114 in → 375 out (367 reasoning)
Title: My Project v3
z-ai/glm-5.2:high PASS 100% 121 in → 218 out (210 reasoning)
Title: My Project v3
z-ai/glm-5.2:low PASS 100% 120 in → 381 out (373 reasoning)
Title: My Project v3
z-ai/glm-5.2:low__json_schema PASS 100% 122 in → 271 out (245 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-5.2:medium PASS 100% 119 in → 349 out (341 reasoning)
Title: My Project v3
z-ai/glm-5.2:medium__json_schema PASS 100% 122 in → 227 out (204 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
z-ai/glm-5.2:none PASS 100% 115 in → 7 out (0 reasoning)
Title: My Project v3
z-ai/glm-5.2:xhigh PASS 100% 120 in → 345 out (335 reasoning)
Title: My Project v3
z-ai/glm-5.2:xhigh__json_schema PASS 100% 120 in → 727 out (708 reasoning)
Title: My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v3" }
anthropic/claude-fable-5:xhigh FAIL 0% 187 in → 270 out (82 reasoning)
``` Title: My Project v3 ``` The resolution keeps `Title: My Project v3` since the higher version is more appropriate, and removes all conflict markers (`<<<<<<< HEAD`, `=======`, `>>>>>>> 9d19478`). The non-conflicting changes to file_a and file_b require no action, as Git merges those automatically.
Failure: Content mismatch: ['project.txt']
anthropic/claude-opus-4.6:max FAIL 0% 131 in → 828 out (347 reasoning)
``` Title: My Project v3 ``` **Reasoning:** The user specified that the higher version is more appropriate. **v3 > v2**, so `Title: My Project v3` (from the main branch) is the correct resolution. The other changes (to `file_a` and `file_b`) applied cleanly without conflicts and require no action.
Failure: Content mismatch: ['project.txt']
anthropic/claude-sonnet-4.6:high FAIL 0% 134 in → 287 out (167 reasoning)
``` Title: My Project v3 ``` **Resolution reasoning:** In a rebase conflict, `HEAD` represents the **main branch** (`My Project v3`) and the incoming commit (`8b608e6`) represents the **branch being rebased** (`My Project v2`). Since the higher version (v3) is more appropriate, the main branch's version is kept.
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:low__json_schema FAIL 0% 118 in → 278 out (308 reasoning)
Title:My Project v3
JSON Schema Structured Output
(raw) { "resolved_content": "Title:My Project v3" }
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output: { { "resolved_content": "Title: My Project v3" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output: project.txt Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:low FAIL 0% 153 in → 462 out (509 reasoning)
My Project v3
Failure: Content mismatch: ['project.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
mistralai/devstral-2512 FAIL 0% 121 in → 29 out
Here's the resolved `project.txt` content with the higher version (v3) kept: ``` Title: My Project v3 ```
Failure: Content mismatch: ['project.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:high FAIL 0% 119 in → 391 out (446 reasoning)
Title: My Project v3
Failure: Content mismatch: ['project.txt']
moonshotai/kimi-k2.6:low FAIL 0% 119 in → 1,618 out (1,738 reasoning)
Title: My Project v3
Failure: Content mismatch: ['project.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:medium FAIL 0% 119 in → 2,197 out (2,425 reasoning)
Title: My Project v3
Failure: Content mismatch: ['project.txt']
moonshotai/kimi-k2.6:xhigh FAIL 0% 118 in → 976 out (1,109 reasoning)
Title: My Project v3
Failure: Content mismatch: ['project.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 134 in → 267 out (286 reasoning)
Title: My Project v38
JSON Schema Structured Output
(raw) { "resolved_content": "Title: My Project v38" }
Failure: Content mismatch: ['project.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
poolside/laguna-xs.2:none FAIL 0% 163 in → 55 out (0 reasoning)
I'll help you resolve this rebase conflict. Since the main branch has the higher version (v3) and that's more appropriate, I'll use that version. Here's the resolved project.txt content: ``` Title: My Project v3 ```
Failure: Content mismatch: ['project.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6344359e-abff-92d6-b2e4-a789d1f955a9","request_id":"6344359e-abff-92d6-b2e4-a789d1f955a9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e3846717-5c6b-9bb2-8caa-91f4989d3cf3","request_id":"e3846717-5c6b-9bb2-8caa-91f4989d3cf3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4b90b8f8-3b77-9f25-a00d-1f3cc1486ac8","request_id":"4b90b8f8-3b77-9f25-a00d-1f3cc1486ac8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1bf3f6e1-f0ba-93a0-ad0f-0fd45fc86b43","request_id":"1bf3f6e1-f0ba-93a0-ad0f-0fd45fc86b43"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9a1ae139-f61e-9035-a2d1-fec878ddf65e","request_id":"9a1ae139-f61e-9035-a2d1-fec878ddf65e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f935efe8-4481-90c9-b774-1548a154eb6c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6ca64e0a-b7ef-993a-b00d-f4209063edb6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-713f7137-e16d-9cf5-a999-fecccbf1a853"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-06407669-d2ca-9aa2-8b63-10d19d5b7b9b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8852154b-68e3-9779-89c9-3c1f813044b3"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-63f74b92-e9f0-9b48-ba97-29c8aac844e9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-db2a260f-f703-9cbe-a969-c89ab5be17cc","request_id":"db2a260f-f703-9cbe-a969-c89ab5be17cc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2897616c-2e5c-98f8-92bd-ef1876287673","request_id":"2897616c-2e5c-98f8-92bd-ef1876287673"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a5820d46-32ba-9736-aaa9-c6fe08b007dd","request_id":"a5820d46-32ba-9736-aaa9-c6fe08b007dd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-467504ba-dced-90db-9a90-097bdfe337ee","request_id":"467504ba-dced-90db-9a90-097bdfe337ee"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5f3356e2-1c44-98c6-8edc-f4df782f65c9","request_id":"5f3356e2-1c44-98c6-8edc-f4df782f65c9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4a7205c1-6837-9dc2-86d1-2a1c9b8ad5cb","request_id":"4a7205c1-6837-9dc2-86d1-2a1c9b8ad5cb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7037b4f5-a6ee-94d2-9790-f4e55c2d08b9","request_id":"7037b4f5-a6ee-94d2-9790-f4e55c2d08b9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-863f19fc-d714-997a-8661-4c4ba9153efb","request_id":"863f19fc-d714-997a-8661-4c4ba9153efb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Model 'tencent/hy3:none' violated the reasoning_level=none invariant: provider reported 567 reasoning token(s)
thinkingmachines/inkling-small:none FAIL 0% 123 in → 9 out (0 reasoning)
Title: My Project v3<|end_message|>
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output: Title: My Project v3{ "resolved_content": "Title: My Project v3" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 122 in → 9 out (0 reasoning)
Title: My Project v3<|end_message|>
Failure: Content mismatch: ['project.txt']
Invalid JSON. Output: Title: My Project v3{ "resolved_content": "Title: My Project v3" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Title: My Project v3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.