Multi-line hunk: address block conflict
Tests ability to resolve a multi-line hunk merge conflict with address/contact blocks. Evaluates combining partial updates from both sides.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 git config user.email 'test@test.com'
  5. 05 printf 'Name: John Doe Email: john@example.com Phone: 555-1234' > contact.txt
  6. 06 git add contact.txt
  7. 07 git commit -m 'Initial contact'
  8. 08 git checkout -b work
  9. 09 printf 'Name: John Doe Email: john@company.com Phone: 555-1234' > contact.txt
  10. 10 git add contact.txt
  11. 11 git commit -m 'Use work email'
  12. 12 git checkout main
  13. 13 printf 'Name: John Doe Email: johndoe@personal.com Phone: 555-9876' > contact.txt
  14. 14 git add contact.txt
  15. 15 git commit -m 'Update contact info'
  16. 16 git merge work
Prompt
Resolve the merge conflict in contact.txt. The base had john@example.com and 555-1234. Branch A kept john@example.com but updated the email to john@company.com. Branch B updated both to johndoe@personal.com and 555-9876. Provide ONLY the resolved file content with the most complete updates.
Expected
Name: John Doe Email: johndoe@personal.com Phone: 555-9876

Scoped model quality, cost, API time, and token usage for merge_conflicts/f004.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 197 in → 329 out (93 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
anthropic/claude-fable-5:high__json_schema PASS 100% 417 in → 324 out (112 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-fable-5:low__json_schema PASS 100% 417 in → 65 out (14 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-fable-5:max PASS 100% 197 in → 4,710 out (612 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 345 in → 1,679 out (800 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-haiku-4.5:low PASS 100% 177 in → 424 out (384 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 345 in → 341 out (288 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 345 in → 807 out (495 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-haiku-4.5:none PASS 100% 148 in → 30 out (0 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 315 in → 35 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 345 in → 617 out (526 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-opus-4.6:low__json_schema PASS 100% 316 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 316 in → 766 out (227 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 316 in → 37 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.6:none__json_schema PASS 100% 316 in → 35 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 316 in → 505 out (219 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-opus-4.7:low__json_schema PASS 100% 422 in → 50 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.7:max PASS 100% 202 in → 892 out (128 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
anthropic/claude-opus-4.7:max__json_schema PASS 100% 422 in → 1,130 out (204 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.7:none__json_schema PASS 100% 422 in → 50 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.8:high PASS 100% 197 in → 728 out (103 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 417 in → 561 out (79 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.8:low PASS 100% 197 in → 283 out (80 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-opus-4.8:max PASS 100% 197 in → 1,561 out (194 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
anthropic/claude-opus-4.8:max__json_schema PASS 100% 417 in → 2,723 out (279 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.8:medium PASS 100% 197 in → 487 out (110 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 417 in → 409 out (77 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 197 in → 850 out (193 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 417 in → 903 out (169 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 316 in → 323 out (161 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 148 in → 30 out (0 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 316 in → 36 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 316 in → 301 out (188 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 316 in → 37 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 148 in → 30 out (0 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 316 in → 37 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 316 in → 228 out (163 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-5:high PASS 100% 197 in → 222 out (66 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 417 in → 148 out (71 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-5:low PASS 100% 197 in → 134 out (69 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 417 in → 128 out (60 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-5:medium PASS 100% 197 in → 172 out (57 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 417 in → 151 out (42 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-5:none__json_schema PASS 100% 417 in → 47 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 197 in → 741 out (149 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 417 in → 396 out (212 reasoning)
--- contact.txt Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"--- contact.txt\nName: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 131 in → 1,384 out (1,375 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
arcee-ai/trinity-large-thinking:low PASS 100% 131 in → 1,457 out (1,454 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
arcee-ai/trinity-large-thinking:medium PASS 100% 131 in → 1,323 out (1,320 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
arcee-ai/trinity-mini:low__json_schema PASS 100% 131 in → 301 out (308 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 131 in → 331 out (340 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
arcee-ai/trinity-mini:xhigh PASS 100% 131 in → 405 out (425 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 131 in → 330 out (332 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 213 in → 15,862 out (15,839 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash-0731:low PASS 100% 134 in → 633 out (610 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash-0731:max PASS 100% 213 in → 1,335 out (1,309 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 213 in → 1,496 out (1,486 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 134 in → 851 out (827 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 280 in → 10,580 out (11,200 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
deepseek/deepseek-v4-flash-0731:none PASS 100% 134 in → 22 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 219 in → 326 out (307 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 134 in → 580 out (533 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash:high PASS 100% 134 in → 674 out (693 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash:low PASS 100% 134 in → 1,242 out (1,219 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash:medium PASS 100% 134 in → 506 out (482 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 134 in → 1,297 out (1,287 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash:none PASS 100% 134 in → 25 out (0 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 136 in → 32 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 231 in → 30 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 326 in → 266 out (235 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
deepseek/deepseek-v4-pro:low PASS 100% 134 in → 1,436 out (1,412 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 326 in → 513 out (479 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 136 in → 1,431 out (1,396 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 136 in → 33 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 213 in → 9,635 out (9,611 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 404 in → 2,311 out (2,277 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3-flash-preview:high PASS 100% 150 in → 856 out (830 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3-flash-preview:high__json_schema PASS 100% 151 in → 1,289 out (1,254 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3-flash-preview:low PASS 100% 150 in → 201 out (175 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3-flash-preview:low__json_schema PASS 100% 210 in → 983 out (949 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3-flash-preview:medium PASS 100% 150 in → 1,425 out (1,399 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3-flash-preview:medium__json_schema PASS 100% 151 in → 1,060 out (1,026 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3-flash-preview:xhigh PASS 100% 150 in → 2,104 out (2,078 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.1-flash-lite-preview:low PASS 100% 151 in → 180 out (154 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 210 in → 177 out (137 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.5-flash-lite:high PASS 100% 150 in → 1,451 out (1,425 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 210 in → 1,709 out (1,669 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.5-flash-lite:low PASS 100% 150 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 151 in → 40 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.5-flash-lite:max PASS 100% 151 in → 4,986 out (4,960 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 210 in → 1,417 out (1,383 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3.5-flash-lite:medium PASS 100% 150 in → 1,571 out (1,545 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 151 in → 1,039 out (999 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 150 in → 1,959 out (1,933 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 151 in → 1,511 out (1,477 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3.5-flash:high PASS 100% 150 in → 1,692 out (1,666 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash:high__json_schema PASS 100% 210 in → 1,110 out (1,073 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
google/gemini-3.5-flash:low PASS 100% 150 in → 799 out (773 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash:low__json_schema PASS 100% 210 in → 414 out (374 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.5-flash:medium PASS 100% 150 in → 1,177 out (1,151 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash:medium__json_schema PASS 100% 210 in → 842 out (802 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.5-flash:xhigh PASS 100% 150 in → 1,570 out (1,544 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 210 in → 1,336 out (1,301 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3.6-flash:high PASS 100% 150 in → 4,288 out (4,262 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.6-flash:high__json_schema PASS 100% 210 in → 3,409 out (3,374 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3.6-flash:low PASS 100% 151 in → 885 out (859 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.6-flash:low__json_schema PASS 100% 210 in → 40 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.6-flash:max PASS 100% 150 in → 3,384 out (3,358 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.6-flash:max__json_schema PASS 100% 151 in → 2,834 out (2,799 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3.6-flash:medium PASS 100% 151 in → 1,631 out (1,605 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.6-flash:medium__json_schema PASS 100% 210 in → 4,024 out (3,984 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.6-flash:xhigh PASS 100% 151 in → 5,255 out (5,229 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 210 in → 1,429 out (1,387 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
google/gemma-4-26b-a4b-it:high PASS 100% 166 in → 1,839 out (1,812 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 166 in → 1,495 out (1,205 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-26b-a4b-it:low PASS 100% 166 in → 1,762 out (1,553 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 165 in → 3,314 out (2,824 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 166 in → 1,352 out (1,078 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-26b-a4b-it:none PASS 100% 163 in → 27 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 163 in → 40 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 166 in → 1,168 out (994 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 166 in → 1,552 out (1,342 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-31b-it:high PASS 100% 166 in → 575 out (440 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-31b-it:high__json_schema PASS 100% 165 in → 428 out (302 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-31b-it:low PASS 100% 166 in → 407 out (346 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-31b-it:low__json_schema PASS 100% 165 in → 908 out (784 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-31b-it:medium PASS 100% 166 in → 245 out (185 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-31b-it:medium__json_schema PASS 100% 165 in → 659 out (505 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-31b-it:none PASS 100% 163 in → 27 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-31b-it:none__json_schema PASS 100% 163 in → 40 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-31b-it:xhigh PASS 100% 166 in → 614 out (1 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 165 in → 516 out (431 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
ibm-granite/granite-4.1-8b PASS 100% 144 in → 23 out
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
ibm-granite/granite-4.1-8b__json_schema PASS 100% 144 in → 33 out
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m2.5:high PASS 100% 161 in → 561 out (538 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.5:high__json_schema PASS 100% 161 in → 663 out (685 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m2.5:medium PASS 100% 161 in → 1,291 out (1,292 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.5:medium__json_schema PASS 100% 176 in → 1,194 out (1,238 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m2.5:xhigh PASS 100% 163 in → 465 out (443 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 176 in → 668 out (637 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m2.7:high PASS 100% 161 in → 997 out (1,083 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.7:low PASS 100% 161 in → 811 out (787 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.7:low__json_schema PASS 100% 294 in → 738 out (709 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
minimax/minimax-m2.7:medium PASS 100% 163 in → 1,197 out (1,175 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.7:medium__json_schema PASS 100% 294 in → 653 out (625 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
minimax/minimax-m2.7:xhigh PASS 100% 161 in → 629 out (637 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m3:high PASS 100% 299 in → 664 out (723 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m3:high__json_schema PASS 100% 286 in → 31 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m3:low PASS 100% 299 in → 439 out (423 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m3:low__json_schema PASS 100% 286 in → 32 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
minimax/minimax-m3:medium PASS 100% 296 in → 652 out (680 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
minimax/minimax-m3:medium__json_schema PASS 100% 286 in → 31 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m3:xhigh__json_schema PASS 100% 286 in → 31 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
mistralai/devstral-2512 PASS 100% 150 in → 26 out
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
mistralai/devstral-2512__json_schema PASS 100% 150 in → 33 out
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
mistralai/mistral-medium-3-5:high PASS 100% 162 in → 4,838 out (4,870 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 150 in → 824 out (697 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
mistralai/mistral-medium-3-5:low PASS 100% 162 in → 6,487 out (6,776 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 150 in → 6,245 out (6,326 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
mistralai/mistral-medium-3-5:medium PASS 100% 162 in → 2,538 out (2,459 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 150 in → 2,832 out (2,864 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
mistralai/mistral-medium-3-5:none PASS 100% 162 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 150 in → 33 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 162 in → 1,817 out (1,749 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 150 in → 1,748 out (1,632 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 134 in → 4,399 out (4,368 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
moonshotai/kimi-k2.7-code:low PASS 100% 134 in → 2,238 out (2,214 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 134 in → 1,900 out (1,868 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
moonshotai/kimi-k2.7-code:medium PASS 100% 133 in → 2,616 out (2,592 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 218 in → 1,227 out (1,197 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
moonshotai/kimi-k3:high PASS 100% 211 in → 680 out (643 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:high__json_schema PASS 100% 211 in → 644 out (607 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:low PASS 100% 211 in → 948 out (911 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:low__json_schema PASS 100% 211 in → 590 out (553 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:max PASS 100% 211 in → 442 out (405 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:max__json_schema PASS 100% 308 in → 1,020 out (977 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
moonshotai/kimi-k3:medium PASS 100% 211 in → 393 out (356 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:medium__json_schema PASS 100% 211 in → 467 out (430 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:xhigh PASS 100% 211 in → 497 out (460 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 211 in → 447 out (410 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 163 in → 348 out (292 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 163 in → 464 out (444 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 163 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 163 in → 972 out (901 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 163 in → 378 out (340 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 163 in → 878 out (770 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 163 in → 324 out (274 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 163 in → 405 out (328 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 163 in → 761 out (644 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 163 in → 822 out (745 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 163 in → 40 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 163 in → 428 out (387 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-mini:high PASS 100% 126 in → 243 out (216 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-mini:high__json_schema PASS 100% 162 in → 419 out (383 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4-mini:low PASS 100% 126 in → 65 out (38 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-mini:low__json_schema PASS 100% 162 in → 85 out (49 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4-mini:medium PASS 100% 126 in → 174 out (147 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-mini:medium__json_schema PASS 100% 162 in → 248 out (212 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4-mini:none PASS 100% 126 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-mini:none__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4-mini:xhigh PASS 100% 126 in → 536 out (509 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 162 in → 552 out (516 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4-nano:high__json_schema PASS 100% 162 in → 224 out (187 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
openai/gpt-5.4-nano:low__json_schema PASS 100% 162 in → 95 out (59 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4-nano:medium PASS 100% 126 in → 402 out (375 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-nano:medium__json_schema PASS 100% 162 in → 235 out (198 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
openai/gpt-5.4-nano:none PASS 100% 126 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-nano:xhigh PASS 100% 126 in → 666 out (639 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 162 in → 428 out (392 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4:high PASS 100% 126 in → 197 out (170 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4:high__json_schema PASS 100% 162 in → 356 out (320 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4:low PASS 100% 126 in → 119 out (92 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4:low__json_schema PASS 100% 162 in → 115 out (79 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4:medium PASS 100% 126 in → 231 out (204 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4:medium__json_schema PASS 100% 162 in → 274 out (238 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4:none PASS 100% 126 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4:none__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4:xhigh PASS 100% 126 in → 4,623 out (4,596 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4:xhigh__json_schema PASS 100% 162 in → 1,452 out (1,416 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.5:high PASS 100% 126 in → 490 out (463 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.5:high__json_schema PASS 100% 162 in → 513 out (477 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.5:low PASS 100% 126 in → 71 out (44 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.5:low__json_schema PASS 100% 162 in → 90 out (54 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.5:medium PASS 100% 126 in → 168 out (141 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.5:medium__json_schema PASS 100% 162 in → 141 out (105 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.5:none PASS 100% 126 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.5:none__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.5:xhigh PASS 100% 126 in → 266 out (239 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.5:xhigh__json_schema PASS 100% 162 in → 448 out (412 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:high PASS 100% 126 in → 93 out (66 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-luna:high__json_schema PASS 100% 162 in → 121 out (84 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
openai/gpt-5.6-luna:low PASS 100% 126 in → 72 out (45 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-luna:low__json_schema PASS 100% 162 in → 118 out (82 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:max PASS 100% 126 in → 176 out (149 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-luna:max__json_schema PASS 100% 162 in → 182 out (146 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:medium PASS 100% 126 in → 72 out (45 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-luna:medium__json_schema PASS 100% 162 in → 123 out (87 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:none PASS 100% 126 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-luna:none__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:xhigh PASS 100% 126 in → 138 out (111 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 162 in → 158 out (122 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-sol:high PASS 100% 126 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-sol:high__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-sol:low PASS 100% 126 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-sol:low__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-sol:max PASS 100% 126 in → 186 out (159 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-sol:max__json_schema PASS 100% 162 in → 206 out (170 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-sol:medium PASS 100% 126 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-sol:medium__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-sol:none PASS 100% 126 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-sol:none__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-sol:xhigh PASS 100% 126 in → 111 out (84 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 162 in → 249 out (213 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-terra:high PASS 100% 126 in → 120 out (93 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-terra:high__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-terra:low PASS 100% 126 in → 89 out (62 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-terra:low__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-terra:max PASS 100% 126 in → 415 out (388 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-terra:max__json_schema PASS 100% 162 in → 497 out (461 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-terra:medium PASS 100% 126 in → 54 out (27 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-terra:none PASS 100% 126 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-terra:xhigh PASS 100% 126 in → 79 out (52 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 162 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-oss-120b:high PASS 100% 185 in → 1,424 out (1,576 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-oss-120b:high__json_schema PASS 100% 187 in → 1,708 out (1,668 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
openai/gpt-oss-120b:medium PASS 100% 187 in → 240 out (219 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-oss-120b:xhigh PASS 100% 174 in → 2,307 out (2,564 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 189 in → 443 out (451 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
openai/gpt-oss-20b:high PASS 100% 187 in → 2,976 out (2,991 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-oss-20b:low PASS 100% 187 in → 289 out (292 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-oss-20b:low__json_schema PASS 100% 174 in → 168 out (152 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-oss-20b:medium PASS 100% 186 in → 654 out (687 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-oss-20b:medium__json_schema PASS 100% 187 in → 528 out (502 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-oss-20b:xhigh PASS 100% 187 in → 1,737 out (1,900 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-m.1:low__json_schema PASS 100% 151 in → 1,028 out (992 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
poolside/laguna-m.1:none__json_schema PASS 100% 151 in → 37 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
poolside/laguna-xs-2.1:high PASS 100% 151 in → 734 out (709 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs-2.1:low PASS 100% 151 in → 422 out (397 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs-2.1:medium PASS 100% 151 in → 957 out (932 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 151 in → 867 out (830 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
poolside/laguna-xs-2.1:none__json_schema PASS 100% 151 in → 31 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
poolside/laguna-xs-2.1:xhigh PASS 100% 151 in → 1,137 out (1,113 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 151 in → 1,076 out (1,040 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
poolside/laguna-xs.2:high PASS 100% 188 in → 476 out (449 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs.2:high__json_schema PASS 100% 188 in → 1,045 out (1,012 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
poolside/laguna-xs.2:low PASS 100% 188 in → 599 out (572 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs.2:low__json_schema PASS 100% 188 in → 1,339 out (1,307 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
poolside/laguna-xs.2:medium PASS 100% 188 in → 375 out (344 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
poolside/laguna-xs.2:medium__json_schema PASS 100% 188 in → 408 out (375 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
poolside/laguna-xs.2:none PASS 100% 188 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs.2:none__json_schema PASS 100% 188 in → 31 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 188 in → 623 out (591 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
qwen/qwen3.6-27b:low__json_schema PASS 100% 148 in → 8,255 out (1 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-27b:none PASS 100% 150 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-27b:none__json_schema PASS 100% 150 in → 33 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
qwen/qwen3.6-27b:xhigh PASS 100% 148 in → 1,783 out (1,708 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 148 in → 2,240 out (2,146 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-35b-a3b:low PASS 100% 148 in → 3,314 out (3,284 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 148 in → 2,353 out (2,225 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 148 in → 2,458 out (2,455 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 148 in → 1,903 out (1,766 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-35b-a3b:none PASS 100% 150 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 150 in → 37 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 148 in → 1,622 out (1,592 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 148 in → 2,260 out (2,172 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-flash:high PASS 100% 148 in → 1,348 out (1,318 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-flash:medium PASS 100% 148 in → 2,293 out (2,263 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-flash:none PASS 100% 150 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-flash:xhigh PASS 100% 148 in → 2,330 out (2,300 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-flash:high PASS 100% 148 in → 1,557 out (1,527 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-flash:low PASS 100% 148 in → 2,924 out (2,894 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-flash:medium PASS 100% 148 in → 1,851 out (1,821 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-flash:none PASS 100% 150 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-max:high PASS 100% 148 in → 1,908 out (1,878 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-max:low PASS 100% 148 in → 338 out (308 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-max:medium PASS 100% 148 in → 607 out (577 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-max:none PASS 100% 150 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-max:xhigh PASS 100% 148 in → 1,242 out (1,212 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-plus:high PASS 100% 148 in → 315 out (285 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-plus:low PASS 100% 148 in → 576 out (542 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
qwen/qwen3.7-plus:medium PASS 100% 148 in → 826 out (796 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-plus:none PASS 100% 150 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-plus:xhigh PASS 100% 148 in → 389 out (359 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
tencent/hy3:high PASS 100% 138 in → 6,211 out (6,187 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
tencent/hy3:high__json_schema PASS 100% 138 in → 2,956 out (2,932 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
tencent/hy3:low PASS 100% 138 in → 1,169 out (1,145 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
tencent/hy3:low__json_schema PASS 100% 138 in → 3,903 out (3,793 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
tencent/hy3:medium PASS 100% 138 in → 3,792 out (3,763 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
tencent/hy3:medium__json_schema PASS 100% 138 in → 2,735 out (2,667 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
tencent/hy3:none__json_schema PASS 100% 141 in → 32 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
tencent/hy3:xhigh PASS 100% 138 in → 4,597 out (4,573 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
tencent/hy3:xhigh__json_schema PASS 100% 138 in → 1,590 out (1,545 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling-small:high PASS 100% 135 in → 692 out (665 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:high__json_schema PASS 100% 135 in → 531 out (504 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:low PASS 100% 135 in → 73 out (52 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:low__json_schema PASS 100% 135 in → 83 out (47 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
thinkingmachines/inkling-small:max PASS 100% 135 in → 704 out (677 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:max__json_schema PASS 100% 135 in → 964 out (930 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling-small:medium PASS 100% 135 in → 596 out (583 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:medium__json_schema PASS 100% 135 in → 276 out (241 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling-small:xhigh PASS 100% 135 in → 324 out (304 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 135 in → 328 out (293 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling:high PASS 100% 135 in → 11,458 out (12,590 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling:high__json_schema PASS 100% 135 in → 6,025 out (5,987 reasoning)
--- contact.txt Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "--- contact.txt\nName: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling:low PASS 100% 135 in → 242 out (214 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling:low__json_schema PASS 100% 135 in → 196 out (161 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling:max PASS 100% 135 in → 19,162 out (19,142 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling:xhigh PASS 100% 135 in → 4,092 out (4,481 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling:xhigh__json_schema PASS 100% 135 in → 6,497 out (6,462 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
x-ai/grok-4.3:high__json_schema PASS 100% 362 in → 1,811 out (1,781 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
x-ai/grok-4.3:low__json_schema PASS 100% 368 in → 818 out (787 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
x-ai/grok-4.3:xhigh PASS 100% 305 in → 3,221 out (3,200 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
x-ai/grok-4.5:high PASS 100% 329 in → 494 out (473 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
x-ai/grok-4.5:high__json_schema PASS 100% 398 in → 1,351 out (1,318 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
x-ai/grok-4.5:max PASS 100% 329 in → 3,250 out (3,229 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
x-ai/grok-4.5:medium__json_schema PASS 100% 398 in → 1,224 out (1,191 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
x-ai/grok-4.5:xhigh__json_schema PASS 100% 398 in → 1,365 out (1,335 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
z-ai/glm-4.7-flash:low PASS 100% 131 in → 4,253 out (4,370 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-4.7-flash:none PASS 100% 131 in → 24 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-4.7-flash:none__json_schema PASS 100% 131 in → 33 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-4.7-flash:xhigh PASS 100% 131 in → 3,607 out (3,781 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-5.2:high PASS 100% 138 in → 625 out (594 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-5.2:high__json_schema PASS 100% 138 in → 555 out (525 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-5.2:low PASS 100% 138 in → 855 out (822 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-5.2:low__json_schema PASS 100% 138 in → 853 out (821 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-5.2:medium PASS 100% 138 in → 187 out (162 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-5.2:medium__json_schema PASS 100% 138 in → 1,297 out (1,242 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
z-ai/glm-5.2:none__json_schema PASS 100% 134 in → 32 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-5.2:xhigh PASS 100% 138 in → 894 out (834 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-5.2:xhigh__json_schema PASS 100% 131 in → 7,966 out (7,942 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
anthropic/claude-fable-5:low FAIL 0% 197 in → 213 out (72 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` **Rationale:** Branch A's change was specifically the email update (john@company.com), while Branch B's unique meaningful update was the phone number (555-9876). Merging both branches' updates gives the most complete result.
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
anthropic/claude-fable-5:medium FAIL 0% 197 in → 361 out (134 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` **Rationale:** Branch A's only change was updating the email (to john@company.com), while Branch B updated both fields. Merging both branches' intentional changes: Branch A's email update takes the email field (it was A's deliberate change), and Branch B's phone update (555-9876) is preserved since Branch A left the phone at the base value. This captures every update made across both branches.
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
anthropic/claude-fable-5:xhigh FAIL 0% 197 in → 654 out (163 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
anthropic/claude-haiku-4.5:high FAIL 0% 177 in → 442 out (348 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Resolution rationale:** Branch A (HEAD) contains the most complete updates with changes to both fields, compared to Branch B which only updated the email while retaining the original phone number.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-haiku-4.5:medium FAIL 0% 177 in → 788 out (509 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` This represents the most complete updates, as Branch B modified both fields (email and phone) compared to the base version, while the other branch only modified the email.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-haiku-4.5:xhigh FAIL 0% 177 in → 1,077 out (771 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` This resolution takes the updates from Branch B (HEAD), which represents the most complete set of changes since both the email and phone fields were updated, compared to Branch A which only updated the email.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:high FAIL 0% 148 in → 1,033 out (315 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` **Reasoning:** | Field | Base | Branch A (work) | Branch B (HEAD) | Resolution | |-------|------|-----------------|-----------------|------------| | Email | john@example.com | john@company.com ✏️ | johndoe@personal.com ✏️ | **john@company.com** | | Phone | 555-1234 | 555-1234 (unchanged) | 555-9876 ✏️ | **555-9876** | - **Phone**: Only Branch B changed it → take `555-9876` (no ambiguity). - **Email**: Both branches changed it, but Branch A's **sole intentional change** was the email update to `john@company.com`. Discarding it would erase that branch's entire purpose. This resolution combines **every deliberate change** from both branches, producing the most complete result.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 316 in → 896 out (363 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:low FAIL 0% 148 in → 24 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:max FAIL 0% 148 in → 1,002 out (324 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Reasoning:** | Field | Base | Branch A (work) | Branch B (HEAD) | Resolution | |-------|------|-----------------|-----------------|------------| | Email | john@example.com | john@company.com *(changed)* | johndoe@personal.com *(changed)* | johndoe@personal.com | | Phone | 555-1234 | 555-1234 *(unchanged)* | 555-9876 *(changed)* | 555-9876 | - **Phone:** Only Branch B changed it → take `555-9876` (clear choice). - **Email:** Both branches changed it (true conflict). Branch B's update is chosen because it is part of a **coordinated, more complete update** to the contact info (both fields updated together), making it the most intentional and current revision.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:medium FAIL 0% 148 in → 29 out (0 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ```
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:none FAIL 0% 148 in → 24 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:xhigh FAIL 0% 148 in → 1,013 out (293 reasoning)
``` Name: John Doe Email: johndoe@personal.com Email: john@company.com Phone: 555-9876 ``` **Reasoning:** | Field | Base | Branch A (`work`) | Branch B (`HEAD`) | Resolution | |-------|------|-------|-------|------------| | Email | john@example.com | john@**company**.com ✏️ | johndoe@**personal**.com ✏️ | **Both** (each branch added a distinct, intentional update) | | Phone | 555-1234 | 555-1234 (unchanged) | 555-**9876** ✏️ | **555-9876** (only Branch B changed it) | - **Phone** was straightforward: only Branch B modified it, so `555-9876` wins. - **Email** was changed by *both* branches to two **different, meaningful addresses** (a work email and a personal email). Keeping both provides the **most complete** contact information rather than discarding a valid update from either branch.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:high FAIL 0% 202 in → 30 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:high__json_schema FAIL 0% 422 in → 46 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:low FAIL 0% 202 in → 30 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:medium FAIL 0% 202 in → 30 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:medium__json_schema FAIL 0% 422 in → 46 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:none FAIL 0% 202 in → 30 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:xhigh FAIL 0% 202 in → 30 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:xhigh__json_schema FAIL 0% 422 in → 46 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:low__json_schema FAIL 0% 417 in → 44 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:none FAIL 0% 197 in → 363 out (0 reasoning)
Here's the resolved file: ``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` **Reasoning for each field:** - **Email:** Branch A specifically updated the email to `john@company.com` (a deliberate change from the base `john@example.com`). Branch B set it to `johndoe@personal.com`. Since Branch A made an intentional, specific update to the company email, that's the meaningful change to preserve. - **Phone:** Branch A kept the base value (`555-1234`, unchanged). Branch B updated it to `555-9876`. The updated value is the more complete change. This combines the intentional updates from both branches (company email from A, new phone from B). **Note:** There's some ambiguity here worth flagging—"most complete updates" could be interpreted as favoring Branch B entirely (`johndoe@personal.com` / `555-9876`) since it updated *both* fields. If the intent was simply "take whichever branch changed the most fields," the answer would be Branch B's version verbatim. Let me know which interpretation you meant and I can adjust.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:none__json_schema FAIL 0% 417 in → 43 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-4.6:high FAIL 0% 148 in → 405 out (179 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Reasoning:** Branch B (HEAD) made the most complete updates by changing **both** the email and phone fields, while Branch A (work) only updated the email. Since the task asks for the most complete updates, HEAD's version is chosen in full.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-4.6:max FAIL 0% 148 in → 779 out (276 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Resolution rationale:** | Field | Base | Branch A (work) | Branch B (HEAD) | Chosen | |-------|------|-----------------|-----------------|--------| | Email | john@example.com | john@company.com | johndoe@personal.com | johndoe@personal.com | | Phone | 555-1234 | 555-1234 *(unchanged)* | 555-9876 | 555-9876 | - **Branch B (HEAD)** made changes to **both** fields → most complete update - **Branch A (work)** only updated the email, leaving the phone unchanged from base - Taking Branch B in full gives the most comprehensive set of updates, as it touches every field that either branch modified
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-4.6:medium FAIL 0% 148 in → 297 out (148 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Resolution rationale:** - **Branch B (HEAD)** made the most complete updates, changing **both** the email (`johndoe@personal.com`) and phone (`555-9876`) from the base values. - **Branch A (work)** only updated the email (`john@company.com`) while leaving the phone unchanged at `555-1234`. - Since Branch B represents the broadest/most complete set of changes and the conflict cannot be automatically merged (both branches modified the same `Email` field with different values), Branch B's version is selected as the most fully updated contact record.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 148 in → 465 out (179 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Resolution rationale:** - **HEAD (Branch B)** updated *both* fields — email (`johndoe@personal.com`) and phone (`555-9876`), representing the most complete set of changes. - **work (Branch A)** only updated the email (`john@company.com`), leaving the phone at the base value (`555-1234`). - Since the prompt specifies keeping the **most complete updates**, Branch B's version is taken in full, as it modified more fields intentionally.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-5:none FAIL 0% 197 in → 193 out (0 reasoning)
Looking at this conflict, I need to determine the most complete set of updates: - **Branch A (work)**: Updated email to `john@company.com`, but phone remained unchanged at `555-1234` - **Branch B (HEAD)**: Updated both email to `johndoe@personal.com` AND phone to `555-9876` Branch B represents the more complete update since it changed both fields, while Branch A only updated the email and left the phone at its original base value. Resolved file content: ``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh FAIL 0% 129 in → 0 out (1,227 reasoning)
(empty output)
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 131 in → 349 out (350 reasoning)
Name: John DoeEmail: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
arcee-ai/trinity-mini:high__json_schema FAIL 0% 131 in → 322 out (318 reasoning)
Name: JohnDoe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: JohnDoe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
arcee-ai/trinity-mini:low FAIL 0% 131 in → 256 out (279 reasoning)
johndoe@personal.com 555-9876
Failure: Content mismatch: ['contact.txt']
arcee-ai/trinity-mini:medium FAIL 0% 131 in → 303 out (298 reasoning)
johndoe@personal.com 555-9876
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-flash-0731:high__json_schema FAIL 0% 134 in → 3,246 out (3,397 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 0% 140 in → 693 out (664 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 134 in → 31 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output: Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:low__json_schema FAIL 0% 136 in → 691 out (649 reasoning)
Name: John Doe Email: john@company.com, johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com, johndoe@personal.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-flash:xhigh FAIL 0% 213 in → 1,675 out (1,819 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-pro:high FAIL 0% 134 in → 1,375 out (1,347 reasoning)
Name: John Doe Email: john@company.com, johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-pro:medium FAIL 0% 134 in → 1,516 out (1,487 reasoning)
Name: John Doe Email: john@company.com, johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-pro:none FAIL 0% 134 in → 90 out (0 reasoning)
Here's the resolved file content with the most complete updates: ``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` The resolution takes Branch B's email (johndoe@personal.com) and phone (555-9876) since Branch B updated both fields, while Branch A only updated the email to john@company.com and left the phone unchanged from the base.
Failure: Content mismatch: ['contact.txt']
google/gemini-3-flash-preview:xhigh__json_schema FAIL 0% 151 in → 10,936 out (10,893 reasoning)
Name: John Doe Email: john@company.com Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nEmail: johndoe@personal.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-flash-lite-preview:high FAIL 0% 151 in → 2,201 out (2,177 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-flash-lite-preview:high__json_schema FAIL 0% 210 in → 2,961 out (2,928 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-flash-lite-preview:medium FAIL 0% 151 in → 719 out (695 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 151 in → 944 out (906 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-flash-lite-preview:xhigh FAIL 0% 150 in → 1,761 out (1,737 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-flash-lite-preview:xhigh__json_schema FAIL 0% 151 in → 3,333 out (3,301 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:high FAIL 0% 150 in → 2,623 out (2,591 reasoning)
Name: John Doe Email: john@company.com, johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:high__json_schema FAIL 0% 210 in → 4,266 out (4,226 reasoning)
Name: John Doe Email: john@company.com, johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com, johndoe@personal.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:low FAIL 0% 150 in → 1,096 out (1,072 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 210 in → 466 out (426 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:medium FAIL 0% 150 in → 1,422 out (1,398 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:medium__json_schema FAIL 0% 210 in → 1,586 out (1,545 reasoning)
Name: John Doe\nEmail: johndoe@personal.com, john@company.com\nPhone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\\nEmail: johndoe@personal.com, john@company.com\\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:xhigh FAIL 0% 150 in → 3,897 out (3,873 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:xhigh__json_schema FAIL 0% 210 in → 3,285 out (3,245 reasoning)
Name: John Doe Email: john@company.com, johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com, johndoe@personal.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
google/gemma-4-26b-a4b-it:medium FAIL 0% 166 in → 8,252 out (7,032 reasoning)
Name: John Doe Email: john@company.com, johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
inclusionai/ling-2.6-flash FAIL 0% 162 in → 24 out
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content" : "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
liquid/lfm-2-24b-a2b FAIL 0% 147 in → 131 out
Here’s a **resolved** version of `contact.txt` that incorporates the most complete updates from both branches (choosing the latest values): ```txt Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Explanation:** - Branch A changed the email to `john@company.com`, but Branch B overrode it with `johndoe@personal.com`. - Branch B also changed the phone number to `555-9876`, which is newer than `555-1234`. - The final resolved file keeps these updated values.
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output: Here’s a **resolved** version of `contact.txt` that incorporates the most complete updates from both branches (choosing Branch B's changes for both Email and Phone, since it contains more recent info): ```txt Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Explanation:** - Both branches changed the email and phone number. - Branch B’s updates are kept as they are more comprehensive. - The final result reflects the latest state of the branch you merged from (`work`).
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:low FAIL 0% 161 in → 2,438 out (2,643 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
minimax/minimax-m2.5:low__json_schema FAIL 0% 161 in → 1,319 out (1,415 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:xhigh FAIL 0% 299 in → 367 out (360 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.6:high FAIL 0% 134 in → 5,527 out (6,105 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.6:high__json_schema FAIL 0% 134 in → 4,775 out (5,436 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.6:low FAIL 0% 134 in → 5,506 out (5,978 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.6:low__json_schema FAIL 0% 133 in → 6,926 out (6,826 reasoning)
Name: John Doe Email: john@company.com Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nEmail: johndoe@personal.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.6:medium FAIL 0% 134 in → 5,953 out (5,932 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.6:medium__json_schema FAIL 0% 133 in → 7,514 out (7,329 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.6:xhigh FAIL 0% 134 in → 6,043 out (6,012 reasoning)
Name: John Doe Email: john@company.com Email: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.7-code:high FAIL 0% 134 in → 623 out (599 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.7-code:high__json_schema FAIL 0% 134 in → 447 out (416 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.7-code:medium__json_schema FAIL 0% 218 in → 867 out (840 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.7-code:xhigh FAIL 0% 134 in → 3,462 out (3,441 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 0% 163 in → 585 out (511 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-nano-30b-a3b:low FAIL 0% 163 in → 1,306 out (1,203 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 163 in → 1,065 out (1,006 reasoning)
Name: John Doe\nEmail: john@company.com\nPhone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\\nEmail: john@company.com\\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema FAIL 0% 163 in → 1,318 out (1,212 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 163 in → 35 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-1234
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-1234" }
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 163 in → 916 out (845 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-super-120b-a12b:none FAIL 0% 163 in → 21 out (0 reasoning)
Email: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema FAIL 0% 163 in → 479 out (393 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.4-nano:high FAIL 0% 126 in → 352 out (327 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.4-nano:low FAIL 0% 126 in → 23 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.4-nano:none__json_schema FAIL 0% 162 in → 32 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-1234
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-1234"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-terra:medium__json_schema FAIL 0% 162 in → 32 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-terra:none__json_schema FAIL 0% 162 in → 32 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-oss-120b:low FAIL 0% 174 in → 89 out (67 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-oss-120b:low__json_schema FAIL 0% 189 in → 282 out (283 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
openai/gpt-oss-120b:medium__json_schema FAIL 0% 185 in → 860 out (926 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
openai/gpt-oss-20b:high__json_schema FAIL 0% 187 in → 26 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-oss-20b:xhigh__json_schema FAIL 0% 187 in → 2,129 out (2,319 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:high FAIL 0% 151 in → 2,492 out (2,446 reasoning)
Name: John Doe Email: john@company.com Email: johndoe@personal.com Phone: 555-1234 Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:high__json_schema FAIL 0% 151 in → 704 out (668 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:low FAIL 0% 151 in → 3,122 out (3,097 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:medium FAIL 0% 151 in → 1,972 out (1,946 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:medium__json_schema FAIL 0% 151 in → 1,539 out (1,502 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:none FAIL 0% 151 in → 25 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:xhigh FAIL 0% 151 in → 1,283 out (1,257 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:xhigh__json_schema FAIL 0% 151 in → 1,993 out (1,958 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
poolside/laguna-xs-2.1:high__json_schema FAIL 0% 151 in → 1,325 out (1,290 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
poolside/laguna-xs-2.1:low__json_schema FAIL 0% 151 in → 2,754 out (2,719 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
poolside/laguna-xs-2.1:none FAIL 0% 151 in → 48 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876 </think>Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
poolside/laguna-xs.2:xhigh FAIL 0% 188 in → 531 out (505 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-27b:high FAIL 0% 148 in → 2,917 out (2,693 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-27b:high__json_schema FAIL 0% 148 in → 2,486 out (2,407 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-27b:low FAIL 0% 148 in → 8,368 out (7,760 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-27b:medium FAIL 0% 148 in → 2,379 out (1 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output: Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:xhigh__json_schema FAIL 0% 148 in → 14,712 out (13,316 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-35b-a3b:high FAIL 0% 148 in → 1,876 out (1,848 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9f5a7cf5-b6c6-9d12-8c3b-c8083f0a4ac1","request_id":"9f5a7cf5-b6c6-9d12-8c3b-c8083f0a4ac1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low FAIL 0% 148 in → 2,517 out (2,489 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6ffcd08c-7388-959f-a404-ae08ee91f7f4","request_id":"6ffcd08c-7388-959f-a404-ae08ee91f7f4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f414cf96-5af9-99e7-baf9-2b8c8b2c9e2c","request_id":"f414cf96-5af9-99e7-baf9-2b8c8b2c9e2c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2b967684-7c71-92a6-b0d4-4f916aabfa6b","request_id":"2b967684-7c71-92a6-b0d4-4f916aabfa6b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dc55c7f3-9012-9f71-83f8-b0b952d3fa8b","request_id":"dc55c7f3-9012-9f71-83f8-b0b952d3fa8b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7af65984-31dd-9529-ba4f-2ade1c2dd62b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e093bc91-d52f-98cd-b78e-b22e52e07010"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max FAIL 0% 148 in → 2,530 out (2,502 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5e5b6d4b-8f49-953f-9d4d-461762622de1"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d79418b1-2d43-9ba2-a275-5b99e4b23187"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c71efe73-29fe-92c7-8810-1e9cb450a39e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh FAIL 0% 148 in → 2,250 out (2,222 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d5f83b6d-291e-902a-be2d-d6d16cfea9ed"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-37fdad6f-c385-9250-9db8-8483bf71dc0c","request_id":"37fdad6f-c385-9250-9db8-8483bf71dc0c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-844f7d32-c2a7-9c80-b8e2-2c3aab98e47b","request_id":"844f7d32-c2a7-9c80-b8e2-2c3aab98e47b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-97f8ed82-41d3-91a9-839c-4692e24a3a17","request_id":"97f8ed82-41d3-91a9-839c-4692e24a3a17"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cc09445b-e55c-9570-8b1e-9d8edd16292b","request_id":"cc09445b-e55c-9570-8b1e-9d8edd16292b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5ce8869b-b48b-92e3-9e30-8b7fe13f4736","request_id":"5ce8869b-b48b-92e3-9e30-8b7fe13f4736"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2b5e7179-1944-9f55-b16a-adc8398a70d4","request_id":"2b5e7179-1944-9f55-b16a-adc8398a70d4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b9d2d1d3-e293-9c01-9f5f-7ed38214763d","request_id":"b9d2d1d3-e293-9c01-9f5f-7ed38214763d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7972d595-b67f-9a98-b34e-21c1cbec7697","request_id":"7972d595-b67f-9a98-b34e-21c1cbec7697"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fb6a253f-63c6-9442-93c5-df2d44d20dd1","request_id":"fb6a253f-63c6-9442-93c5-df2d44d20dd1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e021f7d8-da6e-92e4-9284-1172c7350abe","request_id":"e021f7d8-da6e-92e4-9284-1172c7350abe"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:none FAIL 0% 141 in → 20 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
thinkingmachines/inkling-small:none FAIL 0% 135 in → 24 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876<|end_message|>
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output: Name: John Doe Email: johndoe@personal.com Phone: 555-9876{ "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium FAIL 0% 135 in → 1,070 out (1,045 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output: Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 133 in → 23 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
thinkingmachines/inkling:none__json_schema FAIL 0% 133 in → 28 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content" : "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.3:high FAIL 0% 305 in → 1,656 out (1,637 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.3:low FAIL 0% 311 in → 785 out (766 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.3:max FAIL 0% 305 in → 1,636 out (1,617 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.3:max__json_schema FAIL 0% 362 in → 1,977 out (1,953 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.3:medium FAIL 0% 311 in → 1,685 out (1,666 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.3:medium__json_schema FAIL 0% 368 in → 1,245 out (1,217 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.3:none FAIL 0% 303 in → 22 out (0 reasoning)
```Name: John Doe Email: john@company.com Phone: 555-9876 ```
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.3:none__json_schema FAIL 0% 360 in → 24 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.3:xhigh__json_schema FAIL 0% 362 in → 2,076 out (2,048 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:low FAIL 0% 329 in → 1,174 out (1,155 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:low__json_schema FAIL 0% 398 in → 881 out (850 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:max__json_schema FAIL 0% 398 in → 662 out (634 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:medium FAIL 0% 329 in → 2,913 out (2,894 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:xhigh FAIL 0% 329 in → 990 out (971 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
Failure: Expecting value: line 171 column 1 (char 935)
z-ai/glm-4.7-flash:high__json_schema FAIL 0% 131 in → 708 out (649 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
z-ai/glm-4.7-flash:low__json_schema FAIL 0% 131 in → 5,002 out (4,640 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
z-ai/glm-4.7-flash:medium FAIL 0% 131 in → 9,179 out (9,156 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:none FAIL 0% 132 in → 22 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']