Multi-line address block conflict
Tests ability to resolve a multi-line address block conflict during rebase. Evaluates combining partial updates with correct polarity.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'Name: John Doe Email: john@example.com Phone: 555-1234' > contact.txt
  5. 05 git add contact.txt
  6. 06 git commit -m 'Initial contact'
  7. 07 git checkout -b work
  8. 08 printf 'Name: John Doe Email: john@company.com Phone: 555-1234' > contact.txt
  9. 09 git add contact.txt
  10. 10 git commit -m 'Use work email'
  11. 11 git checkout main
  12. 12 printf 'Name: John Doe Email: johndoe@personal.com Phone: 555-9876' > contact.txt
  13. 13 git add contact.txt
  14. 14 git commit -m 'Update contact info'
  15. 15 git rebase work
Prompt
Resolve the rebase conflict in contact.txt. The base had john@example.com and 555-1234. Branch being rebased kept john@example.com but updated the email to john@company.com. Main branch updated both to johndoe@personal.com and 555-9876. Provide ONLY the resolved file content with the most complete updates.
Expected
Name: John Doe Email: johndoe@personal.com Phone: 555-9876

Scoped model quality, cost, API time, and token usage for rebase/f004.

Loading...
Loading raw attempt evidence…
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 359 in → 348 out (298 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 356 in → 407 out (350 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-haiku-4.5:medium PASS 100% 189 in → 365 out (294 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 357 in → 376 out (328 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-haiku-4.5:none PASS 100% 162 in → 30 out (0 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 328 in → 37 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 359 in → 1,222 out (898 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
anthropic/claude-opus-4.7:high PASS 100% 219 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
anthropic/claude-opus-4.7:high__json_schema PASS 100% 438 in → 50 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.7:low PASS 100% 220 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
anthropic/claude-opus-4.7:max PASS 100% 219 in → 2,100 out (318 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
anthropic/claude-opus-4.7:max__json_schema PASS 100% 437 in → 1,461 out (237 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 438 in → 50 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 437 in → 50 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 432 in → 824 out (178 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 328 in → 386 out (218 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 161 in → 30 out (0 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 329 in → 36 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 328 in → 1,096 out (336 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 161 in → 30 out (0 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 330 in → 36 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 329 in → 532 out (237 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-5:high PASS 100% 213 in → 1,022 out (221 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 433 in → 165 out (59 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-5:low PASS 100% 214 in → 36 out (0 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 433 in → 48 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-5:medium PASS 100% 212 in → 244 out (94 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 433 in → 235 out (98 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-5:none__json_schema PASS 100% 431 in → 47 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 213 in → 1,230 out (205 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 433 in → 335 out (88 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 141 in → 1,069 out (1,059 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
arcee-ai/trinity-large-thinking:low PASS 100% 141 in → 1,615 out (1,616 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
arcee-ai/trinity-large-thinking:medium PASS 100% 142 in → 1,674 out (1,672 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
arcee-ai/trinity-large-thinking:xhigh PASS 100% 142 in → 1,837 out (1,845 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
arcee-ai/trinity-mini:high__json_schema PASS 100% 141 in → 397 out (417 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
arcee-ai/trinity-mini:low PASS 100% 141 in → 1,297 out (1,376 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
arcee-ai/trinity-mini:low__json_schema PASS 100% 142 in → 477 out (509 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
arcee-ai/trinity-mini:medium__json_schema PASS 100% 141 in → 379 out (377 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 147 in → 1,911 out (1,997 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 147 in → 1,805 out (2,221 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash-0731:max PASS 100% 224 in → 10,145 out (10,122 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 222 in → 613 out (591 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 146 in → 2,041 out (2,048 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 147 in → 23 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 146 in → 33 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 144 in → 799 out (791 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash:high PASS 100% 145 in → 3,168 out (3,542 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash:low PASS 100% 145 in → 758 out (724 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 145 in → 1,293 out (1,237 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash:medium PASS 100% 144 in → 1,533 out (1,509 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 146 in → 528 out (494 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
deepseek/deepseek-v4-flash:none PASS 100% 145 in → 23 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 145 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 338 in → 237 out (206 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 144 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
google/gemini-3-flash-preview:high PASS 100% 165 in → 1,224 out (1,198 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3-flash-preview:high__json_schema PASS 100% 224 in → 9,915 out (9,880 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3-flash-preview:low PASS 100% 162 in → 1,445 out (1,419 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3-flash-preview:medium PASS 100% 165 in → 1,962 out (1,936 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3-flash-preview:medium__json_schema PASS 100% 224 in → 10,622 out (10,588 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3-flash-preview:xhigh PASS 100% 165 in → 2,157 out (2,131 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 166 in → 1,693 out (1,658 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 165 in → 155 out (129 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 225 in → 162 out (122 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 164 in → 667 out (641 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 163 in → 775 out (736 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 224 in → 2,238 out (2,198 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 222 in → 1,784 out (1,750 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3.1-pro-preview:low PASS 100% 165 in → 856 out (830 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.1-pro-preview:medium PASS 100% 165 in → 1,105 out (1,079 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 225 in → 579 out (543 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
google/gemini-3.5-flash-lite:high PASS 100% 164 in → 1,592 out (1,566 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash-lite:max PASS 100% 165 in → 3,596 out (3,570 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash:high PASS 100% 164 in → 2,202 out (2,176 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash:high__json_schema PASS 100% 222 in → 1,346 out (1,312 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3.5-flash:low PASS 100% 164 in → 1,200 out (1,174 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash:low__json_schema PASS 100% 225 in → 555 out (515 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemini-3.5-flash:medium PASS 100% 165 in → 1,428 out (1,402 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash:xhigh PASS 100% 165 in → 4,138 out (4,112 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 224 in → 1,160 out (1,126 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3.6-flash:high PASS 100% 164 in → 8,556 out (8,530 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.6-flash:low__json_schema PASS 100% 163 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemini-3.6-flash:max PASS 100% 165 in → 6,850 out (6,824 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemini-3.6-flash:max__json_schema PASS 100% 225 in → 2,826 out (2,789 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 224 in → 2,923 out (2,888 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemma-4-26b-a4b-it:high PASS 100% 177 in → 710 out (558 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 179 in → 1,332 out (1,036 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-26b-a4b-it:low PASS 100% 181 in → 1,387 out (1,089 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 181 in → 2,665 out (2,293 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-26b-a4b-it:medium PASS 100% 178 in → 688 out (549 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 181 in → 2,013 out (1,695 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-26b-a4b-it:none PASS 100% 178 in → 27 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 176 in → 37 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 177 in → 2,071 out (1,730 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-31b-it:high PASS 100% 179 in → 958 out (757 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-31b-it:high__json_schema PASS 100% 179 in → 852 out (678 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-31b-it:low PASS 100% 181 in → 1,044 out (780 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-31b-it:low__json_schema PASS 100% 179 in → 537 out (383 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-31b-it:medium PASS 100% 180 in → 707 out (556 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-31b-it:medium__json_schema PASS 100% 184 in → 354 out (318 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
google/gemma-4-31b-it:none PASS 100% 178 in → 27 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-31b-it:none__json_schema PASS 100% 177 in → 41 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
google/gemma-4-31b-it:xhigh PASS 100% 181 in → 364 out (1 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 181 in → 458 out (330 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
ibm-granite/granite-4.1-8b PASS 100% 155 in → 23 out
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
ibm-granite/granite-4.1-8b__json_schema PASS 100% 155 in → 33 out
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
inclusionai/ling-2.6-flash PASS 100% 177 in → 26 out
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
inclusionai/ling-2.6-flash__json_schema PASS 100% 176 in → 38 out
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m2.5:high__json_schema PASS 100% 173 in → 767 out (798 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
minimax/minimax-m2.5:low PASS 100% 170 in → 744 out (721 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.5:low__json_schema PASS 100% 187 in → 1,573 out (1,542 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m2.5:medium PASS 100% 172 in → 879 out (856 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.5:medium__json_schema PASS 100% 172 in → 1,649 out (1,760 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m2.5:xhigh PASS 100% 187 in → 705 out (682 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 173 in → 1,056 out (1,109 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m2.7:high PASS 100% 174 in → 708 out (685 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.7:low PASS 100% 172 in → 900 out (932 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.7:medium__json_schema PASS 100% 303 in → 1,085 out (1,049 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m2.7:xhigh PASS 100% 176 in → 719 out (697 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 304 in → 1,109 out (1,081 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
minimax/minimax-m3:high PASS 100% 310 in → 246 out (223 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
minimax/minimax-m3:high__json_schema PASS 100% 297 in → 31 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
minimax/minimax-m3:low PASS 100% 310 in → 451 out (478 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
minimax/minimax-m3:medium PASS 100% 309 in → 629 out (683 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
minimax/minimax-m3:medium__json_schema PASS 100% 298 in → 32 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
minimax/minimax-m3:xhigh PASS 100% 306 in → 403 out (413 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
minimax/minimax-m3:xhigh__json_schema PASS 100% 296 in → 32 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
mistralai/devstral-2512__json_schema PASS 100% 163 in → 33 out
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
mistralai/mistral-medium-3-5:high PASS 100% 176 in → 8,326 out (8,478 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 163 in → 1,210 out (1,117 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
mistralai/mistral-medium-3-5:low PASS 100% 176 in → 4,739 out (4,870 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 163 in → 1,197 out (1,127 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
mistralai/mistral-medium-3-5:medium PASS 100% 175 in → 1,214 out (1,209 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 163 in → 1,191 out (1,118 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
mistralai/mistral-medium-3-5:none PASS 100% 174 in → 29 out (0 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
mistralai/mistral-medium-3-5:xhigh PASS 100% 175 in → 3,500 out (3,459 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 163 in → 844 out (743 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
moonshotai/kimi-k2.6:high__json_schema PASS 100% 144 in → 7,408 out (7,375 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
moonshotai/kimi-k2.6:medium PASS 100% 146 in → 3,160 out (3,136 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 144 in → 11,724 out (11,468 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
moonshotai/kimi-k2.7-code:high PASS 100% 144 in → 578 out (556 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 145 in → 607 out (664 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 228 in → 732 out (702 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 147 in → 3,590 out (3,566 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 145 in → 371 out (395 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
moonshotai/kimi-k3:high PASS 100% 222 in → 1,555 out (1,518 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:high__json_schema PASS 100% 318 in → 2,326 out (2,284 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
moonshotai/kimi-k3:low PASS 100% 224 in → 1,110 out (1,073 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:low__json_schema PASS 100% 222 in → 1,026 out (989 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:max PASS 100% 224 in → 1,183 out (1,143 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
moonshotai/kimi-k3:medium PASS 100% 223 in → 1,251 out (1,214 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
moonshotai/kimi-k3:medium__json_schema PASS 100% 221 in → 604 out (564 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
moonshotai/kimi-k3:xhigh PASS 100% 221 in → 980 out (943 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 176 in → 332 out (302 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 177 in → 558 out (471 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 176 in → 863 out (753 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 175 in → 389 out (361 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 176 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 176 in → 36 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 177 in → 471 out (416 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 177 in → 239 out (212 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 177 in → 1,588 out (1,459 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 175 in → 747 out (665 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 177 in → 281 out (218 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 176 in → 265 out (214 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 177 in → 390 out (321 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 176 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 174 in → 36 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 175 in → 736 out (660 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 175 in → 310 out (243 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
openai/gpt-5.4-mini:high PASS 100% 139 in → 543 out (516 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-mini:high__json_schema PASS 100% 171 in → 270 out (234 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4-mini:low PASS 100% 138 in → 167 out (140 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-mini:low__json_schema PASS 100% 174 in → 131 out (95 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4-mini:medium PASS 100% 136 in → 355 out (328 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-mini:medium__json_schema PASS 100% 172 in → 195 out (159 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4-mini:xhigh PASS 100% 137 in → 6,322 out (6,295 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 172 in → 3,398 out (3,362 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4-nano:high PASS 100% 136 in → 242 out (215 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-nano:high__json_schema PASS 100% 174 in → 131 out (94 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
openai/gpt-5.4-nano:low PASS 100% 135 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-nano:low__json_schema PASS 100% 174 in → 35 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
openai/gpt-5.4-nano:medium PASS 100% 137 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-nano:medium__json_schema PASS 100% 174 in → 35 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
openai/gpt-5.4-nano:none PASS 100% 138 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-nano:none__json_schema PASS 100% 172 in → 35 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 136 in → 244 out (217 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 173 in → 2,026 out (1,989 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
openai/gpt-5.4:high PASS 100% 136 in → 1,577 out (1,550 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4:high__json_schema PASS 100% 173 in → 1,983 out (1,947 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4:low PASS 100% 137 in → 205 out (178 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.4:low__json_schema PASS 100% 173 in → 217 out (181 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4:none__json_schema PASS 100% 174 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.4:xhigh PASS 100% 136 in → 3,924 out (3,897 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.5:high PASS 100% 137 in → 1,037 out (1,010 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.5:low__json_schema PASS 100% 173 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.5:medium PASS 100% 135 in → 543 out (516 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.5:medium__json_schema PASS 100% 173 in → 278 out (241 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
openai/gpt-5.5:none PASS 100% 137 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.5:xhigh__json_schema PASS 100% 172 in → 5,732 out (5,696 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:high PASS 100% 138 in → 383 out (356 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-luna:high__json_schema PASS 100% 172 in → 169 out (133 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:low__json_schema PASS 100% 173 in → 96 out (60 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:max PASS 100% 136 in → 377 out (350 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-luna:max__json_schema PASS 100% 173 in → 153 out (117 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:medium__json_schema PASS 100% 175 in → 304 out (268 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:none PASS 100% 136 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-luna:none__json_schema PASS 100% 176 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-luna:xhigh PASS 100% 137 in → 249 out (222 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 173 in → 211 out (175 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-sol:low PASS 100% 138 in → 369 out (342 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-sol:max PASS 100% 135 in → 578 out (551 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-sol:xhigh PASS 100% 138 in → 294 out (267 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-terra:low PASS 100% 138 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-terra:low__json_schema PASS 100% 172 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-terra:max PASS 100% 138 in → 1,197 out (1,170 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-5.6-terra:max__json_schema PASS 100% 175 in → 552 out (516 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-terra:none__json_schema PASS 100% 171 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 173 in → 34 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-oss-120b:high__json_schema PASS 100% 199 in → 33 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
openai/gpt-oss-120b:low PASS 100% 183 in → 91 out (68 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-oss-120b:medium__json_schema PASS 100% 184 in → 617 out (666 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 198 in → 5,588 out (6,574 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
openai/gpt-oss-20b:high PASS 100% 200 in → 987 out (1,081 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-oss-20b:high__json_schema PASS 100% 184 in → 812 out (900 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-oss-20b:low PASS 100% 199 in → 69 out (40 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-oss-20b:low__json_schema PASS 100% 184 in → 114 out (84 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-oss-20b:medium PASS 100% 197 in → 449 out (427 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
openai/gpt-oss-20b:medium__json_schema PASS 100% 186 in → 310 out (309 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 197 in → 990 out (968 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
poolside/laguna-m.1:high PASS 100% 165 in → 298 out (271 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-m.1:low PASS 100% 165 in → 902 out (875 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-m.1:low__json_schema PASS 100% 163 in → 1,778 out (1,742 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
poolside/laguna-m.1:medium PASS 100% 165 in → 2,495 out (2,469 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-m.1:xhigh PASS 100% 164 in → 1,259 out (1,232 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs-2.1:high__json_schema PASS 100% 162 in → 2,476 out (2,440 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
poolside/laguna-xs-2.1:low PASS 100% 164 in → 2,174 out (2,150 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs-2.1:low__json_schema PASS 100% 161 in → 1,581 out (1,549 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
poolside/laguna-xs-2.1:medium PASS 100% 164 in → 1,717 out (1,693 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 164 in → 1,061 out (1,025 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
poolside/laguna-xs-2.1:none__json_schema PASS 100% 164 in → 36 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
poolside/laguna-xs-2.1:xhigh PASS 100% 163 in → 666 out (641 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 165 in → 2,626 out (2,592 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
poolside/laguna-xs.2:high PASS 100% 200 in → 587 out (556 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
poolside/laguna-xs.2:high__json_schema PASS 100% 200 in → 451 out (419 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
poolside/laguna-xs.2:low PASS 100% 201 in → 359 out (332 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs.2:medium__json_schema PASS 100% 200 in → 556 out (524 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
poolside/laguna-xs.2:none PASS 100% 201 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs.2:none__json_schema PASS 100% 202 in → 36 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
poolside/laguna-xs.2:xhigh PASS 100% 202 in → 418 out (391 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 202 in → 454 out (421 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n"}
qwen/qwen3.6-27b:high PASS 100% 161 in → 1,154 out (1,087 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-27b:high__json_schema PASS 100% 162 in → 2,741 out (2,631 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-27b:medium PASS 100% 160 in → 1,541 out (1,420 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-27b:medium__json_schema PASS 100% 162 in → 2,714 out (1 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-27b:none PASS 100% 163 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 165 in → 37 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 162 in → 3,003 out (2,996 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 159 in → 1,416 out (1,324 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-35b-a3b:none PASS 100% 165 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 162 in → 3,598 out (3,405 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
qwen/qwen3.6-flash:high PASS 100% 161 in → 1,675 out (1,645 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-flash:low PASS 100% 163 in → 2,486 out (2,456 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-flash:max PASS 100% 163 in → 2,179 out (2,149 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-flash:medium PASS 100% 161 in → 3,755 out (3,725 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-flash:none PASS 100% 164 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-max:high PASS 100% 163 in → 860 out (830 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-max:low PASS 100% 161 in → 1,502 out (1,472 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-max:medium PASS 100% 163 in → 1,279 out (1,245 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
qwen/qwen3.7-max:none PASS 100% 165 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-max:xhigh PASS 100% 163 in → 913 out (883 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-plus:low PASS 100% 162 in → 721 out (687 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
qwen/qwen3.7-plus:none PASS 100% 164 in → 25 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
qwen/qwen3.7-plus:xhigh PASS 100% 162 in → 770 out (740 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
tencent/hy3:none PASS 100% 152 in → 23 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
tencent/hy3:none__json_schema PASS 100% 151 in → 32 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling-small:high PASS 100% 147 in → 1,052 out (1,025 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:high__json_schema PASS 100% 147 in → 2,266 out (2,232 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling-small:low PASS 100% 146 in → 127 out (107 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:low__json_schema PASS 100% 145 in → 122 out (88 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling-small:max PASS 100% 148 in → 1,215 out (1,277 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:medium PASS 100% 147 in → 715 out (688 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:medium__json_schema PASS 100% 146 in → 498 out (463 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling-small:xhigh PASS 100% 146 in → 4,208 out (4,438 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 145 in → 1,437 out (1,402 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
thinkingmachines/inkling:low PASS 100% 147 in → 57 out (29 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling:medium PASS 100% 145 in → 537 out (509 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
thinkingmachines/inkling:none PASS 100% 145 in → 24 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
x-ai/grok-4.3:high PASS 100% 314 in → 476 out (450 reasoning)
```txt Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
x-ai/grok-4.3:high__json_schema PASS 100% 374 in → 1,203 out (1,173 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
x-ai/grok-4.3:low PASS 100% 321 in → 863 out (842 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
x-ai/grok-4.3:low__json_schema PASS 100% 380 in → 753 out (727 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
x-ai/grok-4.3:max PASS 100% 315 in → 1,275 out (1,254 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
x-ai/grok-4.3:max__json_schema PASS 100% 374 in → 1,011 out (981 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
x-ai/grok-4.3:medium PASS 100% 321 in → 717 out (696 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
x-ai/grok-4.3:medium__json_schema PASS 100% 378 in → 890 out (860 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
x-ai/grok-4.3:none PASS 100% 314 in → 21 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
x-ai/grok-4.3:none__json_schema PASS 100% 370 in → 26 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
x-ai/grok-4.3:xhigh PASS 100% 316 in → 757 out (736 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
x-ai/grok-4.3:xhigh__json_schema PASS 100% 373 in → 799 out (773 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876"}
x-ai/grok-4.5:low__json_schema PASS 100% 409 in → 923 out (888 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
x-ai/grok-4.5:max__json_schema PASS 100% 406 in → 3,218 out (3,183 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
x-ai/grok-4.5:xhigh__json_schema PASS 100% 409 in → 1,379 out (1,346 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-4.7-flash:high PASS 100% 143 in → 1,070 out (1,021 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-4.7-flash:high__json_schema PASS 100% 143 in → 954 out (865 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-4.7-flash:low PASS 100% 142 in → 2,116 out (2,203 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-4.7-flash:low__json_schema PASS 100% 144 in → 1,098 out (1,062 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-4.7-flash:medium PASS 100% 142 in → 1,611 out (1,557 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 143 in → 1,170 out (1,089 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-4.7-flash:none PASS 100% 145 in → 23 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-4.7-flash:none__json_schema PASS 100% 143 in → 33 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-4.7-flash:xhigh PASS 100% 141 in → 1,602 out (1,594 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 144 in → 1,208 out (1,172 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-5.2:high PASS 100% 150 in → 443 out (418 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-5.2:low PASS 100% 150 in → 863 out (838 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-5.2:low__json_schema PASS 100% 142 in → 33 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-5.2:medium PASS 100% 151 in → 1,316 out (1,299 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
z-ai/glm-5.2:medium__json_schema PASS 100% 325 in → 551 out (516 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
z-ai/glm-5.2:xhigh PASS 100% 150 in → 2,122 out (2,139 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
anthropic/claude-fable-5:high FAIL 0% 212 in → 3 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['contact.txt']
anthropic/claude-fable-5:high__json_schema FAIL 0% 434 in → 236 out (91 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-fable-5:low FAIL 0% 211 in → 171 out (66 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` Reasoning: The rebased branch intentionally updated the email to `john@company.com`, while main updated the phone to `555-9876`. Combining both changes preserves the most complete set of updates (the branch's email change and main's phone change).
Failure: Content mismatch: ['contact.txt']
anthropic/claude-fable-5:low__json_schema FAIL 0% 434 in → 118 out (84 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-fable-5:max FAIL 0% 212 in → 9 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['contact.txt']
anthropic/claude-fable-5:max__json_schema FAIL 0% 433 in → 1,293 out (221 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-fable-5:medium FAIL 0% 212 in → 264 out (85 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` **Reasoning:** The branch being rebased deliberately changed the email to `john@company.com` (its only intentional change), while main's only unique update was the phone number `555-9876`. Combining both intentional changes preserves the most complete set of updates from each side.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-fable-5:medium__json_schema FAIL 0% 433 in → 196 out (72 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-fable-5:xhigh FAIL 0% 214 in → 3 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['contact.txt']
anthropic/claude-fable-5:xhigh__json_schema FAIL 0% 436 in → 572 out (96 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-haiku-4.5:high FAIL 0% 191 in → 1,863 out (880 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` The main branch has the most complete updates with both fields modified, whereas the branch being rebased only updated the email while keeping the original phone number.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-haiku-4.5:low FAIL 0% 191 in → 921 out (575 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` The main branch updated both fields comprehensively, making it the most complete version of the contact information.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-haiku-4.5:xhigh FAIL 0% 190 in → 402 out (333 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` The incoming branch (main) has the most complete updates, modifying both the email and phone number, whereas the rebasing branch only updated the email.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:high FAIL 0% 161 in → 1,699 out (591 reasoning)
## Resolved `contact.txt` ``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` ### Reasoning | Field | Base | Main Branch | Rebased Branch | Resolution | |-------|------|-------------|---------------|------------| | Email | `john@example.com` | `johndoe@personal.com` | `john@company.com` | **`john@company.com`** — intentional update by the rebased branch | | Phone | `555-1234` | `555-9876` | `555-1234` (unchanged) | **`555-9876`** — only main changed it, rebased branch had no opinion | The **most complete** resolution cherry-picks the deliberate changes from **both** sides: - **Email** → The rebased branch *specifically* updated it to `john@company.com` (its whole purpose), so that intent is preserved. - **Phone** → The rebased branch never touched it (still `555-1234`), so main's update to `555-9876` is accepted without conflict. This avoids discarding any intentional update from either branch.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 328 in → 1,271 out (369 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:low FAIL 0% 161 in → 24 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:low__json_schema FAIL 0% 328 in → 32 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:max FAIL 0% 160 in → 2,536 out (657 reasoning)
## Resolved `contact.txt` ``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` ## Reasoning | Field | Base | HEAD (being rebased) | Main (onto) | Resolution | |-------|------|----------------------|-------------|------------| | Email | `john@example.com` | `john@company.com` ✅ | `johndoe@personal.com` | **Keep rebased branch's intentional update** | | Phone | `555-1234` | `555-1234` (unchanged) | `555-9876` ✅ | **Keep main's update** (only branch that changed it) | The **most complete** resolution combines the **intentional**, non-overlapping updates from both branches: 1. **Email**: The rebased branch *deliberately* changed it from `john@example.com` → `john@company.com`. This is the specific change being rebased, so it's preserved. 2. **Phone**: Only main updated `555-1234` → `555-9876`. Since the rebased branch never touched it, main's update is accepted outright. This follows the rebase intent: **replay the feature branch's purposeful changes on top of main's current state**.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:max__json_schema FAIL 0% 328 in → 1,568 out (548 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:medium FAIL 0% 159 in → 29 out (0 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ```
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:medium__json_schema FAIL 0% 327 in → 33 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:none FAIL 0% 159 in → 210 out (0 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` **Reasoning:** - **Email:** The base had `john@example.com`. The rebased branch intentionally changed it to `john@company.com`, while main changed it to `johndoe@personal.com`. Both sides updated the email, but `john@company.com` is the change from the branch being rebased (the current work), so it represents the most recent intentional update during the rebase. - **Phone:** The base had `555-1234`. The rebased branch kept `555-1234` (no change), while main updated it to `555-9876`. Since only main updated the phone number, we take `555-9876` as the most complete update. This gives us the **most complete updates** from both sides: the rebased branch's new email and main's new phone number.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:none__json_schema FAIL 0% 327 in → 34 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:xhigh FAIL 0% 160 in → 1,609 out (469 reasoning)
## Resolved `contact.txt` ``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` ## Reasoning | Field | Base | Main Branch | Rebased Branch | Resolution | |-------|------|-------------|---------------|------------| | Email | `john@example.com` | `johndoe@personal.com` | `john@company.com` | **`john@company.com`** — The rebased branch intentionally updated this; its commits are being replayed *on top of* main, representing the latest intent. | | Phone | `555-1234` | `555-9876` | `555-1234` (unchanged) | **`555-9876`** — The rebased branch never touched the phone, so main's update to `555-9876` is accepted. | This gives the **most complete** result: the deliberate email change from the rebased branch is preserved, while main's phone number update (which the rebased branch had no opinion on) is also kept. No change from either branch is lost.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 328 in → 1,344 out (545 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:low__json_schema FAIL 0% 437 in → 46 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:medium FAIL 0% 219 in → 30 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:none FAIL 0% 219 in → 30 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:none__json_schema FAIL 0% 436 in → 46 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.7:xhigh FAIL 0% 217 in → 1,297 out (189 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` This resolution combines the meaningful updates from both branches: - **Email**: `john@company.com` — the intentional update from the branch being rebased (main's email change was superseded by this newer edit) - **Phone**: `555-9876` — the only phone update (from main); the rebased branch left it unchanged at the old value
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:high FAIL 0% 216 in → 2,112 out (363 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` **Reasoning for this resolution:** - **Phone → 555-9876:** The base value was `555-1234`. The branch being rebased left the phone *unchanged*, while main deliberately updated it to `555-9876`. Since only main made a real change here, that update should be preserved (no true conflict on phone). - **Email → john@company.com:** This is the genuine conflict—both sides changed the email from the base (`john@example.com`). Because this is a rebase, the whole point is to replay and preserve the feature branch's intentional work on top of main. The branch being rebased specifically updated the email to `john@company.com`, so that deliberate change is kept. The result combines main's phone update with the rebased branch's email update, giving the most complete set of intentional changes.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:low FAIL 0% 214 in → 451 out (165 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:low__json_schema FAIL 0% 435 in → 163 out (72 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:max FAIL 0% 215 in → 9,063 out (1,186 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ```
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 433 in → 9,359 out (1,164 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:medium FAIL 0% 212 in → 1,511 out (188 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` This resolution preserves every distinct update: - **Email → john@company.com**: The rebased branch explicitly updated this (its intent takes precedence as the commit being replayed). - **Phone → 555-9876**: Only main changed the phone (the branch left it at the base value 555-1234), so main's update is the meaningful one to keep. This keeps the most complete, non-redundant set of changes from both sides rather than discarding either branch's real update.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:medium__json_schema FAIL 0% 432 in → 663 out (147 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:none FAIL 0% 212 in → 368 out (0 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ``` **Reasoning for the resolution:** - **Email:** The base was `john@example.com`. Both branches changed it: - The rebased branch updated it to `john@company.com`. - Main updated it to `johndoe@personal.com`. Your instruction says the branch being rebased "kept john@example.com but updated the email to john@company.com" — so the intended active change on your branch is `john@company.com`. I preserved that. - **Phone:** The base was `555-1234`. Only main changed it to `555-9876` (your rebased branch kept the original `555-1234`). Since main made a real update and your branch made no intentional change, `555-9876` is the more complete/newer value. ⚠️ **Note:** The email is a genuine conflicting update — both sides deliberately changed the same field to different values. I chose `john@company.com` based on your stated intent, but if `johndoe@personal.com` (personal) was meant to supersede the company address, swap it accordingly. Verify with your team which email is authoritative.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:none__json_schema FAIL 0% 434 in → 43 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:xhigh FAIL 0% 213 in → 2,222 out (375 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
anthropic/claude-opus-4.8:xhigh__json_schema FAIL 0% 434 in → 2,477 out (352 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-4.6:high FAIL 0% 162 in → 482 out (179 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Resolution rationale:** - The main branch (`2eaff6d`) updated **both** fields, representing the most complete set of changes - The rebasing branch only updated the email (from `john@example.com` → `john@company.com`) but left the phone unchanged at the base value (`555-1234`) - Since the main branch made deliberate updates to both fields (`johndoe@personal.com` and `555-9876`), those represent the most fully updated state and are accepted in their entirety
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-4.6:max FAIL 0% 161 in → 1,371 out (394 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Resolution rationale:** - The base had `john@example.com` / `555-1234` - The rebased branch only updated the email → `john@company.com` (1 field changed) - The main branch (commit `65edf85`) updated **both** email → `johndoe@personal.com` and phone → `555-9876` (2 fields changed) - Keeping main branch's version preserves the **most complete** set of updates, as it modified both fields versus the single-field change from the rebased branch
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-4.6:medium FAIL 0% 159 in → 331 out (199 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Resolution reasoning:** The main branch (`ae15dcc`) represents the **most complete update** — it changed **both** the email and phone number. The rebased branch only partially updated the email field while leaving the phone unchanged from base. Since the goal is to preserve the most complete set of changes, the main branch's version of both fields is accepted in full.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-4.6:medium__json_schema FAIL 0% 328 in → 463 out (231 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 161 in → 442 out (195 reasoning)
``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` **Resolution rationale:** The main branch (incoming commit `0d76564`) updated **both** fields (email and phone), making it the most complete set of changes. The rebasing branch only updated the email while leaving the phone at the base value (`555-1234`), indicating a partial update. Accepting the main branch's version preserves the more thorough revision.
Failure: Content mismatch: ['contact.txt']
anthropic/claude-sonnet-5:none FAIL 0% 213 in → 271 out (0 reasoning)
Looking at this conflict, I need to determine the most complete and intentional set of updates: - **Branch being rebased**: changed email (john@example.com → john@company.com), left phone unchanged (555-1234) - **Main branch**: changed both email (→ johndoe@personal.com) and phone (→ 555-9876) Since main branch made a more complete update (both fields), and the rebasing branch only touched the email field while leaving phone untouched (matching the original base value), main's changes represent the more complete/recent update. The rebasing branch's email change doesn't have information suggesting it should override main's email change — main's update touches both fields together, suggesting a coordinated contact info update. **Resolved contact.txt:** ``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ```
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 141 in → 1,086 out (1,166 reasoning)
Name: John Doe Email:johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
arcee-ai/trinity-mini:medium FAIL 0% 142 in → 516 out (531 reasoning)
Name: John Doe Email:johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
arcee-ai/trinity-mini:xhigh FAIL 0% 140 in → 1,216 out (1,319 reasoning)
Name: John Doe Email:johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 141 in → 483 out (484 reasoning)
Name:John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name:John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-flash-0731:high FAIL 0% 145 in → 1,418 out (1,458 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 0% 146 in → 3,216 out (3,459 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-flash-0731:medium FAIL 0% 146 in → 2,980 out (3,055 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-flash-0731:xhigh FAIL 0% 150 in → 1,702 out (1,681 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-flash:high__json_schema FAIL 0% 145 in → 2,121 out (2,064 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-flash:xhigh FAIL 0% 224 in → 1,572 out (1,551 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output: Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:high FAIL 0% 146 in → 35 out (0 reasoning)
Here is the resolved file content with the most complete updates: ``` Name: John Doe Email: john@company.com Phone: 555-9876 ```
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-pro:high__json_schema FAIL 0% 145 in → 1,039 out (1,105 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-pro:low FAIL 0% 144 in → 1,845 out (1,931 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-pro:medium FAIL 0% 144 in → 2,638 out (2,617 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-pro:none FAIL 0% 144 in → 22 out (0 reasoning)
``` Name: John Doe Email: john@company.com Phone: 555-9876 ```
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
deepseek/deepseek-v4-pro:xhigh FAIL 0% 224 in → 14,481 out (14,461 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 146 in → 2,093 out (2,062 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
google/gemini-3-flash-preview:low__json_schema FAIL 0% 225 in → 6,338 out (6,300 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-flash-lite-preview:high FAIL 0% 164 in → 2,509 out (2,485 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-flash-lite-preview:high__json_schema FAIL 0% 165 in → 2,780 out (2,742 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-flash-lite-preview:xhigh FAIL 0% 165 in → 1,927 out (1,903 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:high FAIL 0% 163 in → 4,080 out (4,056 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 224 in → 887 out (853 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:xhigh FAIL 0% 163 in → 4,198 out (4,174 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.1-pro-preview:xhigh__json_schema FAIL 0% 222 in → 2,194 out (2,162 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.5-flash-lite:high__json_schema FAIL 0% 224 in → 1,497 out (1,464 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.5-flash-lite:low FAIL 0% 164 in → 893 out (869 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 223 in → 38 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
google/gemini-3.5-flash-lite:max__json_schema FAIL 0% 224 in → 4,456 out (4,424 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.5-flash-lite:medium FAIL 0% 166 in → 1,830 out (1,806 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.5-flash-lite:medium__json_schema FAIL 0% 164 in → 1,975 out (1,941 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe Email: john@company.com Phone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
google/gemini-3.5-flash-lite:xhigh FAIL 0% 163 in → 2,410 out (2,386 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.5-flash-lite:xhigh__json_schema FAIL 0% 165 in → 1,078 out (1,046 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.5-flash:medium__json_schema FAIL 0% 223 in → 3,823 out (3,785 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
google/gemini-3.6-flash:high__json_schema FAIL 0% 225 in → 1,740 out (1,705 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.6-flash:low FAIL 0% 166 in → 963 out (939 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.6-flash:medium FAIL 0% 164 in → 6,233 out (6,209 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemini-3.6-flash:medium__json_schema FAIL 0% 223 in → 2,535 out (2,501 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
google/gemini-3.6-flash:xhigh FAIL 0% 163 in → 10,532 out (10,508 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
google/gemma-4-26b-a4b-it:xhigh FAIL 0% 180 in → 2,146 out (1,719 reasoning)
(empty output)
Failure: Content mismatch: ['contact.txt']
liquid/lfm-2-24b-a2b FAIL 0% 157 in → 218 out
Here is the **resolved** `contact.txt` file with the most complete updates, merging both branches' changes logically: ```txt Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` ### Explanation: - **HEAD** (branch being rebased) has: - Email: `john@company.com` - Phone: `555-1234` - **1fd0129** (base branch) has: - Email: `johndoe@personal.com` - Phone: `555-9876` Since the branch being rebased kept `john@example.com` but updated the email to `john@company.com`, while the base branch updated both fields, the most complete resolution merges both sets of updates into a single, consistent record. If you want to keep both versions or clarify which change came from where, let me know! But for a standard merge, the above is correct.
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output: Here is the **resolved** `contact.txt` file with the most complete updates, merging both branches' changes logically and consistently: ```txt Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` ### Explanation: - **HEAD** (your branch): - Email: `john@company.com` - Phone: `555-1234` - **b044c6b** (merge target / base): - Email: `johndoe@personal.com` - Phone: `555-9876` **Resolution:** - Keep the new email (`johndoe@personal.com`) from your branch. - Keep the new phone number (`555-9876`) from your branch. - Remove the old email and phone from the base. This resolves the conflict by choosing the latest values from your branch, as is typical for a rebase. If you want to keep both emails/phones, you would need to merge them explicitly, but based on your instructions, this is the most complete update.
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:high FAIL 0% 172 in → 1,813 out (1,975 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
minimax/minimax-m2.7:medium FAIL 0% 172 in → 881 out (859 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
mistralai/devstral-2512 FAIL 0% 163 in → 112 out
Here's the resolved file content with the most complete updates: ``` Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ``` This resolution keeps: 1. The name unchanged (as it wasn't modified in either branch) 2. The email from the main branch (johndoe@personal.com) which is more complete than the branch being rebased 3. The phone number from the main branch (555-9876) which is the updated version
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:high FAIL 0% 145 in → 5,292 out (5,268 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.6:low FAIL 0% 144 in → 8,015 out (8,489 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:xhigh FAIL 0% 146 in → 3,985 out (4,207 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0% 143 in → 8,120 out (7,917 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k2.7-code:low FAIL 0% 146 in → 1,325 out (1,301 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.7-code:medium FAIL 0% 143 in → 277 out (253 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k3:max__json_schema FAIL 0% 222 in → 1,505 out (1,471 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
moonshotai/kimi-k3:xhigh__json_schema FAIL 0% 220 in → 1,766 out (1,732 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 176 in → 434 out (367 reasoning)
Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\\nEmail: johndoe@personal.com\\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema FAIL 0% 176 in → 927 out (861 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
nvidia/nemotron-3-super-120b-a12b:low FAIL 0% 177 in → 639 out (558 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.4-mini:none FAIL 0% 138 in → 23 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.4-mini:none__json_schema FAIL 0% 174 in → 32 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.4:medium FAIL 0% 136 in → 1,481 out (1,456 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.4:medium__json_schema FAIL 0% 174 in → 550 out (516 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.4:none FAIL 0% 136 in → 23 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.4:xhigh__json_schema FAIL 0% 173 in → 3,658 out (3,624 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.5:high__json_schema FAIL 0% 172 in → 550 out (516 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.5:low FAIL 0% 137 in → 500 out (475 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.5:none__json_schema FAIL 0% 171 in → 32 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.5:xhigh FAIL 0% 138 in → 231 out (206 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-luna:low FAIL 0% 138 in → 215 out (190 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-luna:medium FAIL 0% 137 in → 148 out (123 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-sol:high FAIL 0% 137 in → 135 out (110 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-sol:high__json_schema FAIL 0% 176 in → 143 out (109 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-sol:low__json_schema FAIL 0% 173 in → 32 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-sol:max__json_schema FAIL 0% 173 in → 284 out (250 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-sol:medium FAIL 0% 137 in → 153 out (128 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-sol:medium__json_schema FAIL 0% 174 in → 248 out (214 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-sol:none FAIL 0% 138 in → 23 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-sol:none__json_schema FAIL 0% 172 in → 32 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-sol:xhigh__json_schema FAIL 0% 174 in → 132 out (98 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-terra:high FAIL 0% 138 in → 475 out (450 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-terra:high__json_schema FAIL 0% 174 in → 32 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-terra:medium FAIL 0% 138 in → 23 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-terra:medium__json_schema FAIL 0% 172 in → 32 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content":"Name: John Doe\nEmail: john@company.com\nPhone: 555-9876"}
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-terra:none FAIL 0% 136 in → 23 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-5.6-terra:xhigh FAIL 0% 138 in → 23 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-oss-120b:high FAIL 0% 195 in → 3,175 out (3,727 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-oss-120b:low__json_schema FAIL 0% 186 in → 234 out (240 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
openai/gpt-oss-120b:medium FAIL 0% 197 in → 320 out (337 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-oss-120b:xhigh FAIL 0% 200 in → 1,398 out (1,544 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
openai/gpt-oss-20b:xhigh FAIL 0% 200 in → 983 out (971 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555‑9876
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:high__json_schema FAIL 0% 164 in → 3,265 out (3,230 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:medium__json_schema FAIL 0% 163 in → 970 out (935 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
poolside/laguna-m.1:none FAIL 0% 162 in → 25 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
poolside/laguna-m.1:xhigh__json_schema FAIL 0% 163 in → 1,169 out (1,133 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
poolside/laguna-xs-2.1:high FAIL 0% 164 in → 582 out (551 reasoning)
--- contact.txt Name: John Doe Email: johndoe@personal.com Phone: 555-9876 ---
Failure: Content mismatch: ['contact.txt']
poolside/laguna-xs-2.1:none FAIL 0% 163 in → 50 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876 </think>Name: John Doe Email: johndoe@personal.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
poolside/laguna-xs.2:low__json_schema FAIL 0% 202 in → 629 out (597 reasoning)
Name: John Doe Email: john@company.com Phone: 555-1234
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-1234"}
Failure: Content mismatch: ['contact.txt']
poolside/laguna-xs.2:medium FAIL 0% 200 in → 429 out (403 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-27b:low FAIL 0% 161 in → 15,922 out (15,559 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-27b:low__json_schema FAIL 0% 163 in → 2,652 out (2,684 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-27b:xhigh FAIL 0% 160 in → 2,890 out (2,818 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-35b-a3b:high FAIL 0% 163 in → 2,329 out (2,142 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-35b-a3b:low FAIL 0% 162 in → 2,574 out (2,367 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-35b-a3b:low__json_schema FAIL 0% 160 in → 2,500 out (2,337 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
qwen/qwen3.6-35b-a3b:medium FAIL 0% 163 in → 2,641 out (2,515 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-35b-a3b:xhigh FAIL 0% 163 in → 2,687 out (2,505 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9b67ea31-84e0-9f16-895a-05eb1c37c981","request_id":"9b67ea31-84e0-9f16-895a-05eb1c37c981"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low FAIL 0% 163 in → 3,470 out (3,442 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3c66f879-c7d8-96a8-af5c-50d54fb3f722","request_id":"3c66f879-c7d8-96a8-af5c-50d54fb3f722"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium FAIL 0% 162 in → 2,468 out (2,440 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5dc4a99e-631b-9dcc-8378-e6f6af343e7d","request_id":"5dc4a99e-631b-9dcc-8378-e6f6af343e7d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none FAIL 0% 164 in → 23 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fe2b1182-c80e-9b94-ab0b-2e89cf108aa7","request_id":"fe2b1182-c80e-9b94-ab0b-2e89cf108aa7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh FAIL 0% 162 in → 2,420 out (2,392 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f93d82cb-1798-9a0d-8974-bda0dd63c72d","request_id":"f93d82cb-1798-9a0d-8974-bda0dd63c72d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high FAIL 0% 162 in → 4,241 out (4,213 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b5af6440-0002-9259-be39-54f1f32b78a6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a46ef355-3a9c-9d65-9a2b-2799515bf974"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-136e1010-def0-9b03-93b0-75a4f0a8497e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5b00da59-c1ef-91b6-a749-9619a87607ff"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ad34e31a-4af6-9f16-bc29-8d5d5206fdfd"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh FAIL 0% 162 in → 3,046 out (3,018 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-85a5d847-5710-92e4-b231-783430a346e8"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4fede31a-2b74-9a99-83b4-bd290db471ed","request_id":"4fede31a-2b74-9a99-83b4-bd290db471ed"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2c236e46-4b4b-94b5-a8b8-6c590c3e2baa","request_id":"2c236e46-4b4b-94b5-a8b8-6c590c3e2baa"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fdae6cf9-9e0b-9eea-95de-6b83c988dc33","request_id":"fdae6cf9-9e0b-9eea-95de-6b83c988dc33"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f238b70b-a0af-9413-9307-85ab35267f30","request_id":"f238b70b-a0af-9413-9307-85ab35267f30"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high FAIL 0% 163 in → 865 out (837 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-366815ce-e7f8-9415-81b8-d85ba37513b5","request_id":"366815ce-e7f8-9415-81b8-d85ba37513b5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a6143c92-ff49-95b4-bba1-5282d7b4b513","request_id":"a6143c92-ff49-95b4-bba1-5282d7b4b513"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium FAIL 0% 161 in → 5,329 out (5,019 reasoning)
``` Name: John Doe Email: johndoe@company.com Phone: 555-9876 ``` **Reasoning:** | Field | Base | Branch Change | Main Change | Resolved | |-------|------|---------------|-------------|----------| | **Email** | `john@example.com` | `john@company.com` (updated domain) | `johndoe@personal.com` (updated username + domain) | `johndoe@company.com` | | **Phone** | `555-1234` | *unchanged* | `555-9876` | `555-9876` | - **Phone:** The branch made no change, so main's update (`555-9876`) is accepted cleanly. - **Email:** Both sides modified the email from the base. To produce the *most complete* result, the changes are merged: - **Username** `johndoe` from main's more specific update (`john` → `johndoe`). - **Domain** `company.com` from the branch, which takes precedence as the top-most applied change in a rebase (domain `example.com` → `company.com`). - Combined: `johndoe@company.com` This resolution preserves every intentional change from both branches without data loss.
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0aeba386-f3b3-929a-8c06-5815f526b689","request_id":"0aeba386-f3b3-929a-8c06-5815f526b689"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-55808b32-0ff3-9170-8708-2ced82a88dc7","request_id":"55808b32-0ff3-9170-8708-2ced82a88dc7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:high FAIL 0% 149 in → 8,451 out (8,430 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
tencent/hy3:high__json_schema FAIL 0% 149 in → 6,929 out (6,908 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
tencent/hy3:low FAIL 0% 149 in → 2,621 out (2,600 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
tencent/hy3:low__json_schema FAIL 0% 147 in → 3,991 out (4,074 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "resolved_content": "" }
Failure: Content mismatch: ['contact.txt']
tencent/hy3:medium FAIL 0% 151 in → 1,805 out (1,784 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
tencent/hy3:medium__json_schema FAIL 0% 150 in → 4,323 out (4,369 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
tencent/hy3:xhigh FAIL 0% 148 in → 16,589 out (16,568 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
tencent/hy3:xhigh__json_schema FAIL 0% 149 in → 8,180 out (8,159 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
thinkingmachines/inkling-small:max__json_schema FAIL 0% 146 in → 15,139 out (15,105 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
thinkingmachines/inkling-small:none FAIL 0% 147 in → 24 out (0 reasoning)
Name: John Doe Email: johndoe@personal.com Phone: 555-9876<|end_message|>
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output: Name: John Doe Email: johndoe@personal.com Phone: 555-9876{ "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high FAIL 0% 145 in → 3,228 out (3,203 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output: Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:max FAIL 0% 145 in → 17,317 out (17,293 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
thinkingmachines/inkling:max__json_schema FAIL 0% 145 in → 17,822 out (17,790 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
Invalid JSON. Output: Name: John Doe Email: johndoe@personal.com Phone: 555-9876
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Name: John Doe Email: johndoe@personal.com Phone: 555-9876{ "resolved_content": "Name: John Doe\nEmail: johndoe@personal.com\nPhone: 555-9876\n" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:xhigh FAIL 0% 145 in → 7,821 out (7,807 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
thinkingmachines/inkling:xhigh__json_schema FAIL 0% 146 in → 2,649 out (2,616 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:high FAIL 0% 338 in → 9,029 out (9,010 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:high__json_schema FAIL 0% 407 in → 3,085 out (3,054 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876" }
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:low FAIL 0% 339 in → 643 out (624 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:max FAIL 0% 339 in → 1,423 out (1,404 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:medium FAIL 0% 340 in → 5,645 out (5,626 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:medium__json_schema FAIL 0% 407 in → 2,500 out (2,467 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) { "resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n" }
Failure: Content mismatch: ['contact.txt']
x-ai/grok-4.5:xhigh FAIL 0% 338 in → 5,938 out (5,919 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
z-ai/glm-5.2:high__json_schema FAIL 0% 326 in → 1,513 out (1,483 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
JSON Schema Structured Output
(raw) {"resolved_content": "Name: John Doe\nEmail: john@company.com\nPhone: 555-9876\n"}
Failure: Content mismatch: ['contact.txt']
z-ai/glm-5.2:none FAIL 0% 142 in → 22 out (0 reasoning)
Name: John Doe Email: john@company.com Phone: 555-9876
Failure: Content mismatch: ['contact.txt']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)