f003 — git_show

← git_show
Show lightweight tag pointing to a commit
Tests ability to inspect a lightweight tag pointing to a commit. Evaluates tag-to-commit resolution.
easy git-show lightweight-tag tag
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'v1 content' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Release version 1.0'
  7. 07 git tag v1.0.0
  8. 08 echo 'v2 content' > app.txt
  9. 09 git add app.txt
  10. 10 git commit -m 'Release version 1.1'
  11. 11 git tag v1.1.0
Prompt
Using git show, what is the commit message of the commit that the lightweight tag 'v1.0.0' points to? Output ONLY the commit message, nothing else.
Expected
Release version 1.0
Fixture Metrics

Scoped model quality, cost, API time, and token usage for git_show/f003.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 774 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-fable-5:high__json_schema PASS 100% 985 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-fable-5:low PASS 100% 780 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-fable-5:low__json_schema PASS 100% 999 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-fable-5:max PASS 100% 760 in → 75 out (34 reasoning)
Release version 1.0
anthropic/claude-fable-5:max__json_schema PASS 100% 995 in → 77 out (22 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-fable-5:medium__json_schema PASS 100% 1,001 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-fable-5:xhigh PASS 100% 768 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 1,011 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-haiku-4.5:high PASS 100% 693 in → 120 out (88 reasoning)
Release version 1.0
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 863 in → 335 out (255 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-haiku-4.5:low PASS 100% 691 in → 157 out (110 reasoning)
Release version 1.0
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 865 in → 107 out (83 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-haiku-4.5:medium PASS 100% 697 in → 126 out (91 reasoning)
Release version 1.0
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 867 in → 302 out (244 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-haiku-4.5:none PASS 100% 670 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 827 in → 16 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 695 in → 137 out (103 reasoning)
Release version 1.0
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 865 in → 218 out (156 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-opus-4.6:high PASS 100% 680 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.6:high__json_schema PASS 100% 828 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.6:low PASS 100% 664 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.6:low__json_schema PASS 100% 842 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.6:max PASS 100% 670 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.6:max__json_schema PASS 100% 836 in → 114 out (64 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.6:medium PASS 100% 672 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 834 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.6:none PASS 100% 666 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.6:none__json_schema PASS 100% 840 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.6:xhigh PASS 100% 664 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 838 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.7:high PASS 100% 775 in → 11 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.7:high__json_schema PASS 100% 1,002 in → 21 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.7:low PASS 100% 773 in → 11 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.7:low__json_schema PASS 100% 1,004 in → 21 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.7:max PASS 100% 775 in → 11 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.7:max__json_schema PASS 100% 1,006 in → 21 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.7:medium PASS 100% 777 in → 11 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 990 in → 21 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.7:none PASS 100% 783 in → 11 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.7:none__json_schema PASS 100% 1,008 in → 21 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.7:xhigh PASS 100% 783 in → 11 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 988 in → 21 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.8:high PASS 100% 782 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.8:high__json_schema PASS 100% 981 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.8:low PASS 100% 778 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.8:low__json_schema PASS 100% 993 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.8:max PASS 100% 780 in → 72 out (21 reasoning)
Release version 1.0
anthropic/claude-opus-4.8:max__json_schema PASS 100% 997 in → 80 out (21 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-opus-4.8:medium PASS 100% 764 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 995 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.8:none PASS 100% 766 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.8:none__json_schema PASS 100% 981 in → 18 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-opus-4.8:xhigh PASS 100% 776 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 997 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-sonnet-4.6:high PASS 100% 670 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 838 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-sonnet-4.6:low PASS 100% 666 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 846 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-sonnet-4.6:max PASS 100% 662 in → 109 out (65 reasoning)
Release version 1.0
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 838 in → 153 out (90 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-sonnet-4.6:medium PASS 100% 668 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 842 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-sonnet-4.6:none PASS 100% 668 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 842 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 668 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 828 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
anthropic/claude-sonnet-5:high PASS 100% 768 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-sonnet-5:high__json_schema PASS 100% 991 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-sonnet-5:low PASS 100% 772 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-sonnet-5:low__json_schema PASS 100% 987 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-sonnet-5:medium PASS 100% 778 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 983 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-sonnet-5:none PASS 100% 780 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-sonnet-5:none__json_schema PASS 100% 995 in → 18 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
anthropic/claude-sonnet-5:xhigh PASS 100% 778 in → 9 out (0 reasoning)
Release version 1.0
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 993 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
arcee-ai/trinity-large-thinking:high PASS 100% 563 in → 496 out (491 reasoning)
Release version 1.0
arcee-ai/trinity-large-thinking:low PASS 100% 545 in → 432 out (425 reasoning)
Release version 1.0
arcee-ai/trinity-large-thinking:medium PASS 100% 545 in → 687 out (682 reasoning)
Release version 1.0
arcee-ai/trinity-large-thinking:xhigh PASS 100% 557 in → 505 out (496 reasoning)
Release version 1.0
arcee-ai/trinity-mini:high PASS 100% 549 in → 344 out (332 reasoning)
Release version 1.0
arcee-ai/trinity-mini:high__json_schema PASS 100% 553 in → 380 out (378 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
arcee-ai/trinity-mini:low PASS 100% 557 in → 339 out (327 reasoning)
Release version 1.0
arcee-ai/trinity-mini:low__json_schema PASS 100% 545 in → 323 out (318 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
arcee-ai/trinity-mini:medium PASS 100% 545 in → 298 out (293 reasoning)
Release version 1.0
arcee-ai/trinity-mini:medium__json_schema PASS 100% 549 in → 281 out (254 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message":"Release version 1.0" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 557 in → 156 out (136 reasoning)
Release version 1.0
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 565 in → 144 out (120 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 559 in → 60 out (49 reasoning)
Release version 1.0
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 551 in → 195 out (174 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 663 in → 136 out (127 reasoning)
Release version 1.0
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 845 in → 403 out (390 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 563 in → 55 out (48 reasoning)
Release version 1.0
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 559 in → 191 out (168 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 551 in → 7 out (0 reasoning)
Release version 1.0
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 561 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 642 in → 58 out (48 reasoning)
Release version 1.0
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 660 in → 47 out (32 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
deepseek/deepseek-v4-flash:low PASS 100% 553 in → 91 out (83 reasoning)
Release version 1.0
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 649 in → 13 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
deepseek/deepseek-v4-flash:medium PASS 100% 565 in → 122 out (96 reasoning)
Release version 1.0
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 655 in → 13 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
deepseek/deepseek-v4-flash:none PASS 100% 573 in → 7 out (0 reasoning)
Release version 1.0
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 561 in → 13 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
deepseek/deepseek-v4-flash:xhigh PASS 100% 634 in → 235 out (227 reasoning)
Release version 1.0
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 648 in → 239 out (200 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
deepseek/deepseek-v4-pro:high PASS 100% 557 in → 115 out (107 reasoning)
Release version 1.0
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 557 in → 13 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
deepseek/deepseek-v4-pro:low PASS 100% 559 in → 72 out (64 reasoning)
Release version 1.0
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 563 in → 123 out (106 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
deepseek/deepseek-v4-pro:medium PASS 100% 565 in → 7 out (0 reasoning)
Release version 1.0
deepseek/deepseek-v4-pro:none PASS 100% 561 in → 7 out (0 reasoning)
Release version 1.0
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 563 in → 12 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 646 in → 203 out (195 reasoning)
Release version 1.0
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 573 in → 16 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
google/gemini-3-flash-preview:high PASS 100% 725 in → 330 out (324 reasoning)
Release version 1.0
google/gemini-3-flash-preview:high__json_schema PASS 100% 796 in → 126 out (114 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3-flash-preview:low PASS 100% 729 in → 342 out (336 reasoning)
Release version 1.0
google/gemini-3-flash-preview:low__json_schema PASS 100% 790 in → 571 out (558 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemini-3-flash-preview:medium PASS 100% 725 in → 336 out (330 reasoning)
Release version 1.0
google/gemini-3-flash-preview:medium__json_schema PASS 100% 786 in → 141 out (129 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3-flash-preview:xhigh PASS 100% 705 in → 209 out (203 reasoning)
Release version 1.0
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 708 in → 415 out (403 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 725 in → 353 out (347 reasoning)
Release version 1.0
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 722 in → 314 out (301 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 730 in → 149 out (143 reasoning)
Release version 1.0
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 720 in → 156 out (143 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 717 in → 266 out (260 reasoning)
Release version 1.0
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 720 in → 307 out (294 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 730 in → 358 out (352 reasoning)
Release version 1.0
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 728 in → 359 out (346 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemini-3.1-pro-preview:high PASS 100% 723 in → 440 out (434 reasoning)
Release version 1.0
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 778 in → 90 out (78 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.1-pro-preview:low PASS 100% 729 in → 110 out (104 reasoning)
Release version 1.0
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 786 in → 146 out (133 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemini-3.1-pro-preview:medium PASS 100% 723 in → 358 out (352 reasoning)
Release version 1.0
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 792 in → 182 out (170 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 735 in → 434 out (428 reasoning)
Release version 1.0
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 778 in → 138 out (126 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.5-flash-lite:high PASS 100% 717 in → 322 out (316 reasoning)
Release version 1.0
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 794 in → 283 out (265 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
google/gemini-3.5-flash-lite:low PASS 100% 725 in → 6 out (0 reasoning)
Release version 1.0
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 790 in → 18 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
google/gemini-3.5-flash-lite:max PASS 100% 738 in → 315 out (309 reasoning)
Release version 1.0
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 788 in → 365 out (347 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
google/gemini-3.5-flash-lite:medium PASS 100% 719 in → 157 out (151 reasoning)
Release version 1.0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 728 in → 362 out (349 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 730 in → 238 out (232 reasoning)
Release version 1.0
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 728 in → 226 out (213 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemini-3.5-flash:high PASS 100% 733 in → 476 out (470 reasoning)
Release version 1.0
google/gemini-3.5-flash:high__json_schema PASS 100% 788 in → 310 out (292 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
google/gemini-3.5-flash:low PASS 100% 723 in → 120 out (114 reasoning)
Release version 1.0
google/gemini-3.5-flash:low__json_schema PASS 100% 792 in → 165 out (153 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.5-flash:medium PASS 100% 725 in → 441 out (435 reasoning)
Release version 1.0
google/gemini-3.5-flash:medium__json_schema PASS 100% 786 in → 188 out (176 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.5-flash:xhigh PASS 100% 727 in → 433 out (427 reasoning)
Release version 1.0
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 794 in → 328 out (316 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.6-flash:high PASS 100% 732 in → 204 out (198 reasoning)
Release version 1.0
google/gemini-3.6-flash:high__json_schema PASS 100% 718 in → 270 out (258 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.6-flash:low PASS 100% 724 in → 148 out (142 reasoning)
Release version 1.0
google/gemini-3.6-flash:low__json_schema PASS 100% 804 in → 12 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.6-flash:max PASS 100% 727 in → 310 out (304 reasoning)
Release version 1.0
google/gemini-3.6-flash:max__json_schema PASS 100% 724 in → 426 out (414 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.6-flash:medium PASS 100% 724 in → 181 out (175 reasoning)
Release version 1.0
google/gemini-3.6-flash:medium__json_schema PASS 100% 790 in → 245 out (233 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemini-3.6-flash:xhigh PASS 100% 725 in → 263 out (257 reasoning)
Release version 1.0
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 780 in → 253 out (241 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
google/gemma-4-26b-a4b-it:high PASS 100% 727 in → 489 out (279 reasoning)
Release version 1.0
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 744 in → 492 out (311 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
google/gemma-4-26b-a4b-it:low PASS 100% 739 in → 428 out (421 reasoning)
Release version 1.0
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 741 in → 523 out (270 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
google/gemma-4-26b-a4b-it:medium PASS 100% 734 in → 312 out (213 reasoning)
Release version 1.0
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 745 in → 475 out (310 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
google/gemma-4-26b-a4b-it:none PASS 100% 738 in → 7 out (0 reasoning)
Release version 1.0
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 742 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 745 in → 519 out (338 reasoning)
Release version 1.0
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 744 in → 516 out (282 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemma-4-31b-it:high PASS 100% 737 in → 292 out (180 reasoning)
Release version 1.0
google/gemma-4-31b-it:high__json_schema PASS 100% 741 in → 237 out (134 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
google/gemma-4-31b-it:low PASS 100% 739 in → 198 out (121 reasoning)
Release version 1.0
google/gemma-4-31b-it:low__json_schema PASS 100% 740 in → 279 out (141 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemma-4-31b-it:medium PASS 100% 745 in → 256 out (165 reasoning)
Release version 1.0
google/gemma-4-31b-it:medium__json_schema PASS 100% 733 in → 314 out (196 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemma-4-31b-it:none PASS 100% 740 in → 7 out (0 reasoning)
Release version 1.0
google/gemma-4-31b-it:none__json_schema PASS 100% 732 in → 14 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
google/gemma-4-31b-it:xhigh PASS 100% 744 in → 295 out (179 reasoning)
Release version 1.0
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 739 in → 353 out (176 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
ibm-granite/granite-4.1-8b PASS 100% 630 in → 7 out
Release version 1.0
ibm-granite/granite-4.1-8b__json_schema PASS 100% 624 in → 16 out
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
inclusionai/ling-2.6-flash PASS 100% 716 in → 7 out
Release version 1.0
inclusionai/ling-2.6-flash__json_schema PASS 100% 718 in → 15 out
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
liquid/lfm-2-24b-a2b PASS 100% 592 in → 7 out
Release version 1.0
minimax/minimax-m2.5:high PASS 100% 661 in → 87 out (79 reasoning)
Release version 1.0
minimax/minimax-m2.5:high__json_schema PASS 100% 661 in → 125 out (92 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
minimax/minimax-m2.5:low PASS 100% 591 in → 206 out (161 reasoning)
Release version 1.0
minimax/minimax-m2.5:low__json_schema PASS 100% 587 in → 229 out (211 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
minimax/minimax-m2.5:medium PASS 100% 585 in → 220 out (212 reasoning)
Release version 1.0
minimax/minimax-m2.5:medium__json_schema PASS 100% 587 in → 159 out (142 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
minimax/minimax-m2.5:xhigh PASS 100% 593 in → 216 out (213 reasoning)
Release version 1.0
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 587 in → 253 out (235 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
minimax/minimax-m2.7:high PASS 100% 594 in → 287 out (297 reasoning)
Release version 1.0
minimax/minimax-m2.7:high__json_schema PASS 100% 717 in → 320 out (306 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
minimax/minimax-m2.7:low PASS 100% 579 in → 103 out (84 reasoning)
Release version 1.0
minimax/minimax-m2.7:low__json_schema PASS 100% 585 in → 17 out (17 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
minimax/minimax-m2.7:medium PASS 100% 583 in → 215 out (208 reasoning)
Release version 1.0
minimax/minimax-m2.7:medium__json_schema PASS 100% 731 in → 338 out (325 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
minimax/minimax-m2.7:xhigh PASS 100% 605 in → 203 out (196 reasoning)
Release version 1.0
minimax/minimax-m3:high PASS 100% 729 in → 91 out (67 reasoning)
Release version 1.0
minimax/minimax-m3:high__json_schema PASS 100% 671 in → 16 out (15 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
minimax/minimax-m3:low PASS 100% 711 in → 85 out (68 reasoning)
Release version 1.0
minimax/minimax-m3:low__json_schema PASS 100% 710 in → 13 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
minimax/minimax-m3:medium PASS 100% 681 in → 94 out (86 reasoning)
Release version 1.0
minimax/minimax-m3:medium__json_schema PASS 100% 720 in → 16 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
minimax/minimax-m3:xhigh PASS 100% 727 in → 89 out (69 reasoning)
Release version 1.0
minimax/minimax-m3:xhigh__json_schema PASS 100% 677 in → 15 out (14 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
mistralai/devstral-2512 PASS 100% 682 in → 7 out
Release version 1.0
mistralai/devstral-2512__json_schema PASS 100% 676 in → 13 out
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
mistralai/mistral-medium-3-5:high PASS 100% 698 in → 369 out (318 reasoning)
Release version 1.0
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 676 in → 406 out (360 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
mistralai/mistral-medium-3-5:low PASS 100% 680 in → 287 out (244 reasoning)
Release version 1.0
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 668 in → 327 out (279 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
mistralai/mistral-medium-3-5:medium PASS 100% 680 in → 361 out (327 reasoning)
Release version 1.0
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 674 in → 483 out (420 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
mistralai/mistral-medium-3-5:none PASS 100% 690 in → 7 out (0 reasoning)
Release version 1.0
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 678 in → 13 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 686 in → 296 out (249 reasoning)
Release version 1.0
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 680 in → 392 out (243 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
moonshotai/kimi-k2.6:high PASS 100% 557 in → 321 out (313 reasoning)
Release version 1.0
moonshotai/kimi-k2.6:low PASS 100% 557 in → 170 out (164 reasoning)
Release version 1.0
moonshotai/kimi-k2.6:low__json_schema PASS 100% 562 in → 352 out (199 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
moonshotai/kimi-k2.6:medium PASS 100% 553 in → 218 out (209 reasoning)
Release version 1.0
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 559 in → 156 out (137 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
moonshotai/kimi-k2.6:xhigh PASS 100% 543 in → 153 out (145 reasoning)
Release version 1.0
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 555 in → 323 out (273 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
moonshotai/kimi-k2.7-code:high PASS 100% 560 in → 96 out (88 reasoning)
Release version 1.0
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 565 in → 130 out (92 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
moonshotai/kimi-k2.7-code:low PASS 100% 558 in → 69 out (61 reasoning)
Release version 1.0
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 545 in → 127 out (107 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
moonshotai/kimi-k2.7-code:medium PASS 100% 557 in → 92 out (80 reasoning)
Release version 1.0
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 642 in → 119 out (105 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 558 in → 106 out (98 reasoning)
Release version 1.0
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 559 in → 87 out (59 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
moonshotai/kimi-k3:high PASS 100% 638 in → 118 out (97 reasoning)
Release version 1.0
moonshotai/kimi-k3:high__json_schema PASS 100% 630 in → 72 out (51 reasoning)
Release version 1.0
moonshotai/kimi-k3:low PASS 100% 630 in → 115 out (94 reasoning)
Release version 1.0
moonshotai/kimi-k3:low__json_schema PASS 100% 628 in → 113 out (92 reasoning)
Release version 1.0
moonshotai/kimi-k3:max PASS 100% 642 in → 114 out (93 reasoning)
Release version 1.0
moonshotai/kimi-k3:max__json_schema PASS 100% 636 in → 113 out (92 reasoning)
Release version 1.0
moonshotai/kimi-k3:medium PASS 100% 628 in → 103 out (82 reasoning)
Release version 1.0
moonshotai/kimi-k3:medium__json_schema PASS 100% 636 in → 111 out (90 reasoning)
Release version 1.0
moonshotai/kimi-k3:xhigh PASS 100% 632 in → 112 out (91 reasoning)
Release version 1.0
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 636 in → 115 out (94 reasoning)
Release version 1.0
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 699 in → 205 out (199 reasoning)
Release version 1.0
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 697 in → 251 out (233 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 695 in → 159 out (145 reasoning)
Release version 1.0
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 705 in → 139 out (94 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 701 in → 304 out (230 reasoning)
Release version 1.0
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 693 in → 360 out (274 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 691 in → 7 out (0 reasoning)
Release version 1.0
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 693 in → 117 out (109 reasoning)
Release version 1.0
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 699 in → 128 out (83 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 697 in → 109 out (86 reasoning)
Release version 1.0
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 689 in → 85 out (66 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 703 in → 100 out (88 reasoning)
Release version 1.0
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 697 in → 142 out (110 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 671 in → 360 out (285 reasoning)
Release version 1.0
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 695 in → 152 out (125 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 705 in → 7 out (0 reasoning)
Release version 1.0
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 691 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 691 in → 145 out (113 reasoning)
Release version 1.0
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 683 in → 127 out (98 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
openai/gpt-5.4-mini:high PASS 100% 577 in → 96 out (84 reasoning)
Release version 1.0
openai/gpt-5.4-mini:high__json_schema PASS 100% 589 in → 93 out (72 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4-mini:low PASS 100% 557 in → 36 out (24 reasoning)
Release version 1.0
openai/gpt-5.4-mini:low__json_schema PASS 100% 595 in → 137 out (116 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4-mini:medium PASS 100% 569 in → 62 out (50 reasoning)
Release version 1.0
openai/gpt-5.4-mini:medium__json_schema PASS 100% 593 in → 80 out (59 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4-mini:none PASS 100% 557 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.4-mini:none__json_schema PASS 100% 601 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4-mini:xhigh PASS 100% 559 in → 94 out (82 reasoning)
Release version 1.0
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 593 in → 287 out (266 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4-nano:high PASS 100% 557 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.4-nano:high__json_schema PASS 100% 599 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4-nano:low PASS 100% 567 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.4-nano:low__json_schema PASS 100% 607 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4-nano:medium PASS 100% 557 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.4-nano:medium__json_schema PASS 100% 603 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4-nano:none PASS 100% 549 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.4-nano:none__json_schema PASS 100% 605 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4-nano:xhigh PASS 100% 563 in → 88 out (76 reasoning)
Release version 1.0
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 605 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4:high PASS 100% 559 in → 84 out (72 reasoning)
Release version 1.0
openai/gpt-5.4:high__json_schema PASS 100% 603 in → 84 out (63 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4:low PASS 100% 559 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.4:low__json_schema PASS 100% 605 in → 31 out (10 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4:medium PASS 100% 557 in → 59 out (47 reasoning)
Release version 1.0
openai/gpt-5.4:medium__json_schema PASS 100% 595 in → 61 out (40 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4:none PASS 100% 561 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.4:none__json_schema PASS 100% 599 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.4:xhigh PASS 100% 561 in → 102 out (90 reasoning)
Release version 1.0
openai/gpt-5.4:xhigh__json_schema PASS 100% 599 in → 84 out (63 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.5:high PASS 100% 557 in → 61 out (49 reasoning)
Release version 1.0
openai/gpt-5.5:high__json_schema PASS 100% 603 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.5:low PASS 100% 571 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.5:low__json_schema PASS 100% 597 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.5:medium PASS 100% 563 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.5:medium__json_schema PASS 100% 601 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.5:none PASS 100% 563 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.5:none__json_schema PASS 100% 605 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.5:xhigh PASS 100% 551 in → 83 out (71 reasoning)
Release version 1.0
openai/gpt-5.5:xhigh__json_schema PASS 100% 599 in → 85 out (64 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-luna:high PASS 100% 561 in → 43 out (31 reasoning)
Release version 1.0
openai/gpt-5.6-luna:high__json_schema PASS 100% 599 in → 57 out (36 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-luna:low PASS 100% 547 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-luna:low__json_schema PASS 100% 595 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-luna:max PASS 100% 555 in → 79 out (67 reasoning)
Release version 1.0
openai/gpt-5.6-luna:max__json_schema PASS 100% 591 in → 60 out (39 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-luna:medium PASS 100% 559 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 605 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-luna:none PASS 100% 565 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-luna:none__json_schema PASS 100% 595 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-luna:xhigh PASS 100% 549 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 593 in → 110 out (89 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-sol:high PASS 100% 569 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-sol:high__json_schema PASS 100% 593 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-sol:low PASS 100% 557 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-sol:low__json_schema PASS 100% 613 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-sol:max PASS 100% 561 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-sol:max__json_schema PASS 100% 597 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-sol:medium PASS 100% 547 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-sol:medium__json_schema PASS 100% 611 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-sol:none PASS 100% 567 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-sol:none__json_schema PASS 100% 599 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-sol:xhigh PASS 100% 563 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 599 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-terra:high PASS 100% 565 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-terra:high__json_schema PASS 100% 607 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-terra:low PASS 100% 557 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-terra:low__json_schema PASS 100% 591 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-terra:max PASS 100% 553 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-terra:max__json_schema PASS 100% 597 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-terra:medium PASS 100% 561 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 607 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-terra:none PASS 100% 551 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-terra:none__json_schema PASS 100% 599 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-5.6-terra:xhigh PASS 100% 547 in → 10 out (0 reasoning)
Release version 1.0
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 597 in → 19 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-oss-120b:high PASS 100% 612 in → 302 out (286 reasoning)
Release version 1.0
openai/gpt-oss-120b:high__json_schema PASS 100% 724 in → 242 out (218 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-oss-120b:low PASS 100% 622 in → 87 out (81 reasoning)
Release version 1.0
openai/gpt-oss-120b:low__json_schema PASS 100% 609 in → 51 out (21 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
openai/gpt-oss-120b:medium PASS 100% 626 in → 103 out (93 reasoning)
Release version 1.0
openai/gpt-oss-120b:medium__json_schema PASS 100% 614 in → 184 out (161 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
openai/gpt-oss-120b:xhigh PASS 100% 633 in → 211 out (0 reasoning)
Release version 1.0
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 624 in → 283 out (284 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
openai/gpt-oss-20b:high PASS 100% 606 in → 226 out (187 reasoning)
Release version 1.0
openai/gpt-oss-20b:high__json_schema PASS 100% 628 in → 300 out (278 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
openai/gpt-oss-20b:low PASS 100% 620 in → 365 out (359 reasoning)
Release version 1.0
openai/gpt-oss-20b:low__json_schema PASS 100% 622 in → 33 out (11 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-oss-20b:medium PASS 100% 620 in → 191 out (160 reasoning)
Release version 1.0
openai/gpt-oss-20b:medium__json_schema PASS 100% 616 in → 134 out (112 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
openai/gpt-oss-20b:xhigh PASS 100% 622 in → 409 out (399 reasoning)
Release version 1.0
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 621 in → 233 out (211 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
poolside/laguna-m.1:high PASS 100% 678 in → 352 out (343 reasoning)
Release version 1.0
poolside/laguna-m.1:high__json_schema PASS 100% 684 in → 267 out (253 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
poolside/laguna-m.1:low PASS 100% 684 in → 406 out (397 reasoning)
Release version 1.0
poolside/laguna-m.1:low__json_schema PASS 100% 696 in → 300 out (286 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
poolside/laguna-m.1:medium PASS 100% 674 in → 418 out (409 reasoning)
Release version 1.0
poolside/laguna-m.1:medium__json_schema PASS 100% 690 in → 354 out (340 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
poolside/laguna-m.1:none PASS 100% 692 in → 8 out (0 reasoning)
Release version 1.0
poolside/laguna-m.1:none__json_schema PASS 100% 686 in → 13 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
poolside/laguna-m.1:xhigh PASS 100% 696 in → 419 out (410 reasoning)
Release version 1.0
poolside/laguna-m.1:xhigh__json_schema PASS 100% 680 in → 482 out (468 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
poolside/laguna-xs-2.1:high PASS 100% 702 in → 38 out (31 reasoning)
Release version 1.0
poolside/laguna-xs-2.1:high__json_schema PASS 100% 696 in → 329 out (310 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
poolside/laguna-xs-2.1:low PASS 100% 694 in → 424 out (417 reasoning)
Release version 1.0
poolside/laguna-xs-2.1:low__json_schema PASS 100% 688 in → 348 out (334 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
poolside/laguna-xs-2.1:medium PASS 100% 702 in → 363 out (356 reasoning)
Release version 1.0
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 692 in → 321 out (302 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
poolside/laguna-xs-2.1:none PASS 100% 694 in → 6 out (0 reasoning)
Release version 1.0
poolside/laguna-xs-2.1:none__json_schema PASS 100% 690 in → 18 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
poolside/laguna-xs-2.1:xhigh PASS 100% 692 in → 203 out (196 reasoning)
Release version 1.0
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 698 in → 282 out (263 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
poolside/laguna-xs.2:high PASS 100% 733 in → 136 out (127 reasoning)
Release version 1.0
poolside/laguna-xs.2:high__json_schema PASS 100% 727 in → 176 out (162 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
poolside/laguna-xs.2:low PASS 100% 725 in → 179 out (170 reasoning)
Release version 1.0
poolside/laguna-xs.2:low__json_schema PASS 100% 725 in → 188 out (174 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
poolside/laguna-xs.2:medium PASS 100% 739 in → 214 out (205 reasoning)
Release version 1.0
poolside/laguna-xs.2:medium__json_schema PASS 100% 725 in → 334 out (320 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
poolside/laguna-xs.2:none PASS 100% 731 in → 8 out (0 reasoning)
Release version 1.0
poolside/laguna-xs.2:none__json_schema PASS 100% 731 in → 13 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
poolside/laguna-xs.2:xhigh PASS 100% 723 in → 158 out (149 reasoning)
Release version 1.0
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 731 in → 273 out (254 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
qwen/qwen3.6-27b:high PASS 100% 699 in → 561 out (473 reasoning)
Release version 1.0
qwen/qwen3.6-27b:high__json_schema PASS 100% 687 in → 290 out (209 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
qwen/qwen3.6-27b:low PASS 100% 695 in → 415 out (309 reasoning)
Release version 1.0
qwen/qwen3.6-27b:low__json_schema PASS 100% 685 in → 465 out (401 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
qwen/qwen3.6-27b:medium PASS 100% 689 in → 363 out (308 reasoning)
Release version 1.0
qwen/qwen3.6-27b:medium__json_schema PASS 100% 691 in → 295 out (236 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
qwen/qwen3.6-27b:none PASS 100% 683 in → 6 out (0 reasoning)
Release version 1.0
qwen/qwen3.6-27b:none__json_schema PASS 100% 695 in → 18 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
qwen/qwen3.6-27b:xhigh PASS 100% 687 in → 373 out (316 reasoning)
Release version 1.0
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 675 in → 767 out (659 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
qwen/qwen3.6-35b-a3b:low PASS 100% 685 in → 521 out (439 reasoning)
Release version 1.0
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 687 in → 347 out (291 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 695 in → 830 out (819 reasoning)
Release version 1.0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 695 in → 357 out (289 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
qwen/qwen3.6-35b-a3b:none PASS 100% 691 in → 7 out (0 reasoning)
Release version 1.0
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 699 in → 13 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 683 in → 830 out (727 reasoning)
Release version 1.0
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 693 in → 247 out (181 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
qwen/qwen3.6-flash:high PASS 100% 693 in → 246 out (235 reasoning)
Release version 1.0
qwen/qwen3.6-flash:low PASS 100% 695 in → 414 out (403 reasoning)
Release version 1.0
qwen/qwen3.6-flash:medium PASS 100% 701 in → 250 out (239 reasoning)
Release version 1.0
qwen/qwen3.6-flash:none PASS 100% 701 in → 6 out (0 reasoning)
Release version 1.0
qwen/qwen3.6-flash:xhigh PASS 100% 697 in → 430 out (419 reasoning)
Release version 1.0
qwen/qwen3.7-flash:high PASS 100% 679 in → 334 out (323 reasoning)
Release version 1.0
qwen/qwen3.7-flash:low PASS 100% 699 in → 302 out (291 reasoning)
Release version 1.0
qwen/qwen3.7-flash:max PASS 100% 689 in → 268 out (257 reasoning)
Release version 1.0
qwen/qwen3.7-flash:medium PASS 100% 689 in → 597 out (586 reasoning)
Release version 1.0
qwen/qwen3.7-flash:none PASS 100% 699 in → 6 out (0 reasoning)
Release version 1.0
qwen/qwen3.7-flash:xhigh PASS 100% 699 in → 409 out (398 reasoning)
Release version 1.0
qwen/qwen3.7-max:high PASS 100% 687 in → 261 out (250 reasoning)
Release version 1.0
qwen/qwen3.7-max:low PASS 100% 691 in → 179 out (168 reasoning)
Release version 1.0
qwen/qwen3.7-max:medium PASS 100% 697 in → 220 out (209 reasoning)
Release version 1.0
qwen/qwen3.7-max:none PASS 100% 691 in → 6 out (0 reasoning)
Release version 1.0
qwen/qwen3.7-max:xhigh PASS 100% 695 in → 311 out (300 reasoning)
Release version 1.0
qwen/qwen3.7-plus:high PASS 100% 689 in → 232 out (221 reasoning)
Release version 1.0
qwen/qwen3.7-plus:low PASS 100% 691 in → 279 out (268 reasoning)
Release version 1.0
qwen/qwen3.7-plus:medium PASS 100% 685 in → 165 out (154 reasoning)
Release version 1.0
qwen/qwen3.7-plus:none PASS 100% 683 in → 6 out (0 reasoning)
Release version 1.0
qwen/qwen3.7-plus:xhigh PASS 100% 689 in → 175 out (164 reasoning)
Release version 1.0
tencent/hy3:high PASS 100% 575 in → 239 out (231 reasoning)
Release version 1.0
tencent/hy3:high__json_schema PASS 100% 583 in → 325 out (261 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
tencent/hy3:low PASS 100% 565 in → 132 out (124 reasoning)
Release version 1.0
tencent/hy3:low__json_schema PASS 100% 563 in → 213 out (182 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
tencent/hy3:medium PASS 100% 569 in → 312 out (304 reasoning)
Release version 1.0
tencent/hy3:medium__json_schema PASS 100% 569 in → 273 out (211 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
tencent/hy3:none PASS 100% 576 in → 7 out (0 reasoning)
Release version 1.0
tencent/hy3:xhigh PASS 100% 559 in → 294 out (286 reasoning)
Release version 1.0
tencent/hy3:xhigh__json_schema PASS 100% 573 in → 256 out (228 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
thinkingmachines/inkling-small:high PASS 100% 566 in → 93 out (81 reasoning)
Release version 1.0
thinkingmachines/inkling-small:high__json_schema PASS 100% 566 in → 48 out (28 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
thinkingmachines/inkling-small:low PASS 100% 568 in → 22 out (10 reasoning)
Release version 1.0
thinkingmachines/inkling-small:low__json_schema PASS 100% 572 in → 27 out (10 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
thinkingmachines/inkling-small:max PASS 100% 576 in → 52 out (40 reasoning)
Release version 1.0
thinkingmachines/inkling-small:max__json_schema PASS 100% 564 in → 55 out (35 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
thinkingmachines/inkling-small:medium PASS 100% 564 in → 27 out (15 reasoning)
Release version 1.0
thinkingmachines/inkling-small:medium__json_schema PASS 100% 566 in → 54 out (35 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
thinkingmachines/inkling-small:xhigh PASS 100% 566 in → 51 out (39 reasoning)
Release version 1.0
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 570 in → 94 out (75 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
thinkingmachines/inkling:high PASS 100% 566 in → 64 out (47 reasoning)
Release version 1.0
thinkingmachines/inkling:low PASS 100% 582 in → 42 out (28 reasoning)
Release version 1.0
thinkingmachines/inkling:low__json_schema PASS 100% 568 in → 64 out (45 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
thinkingmachines/inkling:max PASS 100% 566 in → 115 out (93 reasoning)
Release version 1.0
thinkingmachines/inkling:max__json_schema PASS 100% 572 in → 284 out (265 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
thinkingmachines/inkling:medium PASS 100% 558 in → 252 out (239 reasoning)
Release version 1.0
thinkingmachines/inkling:none PASS 100% 568 in → 10 out (0 reasoning)
Release version 1.0
thinkingmachines/inkling:xhigh PASS 100% 570 in → 304 out (292 reasoning)
Release version 1.0
thinkingmachines/inkling:xhigh__json_schema PASS 100% 568 in → 98 out (79 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
x-ai/grok-4.3:high PASS 100% 724 in → 343 out (337 reasoning)
Release version 1.0
x-ai/grok-4.3:high__json_schema PASS 100% 786 in → 400 out (389 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
x-ai/grok-4.3:low PASS 100% 736 in → 255 out (249 reasoning)
Release version 1.0
x-ai/grok-4.3:low__json_schema PASS 100% 796 in → 279 out (267 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
x-ai/grok-4.3:max PASS 100% 724 in → 244 out (238 reasoning)
Release version 1.0
x-ai/grok-4.3:max__json_schema PASS 100% 784 in → 375 out (360 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
x-ai/grok-4.3:medium PASS 100% 734 in → 473 out (467 reasoning)
Release version 1.0
x-ai/grok-4.3:medium__json_schema PASS 100% 804 in → 542 out (531 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
x-ai/grok-4.3:none PASS 100% 726 in → 6 out (0 reasoning)
Release version 1.0
x-ai/grok-4.3:none__json_schema PASS 100% 786 in → 12 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
x-ai/grok-4.3:xhigh PASS 100% 724 in → 312 out (306 reasoning)
Release version 1.0
x-ai/grok-4.3:xhigh__json_schema PASS 100% 782 in → 513 out (502 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
x-ai/grok-4.5:high PASS 100% 788 in → 201 out (195 reasoning)
Release version 1.0
x-ai/grok-4.5:high__json_schema PASS 100% 866 in → 246 out (233 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
x-ai/grok-4.5:low PASS 100% 796 in → 154 out (148 reasoning)
Release version 1.0
x-ai/grok-4.5:low__json_schema PASS 100% 862 in → 139 out (126 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
x-ai/grok-4.5:max PASS 100% 788 in → 216 out (210 reasoning)
Release version 1.0
x-ai/grok-4.5:max__json_schema PASS 100% 854 in → 174 out (161 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
x-ai/grok-4.5:medium PASS 100% 788 in → 256 out (250 reasoning)
Release version 1.0
x-ai/grok-4.5:medium__json_schema PASS 100% 864 in → 178 out (165 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
x-ai/grok-4.5:xhigh PASS 100% 800 in → 156 out (150 reasoning)
Release version 1.0
x-ai/grok-4.5:xhigh__json_schema PASS 100% 860 in → 241 out (228 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0"}
z-ai/glm-4.7-flash:high PASS 100% 574 in → 384 out (322 reasoning)
Release version 1.0
z-ai/glm-4.7-flash:low PASS 100% 568 in → 384 out (322 reasoning)
Release version 1.0
z-ai/glm-4.7-flash:medium PASS 100% 574 in → 287 out (235 reasoning)
Release version 1.0
z-ai/glm-4.7-flash:none PASS 100% 576 in → 6 out (0 reasoning)
Release version 1.0
z-ai/glm-4.7-flash:none__json_schema PASS 100% 574 in → 13 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) {"commit_message": "Release version 1.0"}
z-ai/glm-4.7-flash:xhigh PASS 100% 580 in → 820 out (792 reasoning)
Release version 1.0
z-ai/glm-5.2:high PASS 100% 589 in → 271 out (235 reasoning)
Release version 1.0
z-ai/glm-5.2:high__json_schema PASS 100% 768 in → 179 out (161 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
z-ai/glm-5.2:low PASS 100% 585 in → 141 out (134 reasoning)
Release version 1.0
z-ai/glm-5.2:low__json_schema PASS 100% 587 in → 366 out (348 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
z-ai/glm-5.2:medium PASS 100% 577 in → 141 out (111 reasoning)
Release version 1.0
z-ai/glm-5.2:none PASS 100% 567 in → 7 out (0 reasoning)
Release version 1.0
z-ai/glm-5.2:none__json_schema PASS 100% 581 in → 15 out (0 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
z-ai/glm-5.2:xhigh PASS 100% 583 in → 326 out (318 reasoning)
Release version 1.0
z-ai/glm-5.2:xhigh__json_schema PASS 100% 579 in → 319 out (261 reasoning)
Release version 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Release version 1.0" }
anthropic/claude-fable-5:medium FAIL 0% 762 in → 5 out (0 reasoning)
(empty output)
Failure: Expected 'Release version 1.0', got ''
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:xhigh FAIL 0% 571 in → 347 out (312 reasoning)
Releaseversion 1.0
Failure: Expected 'Release version 1.0', got ' Releaseversion 1.0'
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 539 in → 289 out (268 reasoning)
Releaseversion 1.0
JSON Schema Structured Output
(raw) { "commit_message": "Releaseversion 1.0" }
Failure: Expected 'Release version 1.0', got 'Releaseversion 1.0'
deepseek/deepseek-v4-flash:high FAIL 0% 563 in → 93 out (69 reasoning)
Releaseversion 1.0
Failure: Expected 'Release version 1.0', got 'Releaseversion 1.0'
deepseek/deepseek-v4-flash:high__json_schema FAIL 0%
Invalid JSON. Output: Releaseversion 1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0%
Invalid JSON. Output: Release version 1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: Release version 1.1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 693 in → 99 out (0 reasoning)
Release version 1.0 --- app.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/app.txt b/app.txt new file mode 100644 index 0000000..17f1424 --- /dev/null +++ b/app.txt @@ -0,0 +1 @@ +v1 content
JSON Schema Structured Output
(raw) {"commit_message":"Release version 1.0\n---\n app.txt | 1 +\n 1 file changed, 1 insertion(+)\n\n diff --git a/app.txt b/app.txt\n new file mode 100644\n index 0000000..17f1424\n --- /dev/null\n +++ b/app.txt\n @@ -0,0 +1 @@\n +v1 content"}
Failure: Expected 'Release version 1.0', got 'Release version 1.0 --- app.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/app.txt b/app.txt new file mode 100644 index 0000000..17f1424 --- /dev/null +++ b/app.txt @@ -0,0 +1 @@ +v1 content'
qwen/qwen3.6-27b:xhigh__json_schema FAIL 0% 689 in → 82 out (0 reasoning)
The lightweight tag 'v1.0.0' points to the commit with hash `0f56403ff007c11d00a23bea18f7f24f7c3a97fb`. The commit message for this commit is "Release version 1.0".
JSON Schema Structured Output
(raw) { "commit_message": "The lightweight tag 'v1.0.0' points to the commit with hash `0f56403ff007c11d00a23bea18f7f24f7c3a97fb`. The commit message for this commit is \"Release version 1.0\"." }
Failure: Expected 'Release version 1.0', got 'The lightweight tag 'v1.0.0' points to the commit with hash `0f56403ff007c11d00a23bea18f7f24f7c3a97fb`. The commit message for this commit is "Release version 1.0".'
qwen/qwen3.6-35b-a3b:high FAIL 0% 689 in → 212 out (154 reasoning)
Release version1.0
Failure: Expected 'Release version 1.0', got ' Release version1.0'
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-14a84078-b6d8-9438-9586-ed4ba266244d","request_id":"14a84078-b6d8-9438-9586-ed4ba266244d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d71cc986-918e-9a85-9ae4-6889ad5d9514","request_id":"d71cc986-918e-9a85-9ae4-6889ad5d9514"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b84e87e8-1c6b-9dbf-8092-55f55ce85176","request_id":"b84e87e8-1c6b-9dbf-8092-55f55ce85176"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e4cccadd-9c3a-9ab6-ac6e-21af6fd705d4","request_id":"e4cccadd-9c3a-9ab6-ac6e-21af6fd705d4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-aa3f5f7a-03f6-9844-9718-1d9c93a5096b","request_id":"aa3f5f7a-03f6-9844-9718-1d9c93a5096b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ca1c6b0c-7721-9297-b656-29161a374ec9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-df5012cb-2f7a-9f5f-8562-81f89220542b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9e815dc4-3a6d-9fd9-9a2f-f54cb2404832"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-650b34d6-ffeb-9699-b3ac-1ddeb40171a4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9d51577d-46fd-993e-9f4b-ccb9856bbfb8"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-501fa00a-6e2c-9422-8640-94a38e563787"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c2aed656-2788-95dc-9a94-70d86dcf3903","request_id":"c2aed656-2788-95dc-9a94-70d86dcf3903"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-18a70995-6aa1-9da6-8992-e8bce0a594c3","request_id":"18a70995-6aa1-9da6-8992-e8bce0a594c3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bd03588d-50f8-979d-a6fb-1a9d602385ee","request_id":"bd03588d-50f8-979d-a6fb-1a9d602385ee"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-702241f8-8b5a-94af-8f13-786e9d8a0014","request_id":"702241f8-8b5a-94af-8f13-786e9d8a0014"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-36f29090-2704-93ef-acad-5202b49b746e","request_id":"36f29090-2704-93ef-acad-5202b49b746e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cfc2af89-f86e-9fd9-a97f-ca52c03832f1","request_id":"cfc2af89-f86e-9fd9-a97f-ca52c03832f1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-91bb88c5-87fc-9e9c-b0ad-1890614e4198","request_id":"91bb88c5-87fc-9e9c-b0ad-1890614e4198"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f0d71cf8-2072-939b-aae1-5decf11e1f67","request_id":"f0d71cf8-2072-939b-aae1-5decf11e1f67"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-515214b7-6017-9560-ac29-11105649e83a","request_id":"515214b7-6017-9560-ac29-11105649e83a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8a18d59f-3569-91f9-b7f3-aaf2e4b7f379","request_id":"8a18d59f-3569-91f9-b7f3-aaf2e4b7f379"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Model 'tencent/hy3:none' violated the reasoning_level=none invariant: provider reported 169 reasoning token(s)
thinkingmachines/inkling-small:none FAIL 0% 572 in → 9 out (0 reasoning)
Release version 1.0<|end_message|>
Failure: Expected 'Release version 1.0', got 'Release version 1.0<|end_message|>'
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: Release version 1.0{ "commit_message": "Release version 1.0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: Release version 1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: Release version 1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: Release version 1.0{"commit_message": "Release version 1.0"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output: Release version 1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:medium__json_schema FAIL 0%
Invalid JSON. Output: Release version 1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)