f005 — git_log_format

← git_log_format
List commits in a date range
Tests ability to list commits in a date range. Evaluates time-based filtering with git log.
medium git-log date-range filter time
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 GIT_AUTHOR_DATE='2025-01-01T12:00:00' GIT_COMMITTER_DATE='2025-01-01T12:00:00' bash -c 'echo a > f.txt && git add f.txt && git commit -m "January commit"'
  5. 05 GIT_AUTHOR_DATE='2025-02-01T12:00:00' GIT_COMMITTER_DATE='2025-02-01T12:00:00' bash -c 'echo b > f.txt && git add f.txt && git commit -m "February commit"'
  6. 06 GIT_AUTHOR_DATE='2025-03-01T12:00:00' GIT_COMMITTER_DATE='2025-03-01T12:00:00' bash -c 'echo c > f.txt && git add f.txt && git commit -m "March commit"'
  7. 07 GIT_AUTHOR_DATE='2025-04-01T12:00:00' GIT_COMMITTER_DATE='2025-04-01T12:00:00' bash -c 'echo d > f.txt && git add f.txt && git commit -m "April commit"'
Prompt
List all commits made between 2025-02-01 and 2025-03-31 (inclusive). Output ONLY the commit messages, one per line, nothing else.
Expected
February commit March commit
Fixture Metrics

Scoped model quality, cost, API time, and token usage for git_log_format/f005.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (760)
anthropic/claude-fable-5.1:high PASS 100% 762 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-fable-5.1:high__json_schema PASS 100% 1,002 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
anthropic/claude-fable-5.1:low PASS 100% 762 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-fable-5.1:low__json_schema PASS 100% 1,002 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-fable-5.1:max PASS 100% 762 in → 166 out (157 reasoning)
March commit February commit
anthropic/claude-fable-5.1:max__json_schema PASS 100% 1,002 in → 97 out (75 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-fable-5.1:medium PASS 100% 762 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-fable-5.1:medium__json_schema PASS 100% 1,002 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-fable-5.1:xhigh PASS 100% 762 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-fable-5.1:xhigh__json_schema PASS 100% 1,002 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
anthropic/claude-fable-5:high PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-fable-5:high__json_schema PASS 100% 1,000 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
anthropic/claude-fable-5:low PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-fable-5:low__json_schema PASS 100% 1,000 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
anthropic/claude-fable-5:max PASS 100% 760 in → 189 out (58 reasoning)
March commit February commit
anthropic/claude-fable-5:max__json_schema PASS 100% 1,000 in → 199 out (50 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
anthropic/claude-fable-5:medium PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-fable-5:medium__json_schema PASS 100% 1,000 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
anthropic/claude-fable-5:xhigh PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 1,000 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
anthropic/claude-haiku-4.5:high PASS 100% 676 in → 313 out (197 reasoning)
February commit March commit
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 866 in → 269 out (191 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
anthropic/claude-haiku-4.5:low PASS 100% 676 in → 274 out (188 reasoning)
February commit March commit
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 866 in → 657 out (552 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
anthropic/claude-haiku-4.5:medium PASS 100% 676 in → 259 out (187 reasoning)
February commit March commit
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 866 in → 729 out (545 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
anthropic/claude-haiku-4.5:none PASS 100% 647 in → 8 out (0 reasoning)
February commit March commit
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 836 in → 19 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 676 in → 266 out (175 reasoning)
March commit February commit
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 866 in → 565 out (412 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
anthropic/claude-opus-4.6:high PASS 100% 647 in → 8 out (0 reasoning)
February commit March commit
anthropic/claude-opus-4.6:high__json_schema PASS 100% 837 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.6:low PASS 100% 647 in → 8 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.6:low__json_schema PASS 100% 837 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.6:max PASS 100% 647 in → 159 out (103 reasoning)
March commit February commit
anthropic/claude-opus-4.6:max__json_schema PASS 100% 837 in → 140 out (89 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.6:medium PASS 100% 647 in → 8 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 837 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.6:none PASS 100% 647 in → 8 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.6:none__json_schema PASS 100% 837 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 647 in → 8 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 837 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.7:high PASS 100% 765 in → 11 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.7:high__json_schema PASS 100% 1,005 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.7:low PASS 100% 765 in → 11 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.7:low__json_schema PASS 100% 1,005 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.7:max PASS 100% 765 in → 172 out (63 reasoning)
March commit February commit
anthropic/claude-opus-4.7:max__json_schema PASS 100% 1,005 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.7:medium PASS 100% 765 in → 11 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 1,005 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.7:none PASS 100% 765 in → 11 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.7:none__json_schema PASS 100% 1,005 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 765 in → 11 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 1,005 in → 24 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.8:high PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.8:high__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.8:low PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.8:low__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.8:max PASS 100% 760 in → 161 out (60 reasoning)
March commit February commit
anthropic/claude-opus-4.8:max__json_schema PASS 100% 1,000 in → 156 out (64 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
anthropic/claude-opus-4.8:medium PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.8:none PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.8:none__json_schema PASS 100% 1,000 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-5.5:high PASS 100% 762 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-5.5:high__json_schema PASS 100% 1,002 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-5.5:low PASS 100% 762 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-5.5:low__json_schema PASS 100% 1,002 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-5.5:max PASS 100% 762 in → 197 out (188 reasoning)
March commit February commit
anthropic/claude-opus-5.5:max__json_schema PASS 100% 1,002 in → 217 out (195 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-5.5:medium PASS 100% 762 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-5.5:medium__json_schema PASS 100% 1,002 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-5.5:xhigh PASS 100% 762 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-5.5:xhigh__json_schema PASS 100% 1,002 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-5:high PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-5:high__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-5:low PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-5:low__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-5:max PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-5:max__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-5:medium PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-5:medium__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-opus-5:xhigh PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-opus-5:xhigh__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-4.6:high PASS 100% 647 in → 8 out (0 reasoning)
March commit February commit
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 837 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-4.6:low PASS 100% 647 in → 8 out (0 reasoning)
March commit February commit
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 837 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-4.6:max PASS 100% 647 in → 135 out (93 reasoning)
March commit February commit
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 837 in → 170 out (105 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 647 in → 8 out (0 reasoning)
March commit February commit
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 837 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-4.6:none PASS 100% 647 in → 8 out (0 reasoning)
March commit February commit
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 837 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 647 in → 8 out (0 reasoning)
March commit February commit
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 837 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-5:high PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-sonnet-5:high__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-5:low PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-sonnet-5:low__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-5:medium PASS 100% 760 in → 22 out (10 reasoning)
March commit February commit
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-5:none PASS 100% 760 in → 9 out (0 reasoning)
February commit March commit
anthropic/claude-sonnet-5:none__json_schema PASS 100% 1,000 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 760 in → 9 out (0 reasoning)
March commit February commit
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 1,000 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
arcee-ai/trinity-large-thinking:high PASS 100% 564 in → 818 out (818 reasoning)
February commit March commit
arcee-ai/trinity-large-thinking:low PASS 100% 564 in → 875 out (867 reasoning)
February commit March commit
arcee-ai/trinity-large-thinking:medium PASS 100% 564 in → 1,638 out (1,636 reasoning)
February commit March commit
arcee-ai/trinity-large-thinking:xhigh PASS 100% 564 in → 855 out (848 reasoning)
February commit March commit
arcee-ai/trinity-mini:high__json_schema PASS 100% 564 in → 335 out (298 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
arcee-ai/trinity-mini:low PASS 100% 564 in → 341 out (343 reasoning)
March commit February commit
arcee-ai/trinity-mini:low__json_schema PASS 100% 564 in → 376 out (365 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
arcee-ai/trinity-mini:medium PASS 100% 564 in → 448 out (487 reasoning)
February commit March commit
arcee-ai/trinity-mini:medium__json_schema PASS 100% 564 in → 348 out (326 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
arcee-ai/trinity-mini:xhigh PASS 100% 564 in → 371 out (351 reasoning)
March commit February commit
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 564 in → 380 out (356 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 652 in → 415 out (409 reasoning)
March commit February commit
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 573 in → 123 out (82 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 573 in → 94 out (88 reasoning)
March commit February commit
deepseek/deepseek-v4-flash-0731:max PASS 100% 652 in → 2,973 out (2,969 reasoning)
March commit February commit
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 652 in → 232 out (202 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list":["February commit","March commit"] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 573 in → 91 out (76 reasoning)
February commit March commit
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 573 in → 99 out (95 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 573 in → 6 out (0 reasoning)
March commit February commit
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 573 in → 21 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 573 in → 83 out (67 reasoning)
March commit February commit
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 573 in → 147 out (145 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"]}
deepseek/deepseek-v4-flash:high PASS 100% 573 in → 109 out (91 reasoning)
February commit March commit
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 680 in → 15 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
deepseek/deepseek-v4-flash:low PASS 100% 573 in → 415 out (408 reasoning)
March commit February commit
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 575 in → 133 out (109 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
deepseek/deepseek-v4-flash:medium PASS 100% 573 in → 153 out (125 reasoning)
February commit March commit
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 573 in → 129 out (112 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
deepseek/deepseek-v4-flash:none PASS 100% 573 in → 6 out (0 reasoning)
February commit March commit
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 573 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 652 in → 598 out (587 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
deepseek/deepseek-v4-pro:high PASS 100% 573 in → 573 out (566 reasoning)
February commit March commit
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 773 in → 459 out (443 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
deepseek/deepseek-v4-pro:low PASS 100% 573 in → 140 out (133 reasoning)
February commit March commit
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 573 in → 162 out (136 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
deepseek/deepseek-v4-pro:medium PASS 100% 573 in → 133 out (125 reasoning)
February commit March commit
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 773 in → 513 out (498 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
deepseek/deepseek-v4-pro:none PASS 100% 573 in → 6 out (0 reasoning)
February commit March commit
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 773 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 652 in → 943 out (937 reasoning)
February commit March commit
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 851 in → 518 out (502 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
deepseek/deepseek-v4.1-flash:high PASS 100% 599 in → 394 out (387 reasoning)
March commit February commit
deepseek/deepseek-v4.1-flash:high__json_schema PASS 100% 599 in → 141 out (111 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
deepseek/deepseek-v4.1-flash:low PASS 100% 599 in → 172 out (165 reasoning)
March commit February commit
deepseek/deepseek-v4.1-flash:low__json_schema PASS 100% 599 in → 139 out (115 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": [ "February commit", "March commit" ] }
deepseek/deepseek-v4.1-flash:max PASS 100% 599 in → 105 out (98 reasoning)
March commit February commit
deepseek/deepseek-v4.1-flash:max__json_schema PASS 100% 599 in → 187 out (163 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": [ "March commit", "February commit" ] }
deepseek/deepseek-v4.1-flash:medium PASS 100% 599 in → 96 out (89 reasoning)
March commit February commit
deepseek/deepseek-v4.1-flash:medium__json_schema PASS 100% 599 in → 193 out (165 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
deepseek/deepseek-v4.1-flash:none PASS 100% 573 in → 6 out (0 reasoning)
March commit February commit
deepseek/deepseek-v4.1-flash:none__json_schema PASS 100% 573 in → 20 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
deepseek/deepseek-v4.1-flash:xhigh PASS 100% 599 in → 104 out (97 reasoning)
March commit February commit
deepseek/deepseek-v4.1-flash:xhigh__json_schema PASS 100% 599 in → 159 out (136 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"] }
google/gemini-3-flash-preview:high PASS 100% 746 in → 468 out (463 reasoning)
March commit February commit
google/gemini-3-flash-preview:high__json_schema PASS 100% 747 in → 294 out (281 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3-flash-preview:low PASS 100% 746 in → 414 out (409 reasoning)
March commit February commit
google/gemini-3-flash-preview:low__json_schema PASS 100% 747 in → 354 out (341 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3-flash-preview:medium PASS 100% 746 in → 355 out (350 reasoning)
March commit February commit
google/gemini-3-flash-preview:medium__json_schema PASS 100% 828 in → 404 out (391 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3-flash-preview:xhigh PASS 100% 746 in → 258 out (253 reasoning)
March commit February commit
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 828 in → 506 out (493 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 746 in → 740 out (735 reasoning)
March commit February commit
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 828 in → 777 out (764 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 746 in → 140 out (135 reasoning)
March commit February commit
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 747 in → 154 out (139 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 746 in → 512 out (507 reasoning)
March commit February commit
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 828 in → 303 out (288 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 747 in → 500 out (495 reasoning)
February commit March commit
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 747 in → 598 out (585 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.1-pro-preview:high PASS 100% 746 in → 511 out (506 reasoning)
March commit February commit
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 828 in → 609 out (596 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.1-pro-preview:low PASS 100% 746 in → 365 out (360 reasoning)
March commit February commit
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 828 in → 138 out (125 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.1-pro-preview:medium PASS 100% 746 in → 335 out (330 reasoning)
March commit February commit
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 828 in → 488 out (475 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 746 in → 527 out (522 reasoning)
March commit February commit
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 828 in → 399 out (386 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.5-flash-lite:high PASS 100% 746 in → 841 out (836 reasoning)
March commit February commit
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 747 in → 657 out (644 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.5-flash-lite:low PASS 100% 746 in → 5 out (0 reasoning)
March commit February commit
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 747 in → 28 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
google/gemini-3.5-flash-lite:max PASS 100% 746 in → 669 out (664 reasoning)
March commit February commit
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 747 in → 810 out (797 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.5-flash-lite:medium PASS 100% 747 in → 365 out (360 reasoning)
March commit February commit
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 828 in → 470 out (457 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 746 in → 753 out (748 reasoning)
March commit February commit
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 747 in → 856 out (843 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.5-flash:high PASS 100% 746 in → 763 out (758 reasoning)
March commit February commit
google/gemini-3.5-flash:high__json_schema PASS 100% 828 in → 520 out (507 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.5-flash:low PASS 100% 746 in → 177 out (172 reasoning)
March commit February commit
google/gemini-3.5-flash:low__json_schema PASS 100% 828 in → 268 out (240 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
google/gemini-3.5-flash:medium PASS 100% 746 in → 476 out (471 reasoning)
March commit February commit
google/gemini-3.5-flash:medium__json_schema PASS 100% 828 in → 444 out (431 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.5-flash:xhigh PASS 100% 746 in → 546 out (541 reasoning)
March commit February commit
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 828 in → 745 out (732 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.6-flash:high PASS 100% 746 in → 613 out (608 reasoning)
March commit February commit
google/gemini-3.6-flash:high__json_schema PASS 100% 828 in → 687 out (674 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.6-flash:low PASS 100% 747 in → 316 out (311 reasoning)
March commit February commit
google/gemini-3.6-flash:low__json_schema PASS 100% 828 in → 313 out (300 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.6-flash:max PASS 100% 746 in → 810 out (805 reasoning)
March commit February commit
google/gemini-3.6-flash:max__json_schema PASS 100% 828 in → 715 out (702 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.6-flash:medium PASS 100% 747 in → 457 out (452 reasoning)
March commit February commit
google/gemini-3.6-flash:medium__json_schema PASS 100% 828 in → 685 out (672 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.6-flash:xhigh PASS 100% 747 in → 666 out (661 reasoning)
March commit February commit
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 828 in → 871 out (858 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.8-flash:high PASS 100% 746 in → 443 out (438 reasoning)
March commit February commit
google/gemini-3.8-flash:high__json_schema PASS 100% 828 in → 480 out (465 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
google/gemini-3.8-flash:low PASS 100% 746 in → 168 out (163 reasoning)
March commit February commit
google/gemini-3.8-flash:low__json_schema PASS 100% 828 in → 151 out (136 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
google/gemini-3.8-flash:max PASS 100% 746 in → 407 out (402 reasoning)
March commit February commit
google/gemini-3.8-flash:max__json_schema PASS 100% 828 in → 463 out (450 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.8-flash:medium PASS 100% 746 in → 206 out (201 reasoning)
March commit February commit
google/gemini-3.8-flash:medium__json_schema PASS 100% 828 in → 340 out (327 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
google/gemini-3.8-flash:xhigh PASS 100% 746 in → 474 out (469 reasoning)
March commit February commit
google/gemini-3.8-flash:xhigh__json_schema PASS 100% 828 in → 625 out (610 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
google/gemma-4-26b-a4b-it:high PASS 100% 762 in → 634 out (490 reasoning)
March commit February commit
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 762 in → 681 out (411 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 762 in → 786 out (481 reasoning)
March commit February commit
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 762 in → 720 out (474 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 762 in → 691 out (568 reasoning)
March commit February commit
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 762 in → 992 out (617 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 759 in → 6 out (0 reasoning)
March commit February commit
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 759 in → 25 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 762 in → 565 out (400 reasoning)
March commit February commit
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 761 in → 675 out (397 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
google/gemma-4-31b-it:high PASS 100% 762 in → 362 out (221 reasoning)
March commit February commit
google/gemma-4-31b-it:high__json_schema PASS 100% 762 in → 236 out (142 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
google/gemma-4-31b-it:low PASS 100% 762 in → 361 out (1 reasoning)
March commit February commit
google/gemma-4-31b-it:low__json_schema PASS 100% 762 in → 440 out (289 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
google/gemma-4-31b-it:medium PASS 100% 762 in → 268 out (177 reasoning)
March commit February commit
google/gemma-4-31b-it:medium__json_schema PASS 100% 761 in → 232 out (121 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
google/gemma-4-31b-it:none PASS 100% 759 in → 6 out (0 reasoning)
March commit February commit
google/gemma-4-31b-it:none__json_schema PASS 100% 759 in → 16 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
google/gemma-4-31b-it:xhigh PASS 100% 762 in → 243 out (1 reasoning)
March commit February commit
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 762 in → 343 out (213 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
ibm-granite/granite-4.1-8b FAIL 100% 606 in → 8 out
February commit March commit ```
Failure: Extra lines: ['```']
ibm-granite/granite-4.1-8b__json_schema FAIL 100% 606 in → 22 out
February commit March commit April commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit", "April commit"] }
Failure: Extra lines: ['April commit']
ibm-granite/granite-4.2-8b:high PASS 100% 613 in → 309 out (293 reasoning)
February commit March commit
ibm-granite/granite-4.2-8b:high__json_schema FAIL 100% 613 in → 35 out (0 reasoning)
February commit March commit April commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit", "April commit" ] }
Failure: Extra lines: ['April commit']
ibm-granite/granite-4.2-8b:max PASS 100% 613 in → 3,917 out (4,005 reasoning)
February commit March commit
ibm-granite/granite-4.2-8b:max__json_schema FAIL 100% 613 in → 25 out (0 reasoning)
February commit March commit April commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit", "April commit"] }
Failure: Extra lines: ['April commit']
ibm-granite/granite-4.2-8b:medium__json_schema FAIL 100% 621 in → 17 out (0 reasoning)
February commit March commit April commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit","April commit"]}
Failure: Extra lines: ['April commit']
ibm-granite/granite-4.2-8b:none__json_schema FAIL 100% 613 in → 25 out (0 reasoning)
February commit March commit April commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit", "April commit"] }
Failure: Extra lines: ['April commit']
ibm-granite/granite-4.2-8b:xhigh PASS 100% 613 in → 842 out (744 reasoning)
February commit March commit
ibm-granite/granite-4.2-8b:xhigh__json_schema FAIL 100% 613 in → 25 out (0 reasoning)
February commit March commit April commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit", "April commit"] }
Failure: Extra lines: ['April commit']
inception/mercury-2.5:high PASS 100% 695 in → 711 out (702 reasoning)
March commit February commit
inception/mercury-2.5:high__json_schema PASS 100% 914 in → 2,592 out (2,481 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
inception/mercury-2.5:low PASS 100% 693 in → 252 out (243 reasoning)
March commit February commit
inception/mercury-2.5:low__json_schema PASS 100% 912 in → 434 out (242 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
inception/mercury-2.5:max PASS 100% 726 in → 691 out (682 reasoning)
March commit February commit
inception/mercury-2.5:max__json_schema PASS 100% 939 in → 1,367 out (1,251 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
inception/mercury-2.5:medium PASS 100% 695 in → 654 out (642 reasoning)
March commit February commit
inception/mercury-2.5:medium__json_schema PASS 100% 939 in → 1,204 out (1,112 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
inception/mercury-2.5:none PASS 100% 715 in → 9 out (0 reasoning)
February commit March commit
inception/mercury-2.5:none__json_schema PASS 100% 916 in → 195 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
inception/mercury-2.5:xhigh PASS 100% 695 in → 647 out (638 reasoning)
March commit February commit
inception/mercury-2.5:xhigh__json_schema PASS 100% 914 in → 946 out (837 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
inclusionai/ling-2.6-flash PASS 100% 770 in → 6 out
February commit March commit
inclusionai/ling-2.6-flash__json_schema PASS 100% 770 in → 28 out
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
liquid/lfm-2-24b-a2b FAIL 100% 595 in → 10 out
April commit March commit February commit
Failure: Extra lines: ['April commit']
meta/muse-spark-1.1:high PASS 100% 566 in → 935 out (920 reasoning)
March commit February commit
meta/muse-spark-1.1:high__json_schema PASS 100% 566 in → 883 out (858 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
meta/muse-spark-1.1:low PASS 100% 566 in → 624 out (609 reasoning)
March commit February commit
meta/muse-spark-1.1:low__json_schema PASS 100% 566 in → 455 out (430 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
meta/muse-spark-1.1:max PASS 100% 566 in → 1,345 out (1,330 reasoning)
March commit February commit
meta/muse-spark-1.1:max__json_schema PASS 100% 566 in → 1,321 out (1,296 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
meta/muse-spark-1.1:medium PASS 100% 566 in → 550 out (535 reasoning)
March commit February commit
meta/muse-spark-1.1:medium__json_schema PASS 100% 566 in → 695 out (672 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
meta/muse-spark-1.1:xhigh PASS 100% 566 in → 1,539 out (1,524 reasoning)
March commit February commit
meta/muse-spark-1.1:xhigh__json_schema PASS 100% 566 in → 937 out (914 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
meta/muse-spark-1.3-contributor:high PASS 100% 566 in → 543 out (511 reasoning)
March commit February commit
meta/muse-spark-1.3-contributor:high__json_schema PASS 100% 566 in → 1,077 out (1,012 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
meta/muse-spark-1.3-contributor:low PASS 100% 566 in → 232 out (217 reasoning)
March commit February commit
meta/muse-spark-1.3-contributor:low__json_schema PASS 100% 566 in → 499 out (457 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
meta/muse-spark-1.3-contributor:medium PASS 100% 566 in → 437 out (405 reasoning)
March commit February commit
meta/muse-spark-1.3-contributor:medium__json_schema PASS 100% 566 in → 664 out (600 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
meta/muse-spark-1.3-contributor:xhigh PASS 100% 566 in → 599 out (568 reasoning)
March commit February commit
meta/muse-spark-1.3-contributor:xhigh__json_schema PASS 100% 566 in → 1,000 out (939 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
minimax/minimax-m2.5:high PASS 100% 599 in → 905 out (845 reasoning)
March commit February commit
minimax/minimax-m2.5:high__json_schema PASS 100% 639 in → 186 out (168 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
minimax/minimax-m2.5:low PASS 100% 599 in → 587 out (582 reasoning)
February commit March commit
minimax/minimax-m2.5:low__json_schema PASS 100% 599 in → 1,405 out (1,622 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
minimax/minimax-m2.5:medium PASS 100% 599 in → 608 out (688 reasoning)
March commit February commit
minimax/minimax-m2.5:medium__json_schema PASS 100% 599 in → 286 out (262 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
minimax/minimax-m2.5:xhigh PASS 100% 599 in → 641 out (675 reasoning)
February commit March commit
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 599 in → 497 out (475 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
minimax/minimax-m2.7:high PASS 100% 599 in → 325 out (317 reasoning)
March commit February commit
minimax/minimax-m2.7:low PASS 100% 599 in → 946 out (939 reasoning)
March commit February commit
minimax/minimax-m2.7:low__json_schema PASS 100% 750 in → 561 out (538 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
minimax/minimax-m2.7:medium PASS 100% 599 in → 1,195 out (1,373 reasoning)
March commit February commit
minimax/minimax-m2.7:medium__json_schema PASS 100% 750 in → 754 out (740 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
minimax/minimax-m2.7:xhigh PASS 100% 599 in → 219 out (211 reasoning)
February commit March commit
minimax/minimax-m3:high PASS 100% 737 in → 168 out (131 reasoning)
March commit February commit
minimax/minimax-m3:high__json_schema PASS 100% 724 in → 23 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
minimax/minimax-m3:low PASS 100% 737 in → 152 out (112 reasoning)
March commit February commit
minimax/minimax-m3:low__json_schema PASS 100% 724 in → 22 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
minimax/minimax-m3:medium PASS 100% 737 in → 130 out (102 reasoning)
March commit February commit
minimax/minimax-m3:medium__json_schema PASS 100% 724 in → 15 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
minimax/minimax-m3:xhigh PASS 100% 737 in → 93 out (71 reasoning)
March commit February commit
minimax/minimax-m3:xhigh__json_schema PASS 100% 724 in → 17 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
mistralai/devstral-2512 PASS 100% 735 in → 6 out
March commit February commit
mistralai/devstral-2512__json_schema PASS 100% 735 in → 22 out
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
mistralai/mistral-medium-3-5:high PASS 100% 747 in → 654 out (453 reasoning)
February commit March commit
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 735 in → 3,189 out (2,749 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
mistralai/mistral-medium-3-5:low PASS 100% 747 in → 5,962 out (5,537 reasoning)
February commit March commit
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 735 in → 349 out (245 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
mistralai/mistral-medium-3-5:medium PASS 100% 747 in → 1,996 out (1,887 reasoning)
February commit March commit
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 735 in → 3,547 out (2,929 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
mistralai/mistral-medium-3-5:none PASS 100% 747 in → 6 out (0 reasoning)
March commit February commit
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 735 in → 15 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 747 in → 1,088 out (756 reasoning)
February commit March commit
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 735 in → 2,478 out (2,117 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
moonshotai/kimi-k2.6:high PASS 100% 564 in → 554 out (549 reasoning)
March commit February commit
moonshotai/kimi-k2.6:high__json_schema PASS 100% 563 in → 3,037 out (3,175 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
moonshotai/kimi-k2.6:low PASS 100% 564 in → 2,026 out (2,019 reasoning)
March commit February commit
moonshotai/kimi-k2.6:low__json_schema PASS 100% 563 in → 602 out (601 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
moonshotai/kimi-k2.6:medium PASS 100% 564 in → 601 out (594 reasoning)
March commit February commit
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 564 in → 510 out (526 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
moonshotai/kimi-k2.6:xhigh PASS 100% 564 in → 1,485 out (1,488 reasoning)
March commit February commit
moonshotai/kimi-k2.7-code:high PASS 100% 564 in → 1,240 out (1,233 reasoning)
March commit February commit
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 564 in → 146 out (129 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
moonshotai/kimi-k2.7-code:low PASS 100% 564 in → 204 out (205 reasoning)
March commit February commit
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 564 in → 113 out (89 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
moonshotai/kimi-k2.7-code:medium PASS 100% 564 in → 231 out (224 reasoning)
March commit February commit
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 564 in → 409 out (391 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 564 in → 107 out (99 reasoning)
March commit February commit
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 564 in → 389 out (373 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
moonshotai/kimi-k3:high PASS 100% 641 in → 163 out (143 reasoning)
March commit February commit
moonshotai/kimi-k3:high__json_schema PASS 100% 641 in → 147 out (127 reasoning)
March commit February commit
moonshotai/kimi-k3:low PASS 100% 641 in → 144 out (124 reasoning)
March commit February commit
moonshotai/kimi-k3:low__json_schema PASS 100% 641 in → 149 out (129 reasoning)
March commit February commit
moonshotai/kimi-k3:max PASS 100% 641 in → 162 out (142 reasoning)
March commit February commit
moonshotai/kimi-k3:max__json_schema PASS 100% 641 in → 147 out (127 reasoning)
March commit February commit
moonshotai/kimi-k3:medium PASS 100% 641 in → 167 out (147 reasoning)
March commit February commit
moonshotai/kimi-k3:medium__json_schema PASS 100% 748 in → 185 out (158 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
moonshotai/kimi-k3:xhigh PASS 100% 641 in → 154 out (134 reasoning)
March commit February commit
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 641 in → 148 out (128 reasoning)
March commit February commit
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 748 in → 488 out (345 reasoning)
February commit March commit
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 748 in → 549 out (497 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 748 in → 433 out (417 reasoning)
February commit March commit
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 748 in → 198 out (118 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 748 in → 242 out (182 reasoning)
February commit March commit
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 748 in → 385 out (264 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
nvidia/nemotron-3-nano-30b-a3b:none FAIL 100% 748 in → 9 out (0 reasoning)
April commit March commit February commit
Failure: Extra lines: ['April commit']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 748 in → 23 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 748 in → 125 out (107 reasoning)
February commit March commit
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 748 in → 341 out (300 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 748 in → 394 out (338 reasoning)
March commit February commit
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 748 in → 415 out (344 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 748 in → 265 out (227 reasoning)
March commit February commit
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 748 in → 180 out (134 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 748 in → 301 out (262 reasoning)
March commit February commit
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 748 in → 331 out (274 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 748 in → 6 out (0 reasoning)
February commit March commit
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 748 in → 33 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 748 in → 183 out (148 reasoning)
March commit February commit
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 748 in → 329 out (261 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
openai/gpt-5.4-mini:high PASS 100% 563 in → 447 out (436 reasoning)
March commit February commit
openai/gpt-5.4-mini:high__json_schema PASS 100% 609 in → 108 out (85 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4-mini:low PASS 100% 563 in → 37 out (26 reasoning)
March commit February commit
openai/gpt-5.4-mini:low__json_schema PASS 100% 609 in → 60 out (37 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4-mini:medium PASS 100% 563 in → 118 out (107 reasoning)
February commit March commit
openai/gpt-5.4-mini:medium__json_schema PASS 100% 609 in → 75 out (52 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4-mini:none PASS 100% 563 in → 9 out (0 reasoning)
February commit March commit
openai/gpt-5.4-mini:none__json_schema PASS 100% 609 in → 21 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
openai/gpt-5.4-mini:xhigh PASS 100% 563 in → 214 out (203 reasoning)
March commit February commit
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 609 in → 539 out (516 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4-nano:high PASS 100% 563 in → 167 out (156 reasoning)
March commit February commit
openai/gpt-5.4-nano:high__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4-nano:low PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.4-nano:low__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4-nano:medium PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.4-nano:medium__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4-nano:none PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.4-nano:none__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4-nano:xhigh PASS 100% 563 in → 72 out (61 reasoning)
February commit March commit
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 609 in → 1,306 out (1,283 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4:high PASS 100% 563 in → 140 out (129 reasoning)
March commit February commit
openai/gpt-5.4:high__json_schema PASS 100% 609 in → 148 out (125 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4:low PASS 100% 563 in → 35 out (24 reasoning)
March commit February commit
openai/gpt-5.4:low__json_schema PASS 100% 609 in → 47 out (24 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4:medium PASS 100% 563 in → 178 out (167 reasoning)
March commit February commit
openai/gpt-5.4:medium__json_schema PASS 100% 609 in → 81 out (58 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4:none PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.4:none__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.4:xhigh PASS 100% 563 in → 136 out (125 reasoning)
March commit February commit
openai/gpt-5.4:xhigh__json_schema PASS 100% 609 in → 186 out (163 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.5:high PASS 100% 563 in → 70 out (59 reasoning)
March commit February commit
openai/gpt-5.5:high__json_schema PASS 100% 609 in → 97 out (74 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.5:low PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.5:low__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.5:medium PASS 100% 563 in → 71 out (60 reasoning)
March commit February commit
openai/gpt-5.5:medium__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.5:none PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.5:none__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.5:xhigh PASS 100% 563 in → 160 out (149 reasoning)
March commit February commit
openai/gpt-5.5:xhigh__json_schema PASS 100% 609 in → 129 out (106 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-luna:high PASS 100% 563 in → 36 out (25 reasoning)
March commit February commit
openai/gpt-5.6-luna:high__json_schema PASS 100% 609 in → 51 out (28 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-luna:low PASS 100% 563 in → 24 out (13 reasoning)
March commit February commit
openai/gpt-5.6-luna:low__json_schema PASS 100% 609 in → 36 out (13 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-luna:max PASS 100% 563 in → 55 out (44 reasoning)
March commit February commit
openai/gpt-5.6-luna:max__json_schema PASS 100% 609 in → 539 out (516 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-luna:medium PASS 100% 563 in → 31 out (20 reasoning)
March commit February commit
openai/gpt-5.6-luna:medium__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-luna:none PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-luna:none__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-luna:xhigh PASS 100% 563 in → 31 out (20 reasoning)
March commit February commit
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 609 in → 54 out (31 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-sol:high PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-sol:high__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-sol:low PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-sol:low__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-sol:max PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-sol:max__json_schema PASS 100% 609 in → 159 out (136 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-sol:medium PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-sol:medium__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-sol:none PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-sol:none__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-sol:xhigh PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-terra:high PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-terra:high__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-terra:low PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-terra:low__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-terra:max PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-terra:max__json_schema PASS 100% 609 in → 105 out (82 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-terra:medium PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-terra:medium__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-terra:none PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-terra:none__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-5.6-terra:xhigh PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 609 in → 51 out (28 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-astra:high PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-6-astra:high__json_schema PASS 100% 609 in → 62 out (39 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-astra:low PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-6-astra:low__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-astra:max PASS 100% 563 in → 90 out (79 reasoning)
March commit February commit
openai/gpt-6-astra:max__json_schema PASS 100% 609 in → 271 out (248 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-astra:medium PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-6-astra:medium__json_schema PASS 100% 609 in → 48 out (25 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-astra:xhigh PASS 100% 563 in → 41 out (30 reasoning)
March commit February commit
openai/gpt-6-astra:xhigh__json_schema PASS 100% 609 in → 93 out (70 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-luna:high PASS 100% 563 in → 37 out (26 reasoning)
March commit February commit
openai/gpt-6-luna:high__json_schema PASS 100% 609 in → 84 out (61 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-luna:low PASS 100% 563 in → 32 out (21 reasoning)
March commit February commit
openai/gpt-6-luna:low__json_schema PASS 100% 609 in → 61 out (38 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-luna:max PASS 100% 563 in → 47 out (36 reasoning)
March commit February commit
openai/gpt-6-luna:max__json_schema PASS 100% 609 in → 73 out (50 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-luna:medium PASS 100% 563 in → 36 out (25 reasoning)
March commit February commit
openai/gpt-6-luna:medium__json_schema PASS 100% 609 in → 67 out (44 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-luna:xhigh PASS 100% 563 in → 37 out (26 reasoning)
March commit February commit
openai/gpt-6-luna:xhigh__json_schema PASS 100% 609 in → 57 out (34 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-sol:high PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-6-sol:high__json_schema PASS 100% 609 in → 98 out (75 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-sol:low PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-6-sol:low__json_schema PASS 100% 609 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-sol:max PASS 100% 563 in → 50 out (39 reasoning)
March commit February commit
openai/gpt-6-sol:max__json_schema PASS 100% 609 in → 302 out (279 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-sol:medium PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-6-sol:medium__json_schema PASS 100% 609 in → 65 out (42 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-6-sol:xhigh PASS 100% 563 in → 9 out (0 reasoning)
March commit February commit
openai/gpt-6-sol:xhigh__json_schema PASS 100% 609 in → 267 out (244 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-oss-120b:high PASS 100% 622 in → 347 out (348 reasoning)
February commit March commit
openai/gpt-oss-120b:high__json_schema PASS 100% 622 in → 471 out (480 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
openai/gpt-oss-120b:low PASS 100% 622 in → 110 out (111 reasoning)
March commit February commit
openai/gpt-oss-120b:low__json_schema PASS 100% 626 in → 262 out (284 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
openai/gpt-oss-120b:medium PASS 100% 624 in → 122 out (115 reasoning)
February commit March commit
openai/gpt-oss-120b:medium__json_schema PASS 100% 624 in → 166 out (135 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
openai/gpt-oss-120b:xhigh PASS 100% 611 in → 956 out (1,044 reasoning)
March commit February commit
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 624 in → 372 out (344 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
openai/gpt-oss-20b:high PASS 100% 611 in → 758 out (868 reasoning)
February commit March commit
openai/gpt-oss-20b:high__json_schema PASS 100% 622 in → 359 out (351 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
openai/gpt-oss-20b:low PASS 100% 624 in → 166 out (161 reasoning)
February commit March commit
openai/gpt-oss-20b:low__json_schema PASS 100% 622 in → 57 out (36 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":[ "February commit", "March commit"]}
openai/gpt-oss-20b:medium PASS 100% 626 in → 108 out (92 reasoning)
February commit March commit
openai/gpt-oss-20b:medium__json_schema PASS 100% 622 in → 188 out (176 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
openai/gpt-oss-20b:xhigh PASS 100% 624 in → 325 out (341 reasoning)
February commit March commit
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 611 in → 275 out (269 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
poolside/laguna-m.1:high PASS 100% 739 in → 313 out (304 reasoning)
February commit March commit
poolside/laguna-m.1:high__json_schema PASS 100% 739 in → 3,649 out (3,633 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
poolside/laguna-m.1:low PASS 100% 739 in → 340 out (332 reasoning)
March commit February commit
poolside/laguna-m.1:low__json_schema PASS 100% 739 in → 3,595 out (3,565 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
poolside/laguna-m.1:medium PASS 100% 739 in → 2,802 out (2,794 reasoning)
March commit February commit
poolside/laguna-m.1:medium__json_schema PASS 100% 739 in → 319 out (292 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
poolside/laguna-m.1:none PASS 100% 739 in → 7 out (0 reasoning)
March commit February commit
poolside/laguna-m.1:none__json_schema PASS 100% 739 in → 26 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
poolside/laguna-m.1:xhigh PASS 100% 739 in → 351 out (343 reasoning)
February commit March commit
poolside/laguna-m.1:xhigh__json_schema PASS 100% 739 in → 1,169 out (1,149 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
poolside/laguna-xs-2.1:high PASS 100% 739 in → 303 out (297 reasoning)
February commit March commit
poolside/laguna-xs-2.1:high__json_schema PASS 100% 739 in → 387 out (372 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
poolside/laguna-xs-2.1:low PASS 100% 739 in → 2,898 out (2,892 reasoning)
March commit February commit
poolside/laguna-xs-2.1:low__json_schema PASS 100% 739 in → 660 out (634 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
poolside/laguna-xs-2.1:medium PASS 100% 739 in → 340 out (334 reasoning)
February commit March commit
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 739 in → 608 out (593 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
poolside/laguna-xs-2.1:none PASS 100% 739 in → 5 out (0 reasoning)
February commit March commit
poolside/laguna-xs-2.1:none__json_schema PASS 100% 739 in → 31 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list" : [ "March commit", "February commit" ] }
poolside/laguna-xs-2.1:xhigh PASS 100% 739 in → 244 out (238 reasoning)
February commit March commit
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 739 in → 681 out (657 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
poolside/laguna-xs.2:high PASS 100% 776 in → 317 out (309 reasoning)
February commit March commit
poolside/laguna-xs.2:high__json_schema PASS 100% 776 in → 533 out (518 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
poolside/laguna-xs.2:low PASS 100% 776 in → 437 out (429 reasoning)
March commit February commit
poolside/laguna-xs.2:low__json_schema PASS 100% 776 in → 499 out (470 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
poolside/laguna-xs.2:medium PASS 100% 776 in → 323 out (315 reasoning)
February commit March commit
poolside/laguna-xs.2:medium__json_schema PASS 100% 776 in → 289 out (260 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
poolside/laguna-xs.2:none PASS 100% 776 in → 7 out (0 reasoning)
March commit February commit
poolside/laguna-xs.2:none__json_schema PASS 100% 776 in → 15 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["February commit", "March commit"]}
poolside/laguna-xs.2:xhigh PASS 100% 776 in → 275 out (267 reasoning)
February commit March commit
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 776 in → 333 out (312 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
prism-ml/ternary-bonsai-2-27b:high PASS 100% 739 in → 199 out (191 reasoning)
March commit February commit
prism-ml/ternary-bonsai-2-27b:high__json_schema PASS 100% 845 in → 217 out (203 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
prism-ml/ternary-bonsai-2-27b:low PASS 100% 739 in → 199 out (191 reasoning)
February commit March commit
prism-ml/ternary-bonsai-2-27b:low__json_schema PASS 100% 845 in → 615 out (586 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
prism-ml/ternary-bonsai-2-27b:max PASS 100% 781 in → 1,579 out (1,571 reasoning)
March commit February commit
prism-ml/ternary-bonsai-2-27b:max__json_schema PASS 100% 883 in → 230 out (201 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
prism-ml/ternary-bonsai-2-27b:medium PASS 100% 739 in → 193 out (185 reasoning)
February commit March commit
prism-ml/ternary-bonsai-2-27b:medium__json_schema PASS 100% 845 in → 381 out (367 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
prism-ml/ternary-bonsai-2-27b:none PASS 100% 741 in → 6 out (0 reasoning)
February commit March commit
prism-ml/ternary-bonsai-2-27b:none__json_schema PASS 100% 847 in → 27 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
prism-ml/ternary-bonsai-2-27b:xhigh PASS 100% 781 in → 115 out (107 reasoning)
February commit March commit
prism-ml/ternary-bonsai-2-27b:xhigh__json_schema PASS 100% 883 in → 189 out (175 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
qwen/qwen3.6-27b:high PASS 100% 739 in → 1,480 out (1,261 reasoning)
March commit February commit
qwen/qwen3.6-27b:high__json_schema PASS 100% 739 in → 1,422 out (1,242 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
qwen/qwen3.6-27b:low PASS 100% 739 in → 1,256 out (1,113 reasoning)
March commit February commit
qwen/qwen3.6-27b:low__json_schema PASS 100% 739 in → 1,259 out (1,121 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
qwen/qwen3.6-27b:medium PASS 100% 739 in → 647 out (574 reasoning)
March commit February commit
qwen/qwen3.6-27b:medium__json_schema PASS 100% 739 in → 1,226 out (1 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
qwen/qwen3.6-27b:none PASS 100% 741 in → 6 out (0 reasoning)
February commit March commit
qwen/qwen3.6-27b:none__json_schema PASS 100% 741 in → 31 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 739 in → 1,208 out (1,084 reasoning)
March commit February commit
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 739 in → 1,255 out (1,075 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 739 in → 856 out (846 reasoning)
March commit February commit
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 739 in → 1,080 out (925 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
qwen/qwen3.6-35b-a3b:low PASS 100% 739 in → 816 out (806 reasoning)
March commit February commit
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 739 in → 1,443 out (1,196 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 739 in → 939 out (821 reasoning)
March commit February commit
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 739 in → 1,310 out (1,071 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
qwen/qwen3.6-35b-a3b:none PASS 100% 741 in → 6 out (0 reasoning)
February commit March commit
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 741 in → 28 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 739 in → 1,438 out (1,428 reasoning)
March commit February commit
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 739 in → 1,927 out (1,739 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
qwen/qwen3.6-flash:high PASS 100% 739 in → 1,060 out (1,050 reasoning)
February commit March commit
qwen/qwen3.6-flash:low PASS 100% 739 in → 936 out (926 reasoning)
March commit February commit
qwen/qwen3.6-flash:medium PASS 100% 739 in → 1,417 out (1,407 reasoning)
March commit February commit
qwen/qwen3.6-flash:none FAIL 100% 741 in → 8 out (0 reasoning)
April commit March commit February commit
Failure: Extra lines: ['April commit']
qwen/qwen3.6-flash:xhigh PASS 100% 739 in → 407 out (397 reasoning)
March commit February commit
qwen/qwen3.7-flash:high PASS 100% 739 in → 1,075 out (1,065 reasoning)
March commit February commit
qwen/qwen3.7-flash:low PASS 100% 739 in → 1,028 out (1,018 reasoning)
March commit February commit
qwen/qwen3.7-flash:low__json_schema PASS 100% 739 in → 761 out (751 reasoning)
March commit February commit
qwen/qwen3.7-flash:max PASS 100% 739 in → 1,393 out (1,383 reasoning)
March commit February commit
qwen/qwen3.7-flash:medium PASS 100% 739 in → 542 out (532 reasoning)
March commit February commit
qwen/qwen3.7-flash:none FAIL 100% 741 in → 8 out (0 reasoning)
April commit March commit February commit
Failure: Extra lines: ['April commit']
qwen/qwen3.7-flash:xhigh PASS 100% 739 in → 1,112 out (1,102 reasoning)
March commit February commit
qwen/qwen3.7-max:high PASS 100% 739 in → 334 out (324 reasoning)
March commit February commit
qwen/qwen3.7-max:low PASS 100% 739 in → 332 out (322 reasoning)
March commit February commit
qwen/qwen3.7-max:medium PASS 100% 739 in → 308 out (298 reasoning)
March commit February commit
qwen/qwen3.7-max:none PASS 100% 741 in → 5 out (0 reasoning)
March commit February commit
qwen/qwen3.7-max:xhigh PASS 100% 739 in → 382 out (372 reasoning)
March commit February commit
qwen/qwen3.7-plus:high PASS 100% 739 in → 681 out (671 reasoning)
March commit February commit
qwen/qwen3.7-plus:low PASS 100% 739 in → 324 out (314 reasoning)
March commit February commit
qwen/qwen3.7-plus:medium PASS 100% 739 in → 621 out (611 reasoning)
March commit February commit
qwen/qwen3.7-plus:none PASS 100% 741 in → 5 out (0 reasoning)
February commit March commit
qwen/qwen3.7-plus:xhigh PASS 100% 739 in → 579 out (569 reasoning)
March commit February commit
qwen/qwen3.8-flash:high PASS 100% 790 in → 1,712 out (1,704 reasoning)
February commit March commit
qwen/qwen3.8-flash:high__json_schema FAIL 100% 792 in → 312 out (204 reasoning)
c5123c440bfbc522b085898f04b1f637264a5fb7 February commit eb3c63770b4ef621ccd003c101db519e1427225c March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "c5123c440bfbc522b085898f04b1f637264a5fb7", "February commit", "eb3c63770b4ef621ccd003c101db519e1427225c", "March commit" ] }
Failure: Extra lines: ['c5123c440bfbc522b085898f04b1f637264a5fb7', 'eb3c63770b4ef621ccd003c101db519e1427225c']
qwen/qwen3.8-flash:low PASS 100% 790 in → 98 out (90 reasoning)
March commit February commit
qwen/qwen3.8-flash:max PASS 100% 790 in → 377 out (369 reasoning)
March commit February commit
qwen/qwen3.8-flash:max__json_schema PASS 100% 792 in → 184 out (168 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
qwen/qwen3.8-flash:medium PASS 100% 790 in → 134 out (126 reasoning)
March commit February commit
qwen/qwen3.8-flash:medium__json_schema PASS 100% 792 in → 180 out (164 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
qwen/qwen3.8-flash:xhigh PASS 100% 790 in → 1,774 out (1,766 reasoning)
March commit February commit
qwen/qwen3.8-flash:xhigh__json_schema PASS 100% 792 in → 853 out (837 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
qwen/qwen3.8-max:high PASS 100% 777 in → 131 out (121 reasoning)
March commit February commit
qwen/qwen3.8-max:high__json_schema FAIL 100% 779 in → 128 out (90 reasoning)
: March commit : February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ ":" ,"March commit", ":" ,"February commit" ] }
Failure: Extra lines: [':']
qwen/qwen3.8-max:low PASS 100% 765 in → 196 out (186 reasoning)
March commit February commit
qwen/qwen3.8-max:low__json_schema PASS 100% 767 in → 191 out (160 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
qwen/qwen3.8-max:max PASS 100% 777 in → 1,915 out (1,905 reasoning)
March commit February commit
qwen/qwen3.8-max:medium PASS 100% 739 in → 214 out (204 reasoning)
March commit February commit
qwen/qwen3.8-max:xhigh PASS 100% 777 in → 166 out (156 reasoning)
March commit February commit
sakana/fugu-max:high PASS 100% 674 in → 187 out (180 reasoning)
March commit February commit
sakana/fugu-max:high__json_schema PASS 100% 751 in → 246 out (231 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
sakana/fugu-max:low PASS 100% 674 in → 494 out (487 reasoning)
March commit February commit
sakana/fugu-max:low__json_schema PASS 100% 751 in → 661 out (646 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
sakana/fugu-max:max PASS 100% 674 in → 29 out (22 reasoning)
February commit March commit
sakana/fugu-max:max__json_schema PASS 100% 751 in → 171 out (155 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
sakana/fugu-max:medium PASS 100% 674 in → 297 out (290 reasoning)
March commit February commit
sakana/fugu-max:medium__json_schema PASS 100% 751 in → 103 out (88 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
sakana/fugu-max:xhigh PASS 100% 674 in → 107 out (100 reasoning)
March commit February commit
sakana/fugu-max:xhigh__json_schema PASS 100% 751 in → 98 out (82 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
stealth/ox-alpha:high PASS 100% 682 in → 7 out (0 reasoning)
March commit February commit
stealth/ox-alpha:low PASS 100% 682 in → 7 out (0 reasoning)
March commit February commit
stealth/ox-alpha:max PASS 100% 682 in → 115 out (0 reasoning)
February commit March commit
tencent/hy3:high PASS 100% 581 in → 822 out (815 reasoning)
March commit February commit
tencent/hy3:high__json_schema PASS 100% 581 in → 563 out (543 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
tencent/hy3:low PASS 100% 581 in → 252 out (197 reasoning)
March commit February commit
tencent/hy3:low__json_schema PASS 100% 581 in → 508 out (455 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
tencent/hy3:medium PASS 100% 581 in → 319 out (312 reasoning)
March commit February commit
tencent/hy3:medium__json_schema PASS 100% 581 in → 196 out (152 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
tencent/hy3:none PASS 100% 584 in → 6 out (0 reasoning)
March commit February commit
tencent/hy3:none__json_schema PASS 100% 584 in → 18 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
tencent/hy3:xhigh PASS 100% 581 in → 449 out (413 reasoning)
March commit February commit
tencent/hy3:xhigh__json_schema PASS 100% 581 in → 304 out (273 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
thinkingmachines/inkling-small:high PASS 100% 572 in → 575 out (584 reasoning)
March commit February commit
thinkingmachines/inkling-small:high__json_schema PASS 100% 572 in → 632 out (612 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
thinkingmachines/inkling-small:low PASS 100% 572 in → 60 out (49 reasoning)
February commit March commit
thinkingmachines/inkling-small:low__json_schema PASS 100% 572 in → 72 out (49 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
thinkingmachines/inkling-small:max PASS 100% 572 in → 673 out (691 reasoning)
February commit March commit
thinkingmachines/inkling-small:max__json_schema PASS 100% 572 in → 287 out (267 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
thinkingmachines/inkling-small:medium PASS 100% 572 in → 324 out (341 reasoning)
March commit February commit
thinkingmachines/inkling-small:medium__json_schema PASS 100% 572 in → 210 out (187 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 572 in → 461 out (450 reasoning)
March commit February commit
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 572 in → 451 out (424 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
thinkingmachines/inkling:high PASS 100% 572 in → 350 out (342 reasoning)
March commit February commit
thinkingmachines/inkling:low PASS 100% 572 in → 69 out (58 reasoning)
February commit March commit
thinkingmachines/inkling:max PASS 100% 572 in → 347 out (336 reasoning)
March commit February commit
thinkingmachines/inkling:max__json_schema PASS 100% 572 in → 230 out (213 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
thinkingmachines/inkling:medium PASS 100% 572 in → 253 out (241 reasoning)
March commit February commit
thinkingmachines/inkling:none__json_schema PASS 100% 570 in → 21 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
thinkingmachines/inkling:xhigh PASS 100% 572 in → 230 out (210 reasoning)
March commit February commit
thinkingmachines/inkling:xhigh__json_schema PASS 100% 572 in → 459 out (433 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
x-ai/grok-4.3:high PASS 100% 739 in → 306 out (301 reasoning)
March commit February commit
x-ai/grok-4.3:high__json_schema PASS 100% 806 in → 488 out (477 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
x-ai/grok-4.3:low PASS 100% 745 in → 350 out (345 reasoning)
March commit February commit
x-ai/grok-4.3:low__json_schema PASS 100% 812 in → 371 out (360 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
x-ai/grok-4.3:max PASS 100% 739 in → 276 out (271 reasoning)
March commit February commit
x-ai/grok-4.3:max__json_schema PASS 100% 806 in → 632 out (621 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
x-ai/grok-4.3:medium PASS 100% 745 in → 1,332 out (1,327 reasoning)
March commit February commit
x-ai/grok-4.3:medium__json_schema PASS 100% 812 in → 703 out (682 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
x-ai/grok-4.3:none PASS 100% 737 in → 5 out (0 reasoning)
March commit February commit
x-ai/grok-4.3:none__json_schema PASS 100% 804 in → 21 out (0 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
x-ai/grok-4.3:xhigh PASS 100% 739 in → 654 out (649 reasoning)
March commit February commit
x-ai/grok-4.3:xhigh__json_schema PASS 100% 806 in → 789 out (778 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
x-ai/grok-4.5:high PASS 100% 773 in → 179 out (174 reasoning)
March commit February commit
x-ai/grok-4.5:high__json_schema PASS 100% 857 in → 436 out (422 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
x-ai/grok-4.5:low PASS 100% 773 in → 114 out (109 reasoning)
March commit February commit
x-ai/grok-4.5:low__json_schema PASS 100% 857 in → 261 out (247 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
x-ai/grok-4.5:max PASS 100% 773 in → 239 out (234 reasoning)
March commit February commit
x-ai/grok-4.5:max__json_schema PASS 100% 857 in → 415 out (401 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
x-ai/grok-4.5:medium PASS 100% 773 in → 196 out (191 reasoning)
March commit February commit
x-ai/grok-4.5:medium__json_schema PASS 100% 857 in → 376 out (352 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
x-ai/grok-4.5:xhigh PASS 100% 773 in → 213 out (208 reasoning)
March commit February commit
x-ai/grok-4.5:xhigh__json_schema PASS 100% 857 in → 367 out (353 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
x-ai/grok-4.7:high PASS 100% 1,809 in → 345 out (340 reasoning)
March commit February commit
x-ai/grok-4.7:high__json_schema PASS 100% 1,893 in → 231 out (217 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
x-ai/grok-4.7:low PASS 100% 1,809 in → 162 out (157 reasoning)
March commit February commit
x-ai/grok-4.7:low__json_schema PASS 100% 1,893 in → 123 out (109 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
x-ai/grok-4.7:max PASS 100% 1,810 in → 158 out (153 reasoning)
March commit February commit
x-ai/grok-4.7:max__json_schema PASS 100% 1,894 in → 308 out (294 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
x-ai/grok-4.7:medium PASS 100% 1,809 in → 158 out (153 reasoning)
March commit February commit
x-ai/grok-4.7:medium__json_schema PASS 100% 1,893 in → 179 out (165 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
x-ai/grok-4.7:xhigh PASS 100% 1,810 in → 220 out (215 reasoning)
March commit February commit
x-ai/grok-4.7:xhigh__json_schema PASS 100% 1,894 in → 185 out (171 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list":["March commit","February commit"]}
xiaomi/mimo-v2.6-flash:high PASS 100% 723 in → 13 out (5 reasoning)
February commit March commit
xiaomi/mimo-v2.6-flash:high__json_schema PASS 100% 723 in → 76 out (57 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
xiaomi/mimo-v2.6-flash:low PASS 100% 723 in → 40 out (32 reasoning)
February commit March commit
xiaomi/mimo-v2.6-flash:low__json_schema PASS 100% 723 in → 55 out (35 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
xiaomi/mimo-v2.6-flash:max PASS 100% 723 in → 17 out (10 reasoning)
March commit February commit
xiaomi/mimo-v2.6-flash:max__json_schema PASS 100% 723 in → 124 out (104 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
xiaomi/mimo-v2.6-flash:medium PASS 100% 723 in → 64 out (56 reasoning)
March commit February commit
xiaomi/mimo-v2.6-flash:medium__json_schema PASS 100% 723 in → 50 out (34 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) {"commit_message_list":["February commit","March commit"]}
xiaomi/mimo-v2.6-flash:none PASS 100% 725 in → 6 out (0 reasoning)
March commit February commit
xiaomi/mimo-v2.6-flash:none__json_schema PASS 100% 725 in → 19 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
xiaomi/mimo-v2.6-flash:xhigh PASS 100% 723 in → 143 out (136 reasoning)
February commit March commit
xiaomi/mimo-v2.6-flash:xhigh__json_schema PASS 100% 723 in → 78 out (58 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
xiaomi/mimo-v2.6-pro:high PASS 100% 723 in → 65 out (58 reasoning)
March commit February commit
xiaomi/mimo-v2.6-pro:high__json_schema PASS 100% 723 in → 97 out (77 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
xiaomi/mimo-v2.6-pro:low PASS 100% 723 in → 65 out (58 reasoning)
February commit March commit
xiaomi/mimo-v2.6-pro:low__json_schema PASS 100% 723 in → 79 out (58 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
xiaomi/mimo-v2.6-pro:max PASS 100% 723 in → 42 out (35 reasoning)
February commit March commit
xiaomi/mimo-v2.6-pro:max__json_schema PASS 100% 723 in → 117 out (93 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit\nMarch commit"] }
xiaomi/mimo-v2.6-pro:medium PASS 100% 723 in → 66 out (59 reasoning)
February commit March commit
xiaomi/mimo-v2.6-pro:medium__json_schema PASS 100% 723 in → 161 out (141 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["February commit", "March commit"] }
xiaomi/mimo-v2.6-pro:xhigh PASS 100% 723 in → 65 out (58 reasoning)
March commit February commit
xiaomi/mimo-v2.6-pro:xhigh__json_schema PASS 100% 723 in → 80 out (58 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
z-ai/glm-4.7-flash:high PASS 100% 600 in → 705 out (517 reasoning)
March commit February commit
z-ai/glm-4.7-flash:low PASS 100% 600 in → 491 out (345 reasoning)
February commit March commit
z-ai/glm-4.7-flash:medium PASS 100% 600 in → 778 out (565 reasoning)
March commit February commit
z-ai/glm-4.7-flash:none FAIL 100% 600 in → 8 out (0 reasoning)
February commit March commit April commit
Failure: Extra lines: ['April commit']
z-ai/glm-4.7-flash:none__json_schema FAIL 100% 600 in → 29 out (0 reasoning)
February commit March commit April commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit", "April commit" ] }
Failure: Extra lines: ['April commit']
z-ai/glm-4.7-flash:xhigh PASS 100% 600 in → 596 out (485 reasoning)
March commit February commit
z-ai/glm-5.2:high PASS 100% 607 in → 166 out (159 reasoning)
March commit February commit
z-ai/glm-5.2:low PASS 100% 607 in → 137 out (130 reasoning)
February commit March commit
z-ai/glm-5.2:low__json_schema PASS 100% 607 in → 104 out (70 reasoning)
February commit March commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "February commit", "March commit" ] }
z-ai/glm-5.2:medium PASS 100% 607 in → 326 out (259 reasoning)
March commit February commit
z-ai/glm-5.2:medium__json_schema PASS 100% 607 in → 183 out (132 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
z-ai/glm-5.2:none PASS 100% 601 in → 6 out (0 reasoning)
February commit March commit
z-ai/glm-5.2:none__json_schema PASS 100% 601 in → 21 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
z-ai/glm-5.2:xhigh PASS 100% 607 in → 403 out (326 reasoning)
March commit February commit
z-ai/glm-5.3-flash:high PASS 100% 607 in → 7 out (0 reasoning)
March commit February commit
z-ai/glm-5.3-flash:high__json_schema PASS 100% 607 in → 26 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "March commit", "February commit" ] }
z-ai/glm-5.3-flash:low PASS 100% 607 in → 7 out (0 reasoning)
March commit February commit
z-ai/glm-5.3-flash:low__json_schema PASS 100% 607 in → 19 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
z-ai/glm-5.3-flash:max PASS 100% 607 in → 101 out (94 reasoning)
March commit February commit
z-ai/glm-5.3-flash:max__json_schema PASS 100% 607 in → 186 out (165 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
z-ai/glm-5.3-flash:medium PASS 100% 607 in → 7 out (0 reasoning)
March commit February commit
z-ai/glm-5.3-flash:medium__json_schema PASS 100% 607 in → 17 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) {"commit_message_list": ["March commit", "February commit"]}
z-ai/glm-5.3-flash:xhigh PASS 100% 607 in → 7 out (0 reasoning)
March commit February commit
z-ai/glm-5.3-flash:xhigh__json_schema PASS 100% 607 in → 20 out (0 reasoning)
March commit February commit
JSON Schema Structured Output
(raw) { "commit_message_list": ["March commit", "February commit"] }
arcee-ai/trinity-mini:high FAIL 50% 564 in → 356 out (349 reasoning)
Marchcommit February commit
Failure: Missing lines: ['March commit']; Extra lines: ['Marchcommit']
deepseek/deepseek-v4-flash:xhigh FAIL 50% 652 in → 510 out (556 reasoning)
Marchcommit February commit
Failure: Missing lines: ['March commit']; Extra lines: ['Marchcommit']
thinkingmachines/inkling-small:none FAIL 50% 572 in → 8 out (0 reasoning)
March commit February commit<|end_message|>
Failure: Missing lines: ['February commit']; Extra lines: ['February commit<|end_message|>']
thinkingmachines/inkling:none FAIL 50% 572 in → 8 out (0 reasoning)
March commit February commit<|end_message|>
Failure: Missing lines: ['February commit']; Extra lines: ['February commit<|end_message|>']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 0%
Invalid JSON. Output: {{ "commit_message_list": [ "February commit", "March commit" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
ibm-granite/granite-4.2-8b:low FAIL 0% 621 in → 51 out (39 reasoning)
c5123c4 February commit eb3c637 March commit
Failure: Missing lines: ['February commit', 'March commit']; Extra lines: ['c5123c4 February commit', 'eb3c637 March commit']
ibm-granite/granite-4.2-8b:low__json_schema FAIL 0% 621 in → 32 out (0 reasoning)
c5123c4 February commit eb3c637 March commit c3a6a94 April commit
JSON Schema Structured Output
(raw) {"commit_message_list":["c5123c4 February commit","eb3c637 March commit","c3a6a94 April commit"]}
Failure: Missing lines: ['February commit', 'March commit']; Extra lines: ['c3a6a94 April commit', 'c5123c4 February commit', 'eb3c637 March commit']
ibm-granite/granite-4.2-8b:medium FAIL 0% 621 in → 34 out (21 reasoning)
c5123c4 February commit eb3c637 March commit
Failure: Missing lines: ['February commit', 'March commit']; Extra lines: ['c5123c4 February commit', 'eb3c637 March commit']
ibm-granite/granite-4.2-8b:none FAIL 0% 613 in → 24 out (0 reasoning)
c5123c4 February commit eb3c637 March commit c3a6a94 April commit
Failure: Missing lines: ['February commit', 'March commit']; Extra lines: ['c3a6a94 April commit', 'c5123c4 February commit', 'eb3c637 March commit']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: April commit March commit February commit January commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3a00aea2-09bc-9df7-9cad-e572d82cb195","request_id":"3a00aea2-09bc-9df7-9cad-e572d82cb195"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-84b9e86e-183f-952e-a48c-c40c89dc7f76","request_id":"84b9e86e-183f-952e-a48c-c40c89dc7f76"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-85140980-d2ce-984d-a039-9a166f64a873","request_id":"85140980-d2ce-984d-a039-9a166f64a873"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1608deff-88ed-993a-a155-4aef148f4c6b","request_id":"1608deff-88ed-993a-a155-4aef148f4c6b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ab8e0e65-7666-9033-b56a-48ef2bd8a659","request_id":"ab8e0e65-7666-9033-b56a-48ef2bd8a659"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a313cf19-0d3b-9de5-9ef4-c020e4ac1b04"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-fda76bac-6467-9c36-a630-21b62c67033f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-74313028-0197-9971-a220-07cdb3d6fcf5"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6f76e192-af17-9dd8-9a89-e44c698156db"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-eae590bf-d994-9b90-a5bd-a382938a40de"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8ce8836c-748f-9d55-8547-4c5535b75eee","request_id":"8ce8836c-748f-9d55-8547-4c5535b75eee"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b2477b01-051d-9f78-a6f5-d7ddd3f3e69e","request_id":"b2477b01-051d-9f78-a6f5-d7ddd3f3e69e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-eb6596db-ab34-9d47-b741-ca0e2b5fcf88","request_id":"eb6596db-ab34-9d47-b741-ca0e2b5fcf88"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-be354145-8fc7-9e14-be22-192504eda932","request_id":"be354145-8fc7-9e14-be22-192504eda932"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cf1cfc40-c892-97da-a4eb-caf4b73decaa","request_id":"cf1cfc40-c892-97da-a4eb-caf4b73decaa"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fbaacaec-caa0-91e1-b681-3857df3384e5","request_id":"fbaacaec-caa0-91e1-b681-3857df3384e5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ae48c5c2-e0e3-9a0d-bf42-56029e158d0f","request_id":"ae48c5c2-e0e3-9a0d-bf42-56029e158d0f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dac486c3-1a35-907f-b275-6c117c28105c","request_id":"dac486c3-1a35-907f-b275-6c117c28105c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1120d226-db0e-9f30-8952-a706fb39cbe4","request_id":"1120d226-db0e-9f30-8952-a706fb39cbe4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c8b3a2c7-c554-96b2-83de-968c2fef22cc","request_id":"c8b3a2c7-c554-96b2-83de-968c2fef22cc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.8-flash:low__json_schema FAIL 0% 792 in → 709 out (611 reasoning)
{"commit_message": "April commit", "date": "2025-04-01"} {"commit_message": "March commit", "date": "2025-03-01"} {"commit_message": "February commit", "date": "2025-02-01"}
JSON Schema Structured Output
(raw) { "commit_message_list": [ "{\"commit_message\": \"April commit\", \"date\": \"2025-04-01\"}", "{\"commit_message\": \"March commit\", \"date\": \"2025-03-01\"}", "{\"commit_message\": \"February commit\", \"date\": \"2025-02-01\"}" ] }
Failure: Missing lines: ['February commit', 'March commit']; Extra lines: ['{"commit_message": "April commit", "date": "2025-04-01"}', '{"commit_message": "February commit", "date": "2025-02-01"}', '{"commit_message": "March commit", "date": "2025-03-01"}']
qwen/qwen3.8-max:max__json_schema FAIL 0% 779 in → 282 out (246 reasoning)
März-Commit Februar-Commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ "März-Commit", "Februar-Commit" ] }
Failure: Missing lines: ['February commit', 'March commit']; Extra lines: ['Februar-Commit', 'März-Commit']
qwen/qwen3.8-max:medium__json_schema FAIL 0% 741 in → 253 out (210 reasoning)
>{"commit_message":"March commit"} >{"commit_message":"February commit"}
JSON Schema Structured Output
(raw) { "commit_message_list": [ ">{\"commit_message\":\"March commit\"}", ">{\"commit_message\":\"February commit\"}" ] }
Failure: Missing lines: ['February commit', 'March commit']; Extra lines: ['>{"commit_message":"February commit"}', '>{"commit_message":"March commit"}']
qwen/qwen3.8-max:xhigh__json_schema FAIL 0% 779 in → 201 out (170 reasoning)
.March commit .February commit
JSON Schema Structured Output
(raw) { "commit_message_list": [ ".March commit", ".February commit" ] }
Failure: Missing lines: ['February commit', 'March commit']; Extra lines: ['.February commit', '.March commit']
stealth/ox-alpha:high__json_schema FAIL 0%
Invalid JSON. Output: February commit March commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
stealth/ox-alpha:low__json_schema FAIL 0%
Invalid JSON. Output: February commit March commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
stealth/ox-alpha:max__json_schema FAIL 0%
Invalid JSON. Output: March commit February commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: March commit February commit{ "commit_message_list": [ "March commit", "February commit" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: February commit March commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: February commit March commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: March commit February commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output: March commit February commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output: March commit February commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output: March commit February commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:high__json_schema FAIL 0%
Invalid JSON. Output: March commit February commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:xhigh__json_schema FAIL 0%
Invalid JSON. Output: March commit February commit
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)