Identify commits by a specific author
Tests ability to identify commits by a specific author. Evaluates author-filtering in git log.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'alice@example.com'
  3. 03 git config user.name 'Alice'
  4. 04 echo 'v1' > code.py
  5. 05 git add code.py
  6. 06 git commit -m 'Initial code by Alice'
  7. 07 git config user.email 'bob@example.com'
  8. 08 git config user.name 'Bob'
  9. 09 echo 'v2' > code.py
  10. 10 git add code.py
  11. 11 git commit -m 'Refactor by Bob'
  12. 12 git config user.email 'alice@example.com'
  13. 13 git config user.name 'Alice'
  14. 14 echo 'v3' > code.py
  15. 15 git add code.py
  16. 16 git commit -m 'Bugfix by Alice'
  17. 17 git config user.email 'bob@example.com'
  18. 18 git config user.name 'Bob'
  19. 19 echo 'v4' > code.py
  20. 20 git add code.py
  21. 21 git commit -m 'Feature by Bob'
Prompt
Which commits were authored by Alice? Output ONLY the commit messages, one per line, nothing else.
Expected
Initial code by Alice Bugfix by Alice

Scoped model quality, cost, API time, and token usage for git_log_format/f003.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high__json_schema PASS 100% 1,053 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
anthropic/claude-fable-5:low PASS 100% 826 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-fable-5:low__json_schema PASS 100% 1,041 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
anthropic/claude-fable-5:max__json_schema PASS 100% 1,041 in → 173 out (47 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
anthropic/claude-fable-5:medium PASS 100% 800 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-fable-5:medium__json_schema PASS 100% 1,030 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 1,061 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
anthropic/claude-haiku-4.5:high PASS 100% 678 in → 298 out (211 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 864 in → 283 out (212 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
anthropic/claude-haiku-4.5:low PASS 100% 671 in → 381 out (268 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 858 in → 590 out (510 reasoning)
Initial code by Alice Bugfix by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Initial code by Alice", "Bugfix by Alice"]}
anthropic/claude-haiku-4.5:medium PASS 100% 696 in → 274 out (215 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 868 in → 521 out (447 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
anthropic/claude-haiku-4.5:none PASS 100% 655 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 847 in → 24 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 677 in → 400 out (265 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 878 in → 969 out (575 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
anthropic/claude-opus-4.6:high PASS 100% 657 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.6:high__json_schema PASS 100% 831 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.6:low PASS 100% 648 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.6:low__json_schema PASS 100% 857 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.6:max PASS 100% 650 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.6:max__json_schema PASS 100% 851 in → 90 out (43 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.6:medium PASS 100% 635 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 832 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.6:none PASS 100% 640 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.6:none__json_schema PASS 100% 830 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 622 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 833 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.7:high PASS 100% 791 in → 22 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.7:high__json_schema PASS 100% 1,054 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.7:low PASS 100% 808 in → 22 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.7:low__json_schema PASS 100% 1,049 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.7:max PASS 100% 801 in → 22 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.7:max__json_schema PASS 100% 1,051 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.7:medium PASS 100% 814 in → 22 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 1,042 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.7:none PASS 100% 808 in → 22 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.7:none__json_schema PASS 100% 1,051 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 826 in → 22 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 1,042 in → 36 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.8:high PASS 100% 821 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.8:high__json_schema PASS 100% 1,034 in → 33 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.8:low PASS 100% 809 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.8:low__json_schema PASS 100% 1,040 in → 33 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.8:max PASS 100% 818 in → 101 out (24 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.8:max__json_schema PASS 100% 1,064 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
anthropic/claude-opus-4.8:medium PASS 100% 793 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 1,049 in → 33 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.8:none PASS 100% 803 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.8:none__json_schema PASS 100% 1,061 in → 32 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 809 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 1,045 in → 33 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-4.6:high PASS 100% 641 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 837 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-4.6:low PASS 100% 647 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 849 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-4.6:max PASS 100% 644 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 849 in → 113 out (70 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 644 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 844 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-4.6:none PASS 100% 646 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 849 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 666 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 862 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-5:high PASS 100% 790 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-5:high__json_schema PASS 100% 1,064 in → 33 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-5:low PASS 100% 807 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-5:low__json_schema PASS 100% 1,021 in → 33 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-5:medium PASS 100% 808 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 1,040 in → 33 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-5:none PASS 100% 793 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-5:none__json_schema PASS 100% 1,032 in → 32 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 815 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 1,026 in → 35 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
arcee-ai/trinity-large-thinking:high PASS 100% 581 in → 956 out (955 reasoning)
Bugfix by Alice Initial code by Alice
arcee-ai/trinity-large-thinking:low PASS 100% 559 in → 504 out (494 reasoning)
Bugfix by Alice Initial code by Alice
arcee-ai/trinity-large-thinking:medium PASS 100% 570 in → 832 out (822 reasoning)
Bugfix by Alice Initial code by Alice
arcee-ai/trinity-large-thinking:xhigh PASS 100% 553 in → 759 out (756 reasoning)
Bugfix by Alice Initial code by Alice
arcee-ai/trinity-mini:high PASS 100% 575 in → 336 out (356 reasoning)
Bugfix by Alice Initial code by Alice
arcee-ai/trinity-mini:high__json_schema PASS 100% 560 in → 289 out (287 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
arcee-ai/trinity-mini:low__json_schema PASS 100% 577 in → 302 out (280 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 553 in → 360 out (352 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 547 in → 301 out (298 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 575 in → 75 out (61 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 676 in → 96 out (73 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
deepseek/deepseek-v4-flash-0731:low PASS 100% 579 in → 54 out (50 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 573 in → 82 out (56 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 654 in → 108 out (101 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 673 in → 91 out (63 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 576 in → 272 out (254 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 585 in → 81 out (56 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 573 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 555 in → 33 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 656 in → 47 out (37 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 572 in → 81 out (55 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
deepseek/deepseek-v4-flash:high PASS 100% 571 in → 58 out (47 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 565 in → 90 out (60 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice"] }
deepseek/deepseek-v4-flash:low PASS 100% 566 in → 54 out (47 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 566 in → 73 out (45 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
deepseek/deepseek-v4-flash:medium PASS 100% 567 in → 68 out (57 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 574 in → 82 out (47 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
deepseek/deepseek-v4-flash:none PASS 100% 580 in → 10 out (0 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 582 in → 30 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
deepseek/deepseek-v4-flash:xhigh PASS 100% 649 in → 148 out (137 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 577 in → 86 out (60 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
deepseek/deepseek-v4-pro:high PASS 100% 592 in → 238 out (228 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 597 in → 190 out (156 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
deepseek/deepseek-v4-pro:low PASS 100% 573 in → 123 out (121 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-pro:medium PASS 100% 586 in → 300 out (289 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 782 in → 489 out (469 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
deepseek/deepseek-v4-pro:none PASS 100% 570 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 558 in → 29 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 632 in → 93 out (82 reasoning)
Bugfix by Alice Initial code by Alice
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 839 in → 2,169 out (2,149 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
google/gemini-3-flash-preview:high PASS 100% 733 in → 259 out (250 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3-flash-preview:high__json_schema PASS 100% 747 in → 270 out (253 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3-flash-preview:low PASS 100% 724 in → 251 out (242 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3-flash-preview:low__json_schema PASS 100% 817 in → 461 out (444 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3-flash-preview:medium PASS 100% 723 in → 355 out (346 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3-flash-preview:medium__json_schema PASS 100% 821 in → 442 out (425 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3-flash-preview:xhigh PASS 100% 726 in → 318 out (309 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 720 in → 374 out (357 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 734 in → 525 out (516 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 739 in → 314 out (282 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemini-3.1-flash-lite-preview:low PASS 100% 750 in → 132 out (123 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 736 in → 162 out (130 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 726 in → 217 out (208 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 819 in → 285 out (253 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 731 in → 776 out (767 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.1-pro-preview:high PASS 100% 741 in → 334 out (325 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 805 in → 553 out (536 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.1-pro-preview:low PASS 100% 719 in → 345 out (336 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 812 in → 203 out (186 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.1-pro-preview:medium PASS 100% 731 in → 308 out (299 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 812 in → 416 out (399 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 748 in → 465 out (456 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 827 in → 367 out (350 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.5-flash-lite:high PASS 100% 733 in → 662 out (653 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 808 in → 319 out (302 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.5-flash-lite:low PASS 100% 728 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 743 in → 32 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemini-3.5-flash-lite:max PASS 100% 751 in → 482 out (473 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 744 in → 459 out (442 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.5-flash-lite:medium PASS 100% 736 in → 214 out (205 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 820 in → 32 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemini-3.5-flash-lite:xhigh PASS 100% 736 in → 660 out (651 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 814 in → 542 out (525 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.5-flash:high PASS 100% 738 in → 271 out (262 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.5-flash:high__json_schema PASS 100% 811 in → 460 out (443 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.5-flash:low PASS 100% 740 in → 144 out (135 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.5-flash:low__json_schema PASS 100% 819 in → 171 out (154 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.5-flash:medium PASS 100% 746 in → 338 out (329 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.5-flash:medium__json_schema PASS 100% 810 in → 213 out (181 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemini-3.5-flash:xhigh PASS 100% 745 in → 405 out (396 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 823 in → 419 out (402 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.6-flash:high PASS 100% 751 in → 311 out (302 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.6-flash:high__json_schema PASS 100% 729 in → 447 out (430 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.6-flash:low PASS 100% 734 in → 179 out (170 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.6-flash:low__json_schema PASS 100% 825 in → 32 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemini-3.6-flash:max PASS 100% 740 in → 347 out (338 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.6-flash:max__json_schema PASS 100% 818 in → 499 out (482 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.6-flash:medium PASS 100% 759 in → 368 out (359 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.6-flash:medium__json_schema PASS 100% 742 in → 279 out (262 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemini-3.6-flash:xhigh PASS 100% 744 in → 438 out (429 reasoning)
Bugfix by Alice Initial code by Alice
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 826 in → 334 out (317 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
google/gemma-4-26b-a4b-it:high PASS 100% 756 in → 692 out (380 reasoning)
Bugfix by Alice Initial code by Alice
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 754 in → 655 out (429 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 755 in → 426 out (337 reasoning)
Bugfix by Alice Initial code by Alice
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 742 in → 498 out (368 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 745 in → 375 out (325 reasoning)
Bugfix by Alice Initial code by Alice
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 749 in → 547 out (322 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 752 in → 10 out (0 reasoning)
Bugfix by Alice Initial code by Alice
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 752 in → 29 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 751 in → 526 out (384 reasoning)
Bugfix by Alice Initial code by Alice
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 753 in → 401 out (309 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemma-4-31b-it:high PASS 100% 754 in → 333 out (1 reasoning)
Bugfix by Alice Initial code by Alice
google/gemma-4-31b-it:high__json_schema PASS 100% 753 in → 150 out (96 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemma-4-31b-it:low PASS 100% 756 in → 266 out (212 reasoning)
Bugfix by Alice Initial code by Alice
google/gemma-4-31b-it:low__json_schema PASS 100% 740 in → 186 out (118 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemma-4-31b-it:medium PASS 100% 739 in → 299 out (223 reasoning)
Bugfix by Alice Initial code by Alice
google/gemma-4-31b-it:medium__json_schema PASS 100% 751 in → 267 out (164 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemma-4-31b-it:none PASS 100% 744 in → 10 out (0 reasoning)
Bugfix by Alice Initial code by Alice
google/gemma-4-31b-it:none__json_schema PASS 100% 740 in → 29 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
google/gemma-4-31b-it:xhigh PASS 100% 754 in → 391 out (1 reasoning)
Bugfix by Alice Initial code by Alice
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 750 in → 355 out (248 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
ibm-granite/granite-4.1-8b PASS 100% 606 in → 10 out
Bugfix by Alice Initial code by Alice
ibm-granite/granite-4.1-8b__json_schema PASS 100% 605 in → 23 out
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
inclusionai/ling-2.6-flash PASS 100% 766 in → 10 out
Bugfix by Alice Initial code by Alice
inclusionai/ling-2.6-flash__json_schema PASS 100% 772 in → 33 out
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
liquid/lfm-2-24b-a2b PASS 100% 612 in → 11 out
Bugfix by Alice Initial code by Alice
minimax/minimax-m2.5:high PASS 100% 582 in → 266 out (254 reasoning)
Bugfix by Alice Initial code by Alice
minimax/minimax-m2.5:high__json_schema PASS 100% 593 in → 172 out (167 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
minimax/minimax-m2.5:low__json_schema PASS 100% 605 in → 200 out (204 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
minimax/minimax-m2.5:medium PASS 100% 605 in → 197 out (187 reasoning)
Bugfix by Alice Initial code by Alice
minimax/minimax-m2.5:medium__json_schema PASS 100% 581 in → 339 out (354 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 600 in → 431 out (428 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
minimax/minimax-m2.7:high PASS 100% 596 in → 269 out (257 reasoning)
Bugfix by Alice Initial code by Alice
minimax/minimax-m2.7:high__json_schema PASS 100% 736 in → 584 out (557 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
minimax/minimax-m2.7:low PASS 100% 579 in → 283 out (273 reasoning)
Bugfix by Alice Initial code by Alice
minimax/minimax-m2.7:low__json_schema PASS 100% 604 in → 410 out (412 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
minimax/minimax-m2.7:medium PASS 100% 590 in → 411 out (399 reasoning)
Bugfix by Alice Initial code by Alice
minimax/minimax-m2.7:medium__json_schema PASS 100% 756 in → 940 out (913 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
minimax/minimax-m2.7:xhigh PASS 100% 605 in → 701 out (758 reasoning)
Bugfix by Alice Initial code by Alice
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 728 in → 581 out (554 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
minimax/minimax-m3:high PASS 100% 725 in → 63 out (56 reasoning)
Bugfix by Alice Initial code by Alice
minimax/minimax-m3:high__json_schema PASS 100% 721 in → 21 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
minimax/minimax-m3:low PASS 100% 716 in → 56 out (46 reasoning)
Bugfix by Alice Initial code by Alice
minimax/minimax-m3:low__json_schema PASS 100% 711 in → 32 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
minimax/minimax-m3:medium PASS 100% 741 in → 126 out (104 reasoning)
Bugfix by Alice Initial code by Alice
minimax/minimax-m3:medium__json_schema PASS 100% 723 in → 20 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
minimax/minimax-m3:xhigh PASS 100% 725 in → 102 out (88 reasoning)
Bugfix by Alice Initial code by Alice
minimax/minimax-m3:xhigh__json_schema PASS 100% 728 in → 22 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
mistralai/devstral-2512 PASS 100% 735 in → 10 out
Bugfix by Alice Initial code by Alice
mistralai/devstral-2512__json_schema PASS 100% 731 in → 26 out
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
mistralai/mistral-medium-3-5:high PASS 100% 717 in → 595 out (455 reasoning)
Bugfix by Alice Initial code by Alice
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 722 in → 186 out (173 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
mistralai/mistral-medium-3-5:low PASS 100% 740 in → 262 out (240 reasoning)
Bugfix by Alice Initial code by Alice
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 705 in → 2,290 out (2,092 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
mistralai/mistral-medium-3-5:medium PASS 100% 734 in → 1,108 out (993 reasoning)
Bugfix by Alice Initial code by Alice
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 701 in → 212 out (190 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
mistralai/mistral-medium-3-5:none PASS 100% 725 in → 10 out (0 reasoning)
Bugfix by Alice Initial code by Alice
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 701 in → 19 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 738 in → 537 out (539 reasoning)
Bugfix by Alice Initial code by Alice
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 725 in → 204 out (200 reasoning)
Initial code by Alice Bugfix by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Initial code by Alice", "Bugfix by Alice"]}
moonshotai/kimi-k2.6:high PASS 100% 559 in → 206 out (195 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k2.6:high__json_schema PASS 100% 562 in → 135 out (115 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
moonshotai/kimi-k2.6:low PASS 100% 572 in → 355 out (344 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k2.6:low__json_schema PASS 100% 561 in → 468 out (438 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
moonshotai/kimi-k2.6:medium PASS 100% 563 in → 690 out (681 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 574 in → 1,484 out (341 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list" : [ "Bugfix by Alice", "Initial code by Alice" ] }
moonshotai/kimi-k2.6:xhigh PASS 100% 574 in → 104 out (105 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 559 in → 1,191 out (656 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
moonshotai/kimi-k2.7-code:high PASS 100% 552 in → 97 out (86 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k2.7-code:low PASS 100% 556 in → 79 out (68 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k2.7-code:medium PASS 100% 558 in → 101 out (90 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 567 in → 158 out (137 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 566 in → 222 out (194 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 552 in → 84 out (60 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
moonshotai/kimi-k3:high PASS 100% 644 in → 108 out (84 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k3:high__json_schema PASS 100% 645 in → 120 out (96 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k3:low PASS 100% 645 in → 111 out (87 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k3:low__json_schema PASS 100% 653 in → 71 out (47 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k3:max PASS 100% 638 in → 83 out (59 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k3:max__json_schema PASS 100% 742 in → 88 out (57 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
moonshotai/kimi-k3:medium PASS 100% 629 in → 136 out (112 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k3:medium__json_schema PASS 100% 636 in → 109 out (85 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k3:xhigh PASS 100% 633 in → 76 out (52 reasoning)
Bugfix by Alice Initial code by Alice
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 636 in → 114 out (90 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 739 in → 78 out (79 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 742 in → 106 out (100 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 730 in → 69 out (64 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 727 in → 120 out (113 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 734 in → 115 out (122 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 746 in → 224 out (228 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 730 in → 10 out (0 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 741 in → 26 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 730 in → 73 out (72 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 741 in → 126 out (119 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 726 in → 83 out (68 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 731 in → 110 out (82 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 724 in → 100 out (85 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 732 in → 99 out (74 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 742 in → 261 out (209 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 740 in → 109 out (76 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 718 in → 10 out (0 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 743 in → 26 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 741 in → 89 out (92 reasoning)
Bugfix by Alice Initial code by Alice
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 727 in → 147 out (118 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
openai/gpt-5.4-mini:high PASS 100% 558 in → 66 out (51 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4-mini:high__json_schema PASS 100% 598 in → 68 out (41 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4-mini:low PASS 100% 576 in → 42 out (27 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4-mini:low__json_schema PASS 100% 602 in → 42 out (15 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4-mini:medium PASS 100% 554 in → 50 out (35 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4-mini:medium__json_schema PASS 100% 595 in → 62 out (35 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4-mini:none PASS 100% 564 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4-mini:none__json_schema PASS 100% 603 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4-mini:xhigh PASS 100% 570 in → 80 out (65 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 609 in → 1,005 out (978 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4-nano:high PASS 100% 566 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4-nano:high__json_schema PASS 100% 607 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4-nano:low PASS 100% 557 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4-nano:low__json_schema PASS 100% 599 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4-nano:medium PASS 100% 558 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4-nano:medium__json_schema PASS 100% 592 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4-nano:none PASS 100% 551 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4-nano:none__json_schema PASS 100% 614 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4-nano:xhigh PASS 100% 549 in → 144 out (129 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 596 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4:high PASS 100% 558 in → 56 out (41 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4:high__json_schema PASS 100% 613 in → 70 out (43 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4:low PASS 100% 567 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4:low__json_schema PASS 100% 594 in → 37 out (10 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4:medium PASS 100% 574 in → 54 out (39 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4:medium__json_schema PASS 100% 611 in → 58 out (31 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4:none PASS 100% 540 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4:none__json_schema PASS 100% 602 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.4:xhigh PASS 100% 566 in → 91 out (76 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.4:xhigh__json_schema PASS 100% 600 in → 335 out (308 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.5:high PASS 100% 557 in → 52 out (37 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.5:high__json_schema PASS 100% 606 in → 66 out (39 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.5:low PASS 100% 557 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.5:low__json_schema PASS 100% 598 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.5:medium PASS 100% 553 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.5:medium__json_schema PASS 100% 603 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.5:none PASS 100% 561 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.5:none__json_schema PASS 100% 592 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.5:xhigh PASS 100% 557 in → 64 out (49 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.5:xhigh__json_schema PASS 100% 596 in → 85 out (58 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-luna:high PASS 100% 556 in → 36 out (21 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-luna:high__json_schema PASS 100% 605 in → 47 out (20 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-luna:low PASS 100% 558 in → 26 out (11 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-luna:low__json_schema PASS 100% 591 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-luna:max PASS 100% 561 in → 48 out (33 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-luna:max__json_schema PASS 100% 602 in → 63 out (36 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-luna:medium PASS 100% 561 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-luna:medium__json_schema PASS 100% 619 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-luna:none PASS 100% 561 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-luna:none__json_schema PASS 100% 603 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-luna:xhigh PASS 100% 549 in → 46 out (31 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 596 in → 59 out (32 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-sol:high PASS 100% 556 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-sol:high__json_schema PASS 100% 606 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-sol:low PASS 100% 574 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-sol:low__json_schema PASS 100% 607 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-sol:max PASS 100% 548 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-sol:max__json_schema PASS 100% 607 in → 176 out (149 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-sol:medium PASS 100% 558 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-sol:medium__json_schema PASS 100% 615 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-sol:none PASS 100% 569 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-sol:none__json_schema PASS 100% 613 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-sol:xhigh PASS 100% 564 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 603 in → 118 out (91 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-terra:high PASS 100% 556 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-terra:high__json_schema PASS 100% 602 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-terra:low PASS 100% 565 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-terra:low__json_schema PASS 100% 594 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-terra:max PASS 100% 554 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-terra:max__json_schema PASS 100% 606 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-terra:medium PASS 100% 561 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-terra:medium__json_schema PASS 100% 601 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-terra:none PASS 100% 555 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-terra:none__json_schema PASS 100% 620 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-5.6-terra:xhigh PASS 100% 583 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 610 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-oss-120b:high PASS 100% 613 in → 225 out (232 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-oss-120b:high__json_schema PASS 100% 628 in → 234 out (225 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
openai/gpt-oss-120b:low PASS 100% 638 in → 45 out (27 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-oss-120b:low__json_schema PASS 100% 634 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
openai/gpt-oss-120b:medium PASS 100% 635 in → 108 out (96 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-oss-120b:medium__json_schema PASS 100% 624 in → 95 out (61 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
openai/gpt-oss-120b:xhigh PASS 100% 616 in → 188 out (181 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 622 in → 262 out (286 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
openai/gpt-oss-20b:high PASS 100% 624 in → 289 out (271 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-oss-20b:high__json_schema PASS 100% 602 in → 821 out (789 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":[ "Bugfix by Alice", "Initial code by Alice" ]}
openai/gpt-oss-20b:low PASS 100% 613 in → 44 out (27 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-oss-20b:low__json_schema PASS 100% 608 in → 57 out (19 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
openai/gpt-oss-20b:medium PASS 100% 626 in → 258 out (225 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-oss-20b:medium__json_schema PASS 100% 626 in → 18 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
openai/gpt-oss-20b:xhigh PASS 100% 619 in → 280 out (285 reasoning)
Bugfix by Alice Initial code by Alice
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 616 in → 19 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
poolside/laguna-m.1:high PASS 100% 721 in → 446 out (434 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-m.1:high__json_schema PASS 100% 709 in → 393 out (373 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
poolside/laguna-m.1:low PASS 100% 713 in → 497 out (485 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-m.1:low__json_schema PASS 100% 710 in → 293 out (273 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
poolside/laguna-m.1:medium PASS 100% 713 in → 264 out (252 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-m.1:medium__json_schema PASS 100% 734 in → 303 out (274 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ]}
poolside/laguna-m.1:none PASS 100% 718 in → 11 out (0 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-m.1:none__json_schema PASS 100% 721 in → 19 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
poolside/laguna-m.1:xhigh PASS 100% 716 in → 348 out (336 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-m.1:xhigh__json_schema PASS 100% 737 in → 647 out (616 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
poolside/laguna-xs-2.1:high PASS 100% 716 in → 433 out (423 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-xs-2.1:high__json_schema PASS 100% 719 in → 370 out (341 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
poolside/laguna-xs-2.1:low PASS 100% 715 in → 354 out (344 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-xs-2.1:low__json_schema PASS 100% 716 in → 941 out (927 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
poolside/laguna-xs-2.1:medium PASS 100% 729 in → 343 out (333 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 727 in → 353 out (319 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
poolside/laguna-xs-2.1:none PASS 100% 713 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-xs-2.1:none__json_schema PASS 100% 727 in → 26 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
poolside/laguna-xs-2.1:xhigh PASS 100% 716 in → 228 out (218 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 733 in → 320 out (291 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
poolside/laguna-xs.2:high PASS 100% 768 in → 94 out (82 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-xs.2:high__json_schema PASS 100% 765 in → 244 out (211 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
poolside/laguna-xs.2:low PASS 100% 758 in → 312 out (300 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-xs.2:low__json_schema PASS 100% 783 in → 252 out (235 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
poolside/laguna-xs.2:medium PASS 100% 752 in → 236 out (226 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-xs.2:medium__json_schema PASS 100% 750 in → 251 out (233 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
poolside/laguna-xs.2:none PASS 100% 747 in → 11 out (0 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-xs.2:none__json_schema PASS 100% 751 in → 19 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
poolside/laguna-xs.2:xhigh PASS 100% 780 in → 261 out (251 reasoning)
Bugfix by Alice Initial code by Alice
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 755 in → 145 out (128 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
qwen/qwen3.6-27b:high PASS 100% 733 in → 843 out (807 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-27b:high__json_schema PASS 100% 736 in → 419 out (385 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
qwen/qwen3.6-27b:low PASS 100% 742 in → 1,030 out (947 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-27b:low__json_schema PASS 100% 721 in → 33 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
qwen/qwen3.6-27b:medium PASS 100% 729 in → 455 out (369 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-27b:medium__json_schema PASS 100% 737 in → 817 out (762 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
qwen/qwen3.6-27b:none PASS 100% 740 in → 10 out (0 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-27b:none__json_schema PASS 100% 728 in → 28 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 726 in → 663 out (637 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 728 in → 423 out (414 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
qwen/qwen3.6-35b-a3b:high PASS 100% 731 in → 331 out (314 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 727 in → 498 out (472 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 730 in → 296 out (263 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 737 in → 1,125 out (1,025 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 741 in → 924 out (872 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 733 in → 1,003 out (958 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 738 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 752 in → 33 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 726 in → 1,005 out (991 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 720 in → 512 out (469 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
qwen/qwen3.6-flash:high PASS 100% 722 in → 636 out (622 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-flash:low PASS 100% 745 in → 965 out (951 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-flash:medium PASS 100% 747 in → 943 out (929 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-flash:none PASS 100% 725 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.6-flash:xhigh PASS 100% 741 in → 344 out (330 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-flash:high PASS 100% 724 in → 688 out (674 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-flash:low PASS 100% 720 in → 1,141 out (1,127 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-flash:max PASS 100% 728 in → 604 out (590 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-flash:medium PASS 100% 723 in → 870 out (856 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-flash:none PASS 100% 716 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-flash:none__json_schema PASS 100% 718 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-flash:xhigh PASS 100% 743 in → 328 out (314 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-max:high PASS 100% 710 in → 209 out (195 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-max:low PASS 100% 727 in → 196 out (182 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-max:medium PASS 100% 730 in → 222 out (208 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-max:none PASS 100% 718 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-max:xhigh PASS 100% 731 in → 201 out (187 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-plus:high PASS 100% 728 in → 485 out (471 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-plus:low PASS 100% 731 in → 292 out (278 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-plus:medium PASS 100% 742 in → 300 out (286 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-plus:none PASS 100% 738 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
qwen/qwen3.7-plus:xhigh PASS 100% 731 in → 412 out (398 reasoning)
Bugfix by Alice Initial code by Alice
tencent/hy3:high PASS 100% 550 in → 261 out (250 reasoning)
Bugfix by Alice Initial code by Alice
tencent/hy3:high__json_schema PASS 100% 581 in → 213 out (184 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
tencent/hy3:low PASS 100% 569 in → 211 out (197 reasoning)
Bugfix by Alice Initial code by Alice
tencent/hy3:low__json_schema PASS 100% 565 in → 207 out (160 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
tencent/hy3:medium PASS 100% 566 in → 205 out (194 reasoning)
Bugfix by Alice Initial code by Alice
tencent/hy3:medium__json_schema PASS 100% 583 in → 185 out (120 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
tencent/hy3:none PASS 100% 571 in → 10 out (0 reasoning)
Bugfix by Alice Initial code by Alice
tencent/hy3:none__json_schema PASS 100% 565 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
tencent/hy3:xhigh PASS 100% 586 in → 250 out (239 reasoning)
Bugfix by Alice Initial code by Alice
tencent/hy3:xhigh__json_schema PASS 100% 580 in → 105 out (79 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
thinkingmachines/inkling-small:high PASS 100% 564 in → 202 out (183 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling-small:high__json_schema PASS 100% 583 in → 291 out (262 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
thinkingmachines/inkling-small:low PASS 100% 555 in → 49 out (34 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling-small:low__json_schema PASS 100% 556 in → 57 out (30 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
thinkingmachines/inkling-small:max PASS 100% 570 in → 118 out (106 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling-small:max__json_schema PASS 100% 578 in → 321 out (296 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
thinkingmachines/inkling-small:medium PASS 100% 586 in → 224 out (222 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling-small:medium__json_schema PASS 100% 576 in → 80 out (53 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
thinkingmachines/inkling-small:none PASS 100% 578 in → 12 out (0 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling-small:xhigh PASS 100% 556 in → 130 out (119 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 570 in → 59 out (34 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
thinkingmachines/inkling:high PASS 100% 582 in → 211 out (195 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling:high__json_schema PASS 100% 584 in → 207 out (176 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
thinkingmachines/inkling:low PASS 100% 564 in → 37 out (22 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling:max PASS 100% 573 in → 227 out (212 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling:max__json_schema PASS 100% 580 in → 190 out (159 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
thinkingmachines/inkling:medium PASS 100% 560 in → 47 out (32 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling:medium__json_schema PASS 100% 557 in → 203 out (166 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
thinkingmachines/inkling:none PASS 100% 572 in → 13 out (0 reasoning)
Bugfix by Alice Initial code by Alice
thinkingmachines/inkling:none__json_schema PASS 100% 569 in → 23 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
thinkingmachines/inkling:xhigh PASS 100% 574 in → 222 out (208 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.3:high PASS 100% 733 in → 561 out (552 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.3:high__json_schema PASS 100% 783 in → 347 out (332 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
x-ai/grok-4.3:low PASS 100% 748 in → 436 out (427 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.3:low__json_schema PASS 100% 811 in → 288 out (273 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
x-ai/grok-4.3:max PASS 100% 733 in → 435 out (426 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.3:max__json_schema PASS 100% 781 in → 555 out (540 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
x-ai/grok-4.3:medium PASS 100% 755 in → 288 out (279 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.3:medium__json_schema PASS 100% 798 in → 575 out (550 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
x-ai/grok-4.3:none PASS 100% 729 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.3:none__json_schema PASS 100% 808 in → 25 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
x-ai/grok-4.3:xhigh PASS 100% 724 in → 368 out (359 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.3:xhigh__json_schema PASS 100% 790 in → 373 out (358 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
x-ai/grok-4.5:high PASS 100% 761 in → 241 out (232 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.5:high__json_schema PASS 100% 847 in → 227 out (199 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
x-ai/grok-4.5:low PASS 100% 767 in → 282 out (273 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.5:low__json_schema PASS 100% 848 in → 128 out (100 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
x-ai/grok-4.5:max PASS 100% 782 in → 88 out (79 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.5:max__json_schema PASS 100% 873 in → 294 out (266 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
x-ai/grok-4.5:medium PASS 100% 774 in → 178 out (169 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.5:medium__json_schema PASS 100% 852 in → 292 out (274 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
x-ai/grok-4.5:xhigh PASS 100% 785 in → 338 out (329 reasoning)
Bugfix by Alice Initial code by Alice
x-ai/grok-4.5:xhigh__json_schema PASS 100% 866 in → 228 out (210 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list":["Bugfix by Alice","Initial code by Alice"]}
z-ai/glm-4.7-flash:high PASS 100% 603 in → 470 out (439 reasoning)
Bugfix by Alice Initial code by Alice
z-ai/glm-4.7-flash:high__json_schema PASS 100% 587 in → 523 out (444 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
z-ai/glm-4.7-flash:low PASS 100% 592 in → 448 out (385 reasoning)
Bugfix by Alice Initial code by Alice
z-ai/glm-4.7-flash:medium PASS 100% 603 in → 531 out (472 reasoning)
Bugfix by Alice Initial code by Alice
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 601 in → 500 out (412 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
z-ai/glm-4.7-flash:none PASS 100% 602 in → 9 out (0 reasoning)
Bugfix by Alice Initial code by Alice
z-ai/glm-4.7-flash:none__json_schema PASS 100% 600 in → 21 out (0 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
z-ai/glm-4.7-flash:xhigh PASS 100% 599 in → 521 out (422 reasoning)
Bugfix by Alice Initial code by Alice
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 592 in → 500 out (426 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) {"commit_message_list": ["Bugfix by Alice", "Initial code by Alice"]}
z-ai/glm-5.2:high PASS 100% 595 in → 110 out (98 reasoning)
Bugfix by Alice Initial code by Alice
z-ai/glm-5.2:high__json_schema PASS 100% 606 in → 262 out (240 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
z-ai/glm-5.2:low PASS 100% 599 in → 188 out (177 reasoning)
Bugfix by Alice Initial code by Alice
z-ai/glm-5.2:low__json_schema PASS 100% 601 in → 265 out (241 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": ["Bugfix by Alice", "Initial code by Alice"] }
z-ai/glm-5.2:medium PASS 100% 599 in → 340 out (327 reasoning)
Bugfix by Alice Initial code by Alice
z-ai/glm-5.2:medium__json_schema PASS 100% 774 in → 410 out (383 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
z-ai/glm-5.2:none PASS 100% 592 in → 10 out (0 reasoning)
Bugfix by Alice Initial code by Alice
z-ai/glm-5.2:xhigh PASS 100% 616 in → 304 out (264 reasoning)
Bugfix by Alice Initial code by Alice
z-ai/glm-5.2:xhigh__json_schema PASS 100% 602 in → 355 out (298 reasoning)
Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
arcee-ai/trinity-mini:low FAIL 50% 566 in → 318 out (325 reasoning)
Bugfix byAlice Initial code by Alice
Failure: Missing lines: ['Bugfix by Alice']; Extra lines: ['Bugfix byAlice']
arcee-ai/trinity-mini:medium FAIL 50% 556 in → 329 out (339 reasoning)
Bugfixby Alice Initial code by Alice
Failure: Missing lines: ['Bugfix by Alice']; Extra lines: ['Bugfixby Alice']
arcee-ai/trinity-mini:xhigh FAIL 50% 560 in → 374 out (407 reasoning)
Bugfix by Alice Initial codeby Alice
Failure: Missing lines: ['Initial code by Alice']; Extra lines: ['Initial codeby Alice']
minimax/minimax-m2.5:low FAIL 50% 593 in → 200 out (191 reasoning)
by Alice Initial code by Alice
Failure: Missing lines: ['Bugfix by Alice']; Extra lines: ['by Alice']
minimax/minimax-m2.5:xhigh FAIL 50% 608 in → 714 out (738 reasoning)
by Alice Initial code by Alice
Failure: Missing lines: ['Bugfix by Alice']; Extra lines: ['by Alice']
anthropic/claude-fable-5:high FAIL 0% 808 in → 3 out (0 reasoning)
(empty output)
Failure: Missing lines: ['Bugfix by Alice', 'Initial code by Alice']
anthropic/claude-fable-5:max FAIL 0% 825 in → 6 out (0 reasoning)
(empty output)
Failure: Missing lines: ['Bugfix by Alice', 'Initial code by Alice']
anthropic/claude-fable-5:xhigh FAIL 0% 801 in → 3 out (0 reasoning)
(empty output)
Failure: Missing lines: ['Bugfix by Alice', 'Initial code by Alice']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-08267eaa-d097-98dc-bdc6-23fe288ff2b6","request_id":"08267eaa-d097-98dc-bdc6-23fe288ff2b6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1ef25d66-b849-94b4-b51f-2e59c7b63a37","request_id":"1ef25d66-b849-94b4-b51f-2e59c7b63a37"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d440f42e-08cc-906d-9620-b5d29902f44f","request_id":"d440f42e-08cc-906d-9620-b5d29902f44f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9d6270cd-f384-9f53-acab-a2cb47c5142c","request_id":"9d6270cd-f384-9f53-acab-a2cb47c5142c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-52ce1747-ea5e-9413-999d-37923a07de67","request_id":"52ce1747-ea5e-9413-999d-37923a07de67"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5bd673cf-3551-9d49-b06b-a85db0855c7d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4271feb8-6ae0-9b45-a010-9038e3b37426"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4393f32d-ca4d-97f7-bd94-2d7d4a82ae12"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-95673a55-f210-9d6e-b764-d4e8c85263cd"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f2e4293a-2e7a-9d99-a58a-e2bb9df24cb1"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-729b2013-0db1-9d76-9458-a30942ac3569","request_id":"729b2013-0db1-9d76-9458-a30942ac3569"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b8efd2e0-15e9-98a9-9b24-6bc4b6f112a4","request_id":"b8efd2e0-15e9-98a9-9b24-6bc4b6f112a4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-20f558b6-c8ff-9763-89d6-e0bc5457d8e0","request_id":"20f558b6-c8ff-9763-89d6-e0bc5457d8e0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4dae050a-85f9-93d2-98a1-1a7747756e3b","request_id":"4dae050a-85f9-93d2-98a1-1a7747756e3b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-27d79cab-4371-926e-808a-e4b8647b1124","request_id":"27d79cab-4371-926e-808a-e4b8647b1124"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dd61499d-0690-98e9-b16c-ae0752d78952","request_id":"dd61499d-0690-98e9-b16c-ae0752d78952"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4ecc98b9-6a30-977a-9125-a0f5673868d3","request_id":"4ecc98b9-6a30-977a-9125-a0f5673868d3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a6eaaa52-ad9f-9b41-8d3c-293432ce8738","request_id":"a6eaaa52-ad9f-9b41-8d3c-293432ce8738"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6d62cbc7-240a-9b09-890f-769212bb5091","request_id":"6d62cbc7-240a-9b09-890f-769212bb5091"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-735afab7-a71c-98a2-8bb4-6a06bc002363","request_id":"735afab7-a71c-98a2-8bb4-6a06bc002363"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: Bugfix by Alice Initial code by Alice{ "commit_message_list": [ "Bugfix by Alice", "Initial code by Alice" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Bugfix by Alice Initial code by Alice
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:none__json_schema FAIL 0% 599 in → 29 out (0 reasoning)
Initial commit Add feature X Refactor feature X
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Initial commit", "Add feature X", "Refactor feature X" ] }
Failure: Missing lines: ['Bugfix by Alice', 'Initial code by Alice']; Extra lines: ['Add feature X', 'Initial commit', 'Refactor feature X']