List commit messages matching a grep pattern
Tests ability to list commit messages matching a grep pattern. Evaluates log filtering with message output.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'alpha' > file.txt
  5. 05 git add file.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 echo 'beta' > file.txt
  8. 08 git add file.txt
  9. 09 git commit -m 'Add authentication module'
  10. 10 echo 'gamma' > file.txt
  11. 11 git add file.txt
  12. 12 git commit -m 'Fix login bug'
  13. 13 echo 'delta' > file.txt
  14. 14 git add file.txt
  15. 15 git commit -m 'Add authorization checks'
  16. 16 echo 'epsilon' > file.txt
  17. 17 git add file.txt
  18. 18 git commit -m 'Update README'
Prompt
List the commit messages of all commits that contain the word 'Add' in their message. Output one commit message per line, nothing else.
Expected
Add authentication module Add authorization checks

Scoped model quality, cost, API time, and token usage for git_log_format/f002.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 960 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-fable-5:high__json_schema PASS 100% 1,183 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-fable-5:low PASS 100% 954 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-fable-5:low__json_schema PASS 100% 1,176 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-fable-5:max PASS 100% 959 in → 95 out (32 reasoning)
Add authorization checks Add authentication module
anthropic/claude-fable-5:max__json_schema PASS 100% 1,195 in → 184 out (42 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-fable-5:medium PASS 100% 955 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-fable-5:medium__json_schema PASS 100% 1,230 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-fable-5:xhigh PASS 100% 928 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 1,187 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-haiku-4.5:high PASS 100% 823 in → 197 out (153 reasoning)
Add authorization checks Add authentication module
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 1,010 in → 258 out (198 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-haiku-4.5:low PASS 100% 830 in → 176 out (144 reasoning)
Add authorization checks Add authentication module
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 1,010 in → 194 out (167 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-haiku-4.5:medium PASS 100% 815 in → 162 out (141 reasoning)
Add authorization checks Add authentication module
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 994 in → 250 out (215 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-haiku-4.5:none PASS 100% 790 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 950 in → 21 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 830 in → 197 out (147 reasoning)
Add authorization checks Add authentication module
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 1,005 in → 453 out (410 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-opus-4.6:high PASS 100% 792 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.6:high__json_schema PASS 100% 970 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.6:low PASS 100% 793 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.6:low__json_schema PASS 100% 980 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.6:max PASS 100% 799 in → 116 out (92 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.6:max__json_schema PASS 100% 978 in → 131 out (91 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.6:medium PASS 100% 807 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 982 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.6:none PASS 100% 780 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.6:none__json_schema PASS 100% 975 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 811 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 990 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.7:high PASS 100% 948 in → 18 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.7:high__json_schema PASS 100% 1,214 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.7:low PASS 100% 951 in → 18 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.7:low__json_schema PASS 100% 1,203 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.7:max PASS 100% 960 in → 18 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.7:max__json_schema PASS 100% 1,201 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.7:medium PASS 100% 956 in → 18 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 1,191 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.7:none PASS 100% 942 in → 18 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.7:none__json_schema PASS 100% 1,188 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 943 in → 18 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 1,210 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.8:high PASS 100% 949 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.8:high__json_schema PASS 100% 1,217 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.8:low PASS 100% 944 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.8:low__json_schema PASS 100% 1,194 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.8:max PASS 100% 942 in → 113 out (44 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.8:max__json_schema PASS 100% 1,207 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
anthropic/claude-opus-4.8:medium PASS 100% 949 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 1,200 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.8:none PASS 100% 957 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.8:none__json_schema PASS 100% 1,193 in → 28 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 927 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 1,190 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-4.6:high PASS 100% 803 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 983 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-4.6:low PASS 100% 796 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 982 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-4.6:max PASS 100% 815 in → 138 out (103 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 972 in → 160 out (112 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 792 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 989 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-4.6:none PASS 100% 786 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 991 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 796 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 994 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-5:high PASS 100% 976 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-5:high__json_schema PASS 100% 1,220 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-5:low PASS 100% 941 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-5:low__json_schema PASS 100% 1,193 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-5:medium PASS 100% 936 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 1,183 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-5:none PASS 100% 954 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-5:none__json_schema PASS 100% 1,196 in → 28 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 959 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 1,194 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
arcee-ai/trinity-large-thinking:high PASS 100% 691 in → 1,066 out (1,064 reasoning)
Add authorization checks Add authentication module
arcee-ai/trinity-large-thinking:low PASS 100% 695 in → 897 out (892 reasoning)
Add authorization checks Add authentication module
arcee-ai/trinity-large-thinking:medium PASS 100% 690 in → 532 out (523 reasoning)
Add authorization checks Add authentication module
arcee-ai/trinity-large-thinking:xhigh PASS 100% 687 in → 917 out (914 reasoning)
Add authorization checks Add authentication module
arcee-ai/trinity-mini:high__json_schema PASS 100% 685 in → 458 out (497 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list":[ "Add authorization checks", "Add authentication module" ] }
arcee-ai/trinity-mini:low PASS 100% 683 in → 492 out (569 reasoning)
Add authorization checks Add authentication module
arcee-ai/trinity-mini:low__json_schema PASS 100% 713 in → 424 out (445 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 695 in → 334 out (353 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 667 in → 365 out (407 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 707 in → 49 out (47 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 707 in → 216 out (182 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 691 in → 25 out (23 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash-0731:max PASS 100% 792 in → 141 out (158 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 774 in → 169 out (164 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 684 in → 56 out (64 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 731 in → 101 out (88 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 698 in → 7 out (0 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 694 in → 20 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 699 in → 142 out (152 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 718 in → 112 out (100 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": ["Add authorization checks", "Add authentication module"] }
deepseek/deepseek-v4-flash:high PASS 100% 702 in → 111 out (102 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 714 in → 121 out (97 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": ["Add authorization checks\nAdd authentication module"] }
deepseek/deepseek-v4-flash:low PASS 100% 700 in → 140 out (131 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 698 in → 151 out (120 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
deepseek/deepseek-v4-flash:medium PASS 100% 702 in → 113 out (106 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 712 in → 120 out (89 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
deepseek/deepseek-v4-flash:none PASS 100% 698 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 847 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
deepseek/deepseek-v4-flash:xhigh PASS 100% 821 in → 278 out (269 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 804 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
deepseek/deepseek-v4-pro:high PASS 100% 695 in → 130 out (121 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 899 in → 95 out (77 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
deepseek/deepseek-v4-pro:low PASS 100% 716 in → 147 out (138 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 708 in → 24 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
deepseek/deepseek-v4-pro:medium PASS 100% 689 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 716 in → 24 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
deepseek/deepseek-v4-pro:none PASS 100% 682 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 702 in → 18 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": ["Add authorization checks", "Add authentication module"] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 776 in → 231 out (222 reasoning)
Add authorization checks Add authentication module
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 980 in → 976 out (959 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3-flash-preview:high PASS 100% 910 in → 159 out (152 reasoning)
Add authorization checks Add authentication module
google/gemini-3-flash-preview:high__json_schema PASS 100% 916 in → 309 out (294 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3-flash-preview:low PASS 100% 910 in → 213 out (206 reasoning)
Add authorization checks Add authentication module
google/gemini-3-flash-preview:low__json_schema PASS 100% 912 in → 15 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3-flash-preview:medium PASS 100% 923 in → 333 out (326 reasoning)
Add authorization checks Add authentication module
google/gemini-3-flash-preview:medium__json_schema PASS 100% 1,000 in → 362 out (347 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3-flash-preview:xhigh PASS 100% 916 in → 514 out (507 reasoning)
Add authorization checks Add authentication module
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 907 in → 396 out (381 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 915 in → 538 out (531 reasoning)
Add authorization checks Add authentication module
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 917 in → 279 out (262 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 922 in → 112 out (105 reasoning)
Add authorization checks Add authentication module
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 992 in → 131 out (101 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 875 in → 203 out (196 reasoning)
Add authorization checks Add authentication module
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 907 in → 305 out (276 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 916 in → 551 out (544 reasoning)
Add authorization checks Add authentication module
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 895 in → 341 out (326 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.1-pro-preview:high PASS 100% 905 in → 354 out (347 reasoning)
Add authorization checks Add authentication module
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 979 in → 508 out (493 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.1-pro-preview:low PASS 100% 914 in → 227 out (220 reasoning)
Add authorization checks Add authentication module
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 994 in → 261 out (246 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.1-pro-preview:medium PASS 100% 906 in → 149 out (142 reasoning)
Add authorization checks Add authentication module
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 998 in → 427 out (412 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 904 in → 337 out (330 reasoning)
Add authorization checks Add authentication module
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 987 in → 254 out (239 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.5-flash-lite:high PASS 100% 913 in → 400 out (393 reasoning)
Add authorization checks Add authentication module
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 999 in → 322 out (307 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.5-flash-lite:low PASS 100% 899 in → 7 out (0 reasoning)
Add authorization checks Add authentication module
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 995 in → 30 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemini-3.5-flash-lite:max PASS 100% 901 in → 349 out (342 reasoning)
Add authorization checks Add authentication module
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 908 in → 432 out (417 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.5-flash-lite:medium PASS 100% 908 in → 208 out (201 reasoning)
Add authorization checks Add authentication module
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 984 in → 260 out (230 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemini-3.5-flash-lite:xhigh PASS 100% 917 in → 328 out (321 reasoning)
Add authorization checks Add authentication module
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 910 in → 515 out (500 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.5-flash:high PASS 100% 920 in → 369 out (362 reasoning)
Add authorization checks Add authentication module
google/gemini-3.5-flash:high__json_schema PASS 100% 984 in → 311 out (296 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.5-flash:low PASS 100% 892 in → 194 out (187 reasoning)
Add authorization checks Add authentication module
google/gemini-3.5-flash:low__json_schema PASS 100% 977 in → 119 out (104 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.5-flash:medium PASS 100% 885 in → 273 out (266 reasoning)
Add authorization checks Add authentication module
google/gemini-3.5-flash:medium__json_schema PASS 100% 1,003 in → 120 out (105 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.5-flash:xhigh PASS 100% 901 in → 217 out (210 reasoning)
Add authorization checks Add authentication module
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 989 in → 280 out (265 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.6-flash:high PASS 100% 920 in → 364 out (357 reasoning)
Add authorization checks Add authentication module
google/gemini-3.6-flash:high__json_schema PASS 100% 918 in → 404 out (389 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.6-flash:low PASS 100% 900 in → 109 out (102 reasoning)
Add authorization checks Add authentication module
google/gemini-3.6-flash:low__json_schema PASS 100% 917 in → 15 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.6-flash:max PASS 100% 903 in → 414 out (407 reasoning)
Add authorization checks Add authentication module
google/gemini-3.6-flash:max__json_schema PASS 100% 894 in → 325 out (310 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.6-flash:medium PASS 100% 916 in → 378 out (371 reasoning)
Add authorization checks Add authentication module
google/gemini-3.6-flash:medium__json_schema PASS 100% 896 in → 322 out (307 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemini-3.6-flash:xhigh PASS 100% 893 in → 334 out (327 reasoning)
Add authorization checks Add authentication module
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 998 in → 373 out (358 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemma-4-26b-a4b-it:high PASS 100% 922 in → 434 out (367 reasoning)
Add authorization checks Add authentication module
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 928 in → 563 out (380 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 926 in → 472 out (417 reasoning)
Add authorization checks Add authentication module
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 931 in → 675 out (428 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 946 in → 625 out (395 reasoning)
Add authorization checks Add authentication module
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 921 in → 483 out (416 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 919 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 936 in → 27 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 929 in → 664 out (511 reasoning)
Add authorization checks Add authentication module
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 921 in → 418 out (331 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemma-4-31b-it:high PASS 100% 944 in → 220 out (175 reasoning)
Add authorization checks Add authentication module
google/gemma-4-31b-it:high__json_schema PASS 100% 920 in → 238 out (183 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemma-4-31b-it:low PASS 100% 915 in → 195 out (157 reasoning)
Add authorization checks Add authentication module
google/gemma-4-31b-it:low__json_schema PASS 100% 928 in → 199 out (182 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
google/gemma-4-31b-it:medium PASS 100% 934 in → 196 out (157 reasoning)
Add authorization checks Add authentication module
google/gemma-4-31b-it:medium__json_schema PASS 100% 919 in → 217 out (157 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemma-4-31b-it:none PASS 100% 913 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
google/gemma-4-31b-it:none__json_schema PASS 100% 930 in → 22 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
google/gemma-4-31b-it:xhigh PASS 100% 934 in → 210 out (1 reasoning)
Add authorization checks Add authentication module
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 922 in → 200 out (134 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
ibm-granite/granite-4.1-8b PASS 100% 746 in → 8 out
Add authorization checks Add authentication module
ibm-granite/granite-4.1-8b__json_schema PASS 100% 750 in → 20 out
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": ["Add authorization checks", "Add authentication module"] }
inclusionai/ling-2.6-flash PASS 100% 940 in → 8 out
Add authorization checks Add authentication module
liquid/lfm-2-24b-a2b PASS 100% 741 in → 12 out
Add authorization checks Add authentication module Add authentication module
minimax/minimax-m2.5:high PASS 100% 723 in → 258 out (249 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m2.5:high__json_schema PASS 100% 749 in → 604 out (653 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": ["Add authorization checks", "Add authentication module"] }
minimax/minimax-m2.5:low PASS 100% 729 in → 243 out (291 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m2.5:low__json_schema PASS 100% 721 in → 203 out (220 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
minimax/minimax-m2.5:medium PASS 100% 789 in → 223 out (214 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m2.5:medium__json_schema PASS 100% 793 in → 236 out (211 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
minimax/minimax-m2.5:xhigh PASS 100% 727 in → 289 out (331 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 738 in → 240 out (272 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
minimax/minimax-m2.7:high PASS 100% 739 in → 308 out (367 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m2.7:low PASS 100% 704 in → 288 out (352 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m2.7:low__json_schema PASS 100% 700 in → 267 out (307 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
minimax/minimax-m2.7:medium PASS 100% 708 in → 146 out (171 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m2.7:medium__json_schema PASS 100% 862 in → 339 out (314 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
minimax/minimax-m2.7:xhigh PASS 100% 718 in → 368 out (360 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 871 in → 479 out (463 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
minimax/minimax-m3:high PASS 100% 860 in → 125 out (106 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m3:high__json_schema PASS 100% 860 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
minimax/minimax-m3:low PASS 100% 875 in → 125 out (125 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m3:low__json_schema PASS 100% 851 in → 31 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
minimax/minimax-m3:medium PASS 100% 870 in → 111 out (109 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m3:medium__json_schema PASS 100% 838 in → 25 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
minimax/minimax-m3:xhigh PASS 100% 866 in → 106 out (104 reasoning)
Add authorization checks Add authentication module
minimax/minimax-m3:xhigh__json_schema PASS 100% 848 in → 18 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
mistralai/devstral-2512 PASS 100% 894 in → 8 out
Add authorization checks Add authentication module
mistralai/devstral-2512__json_schema PASS 100% 903 in → 24 out
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
mistralai/mistral-medium-3-5:high PASS 100% 906 in → 443 out (478 reasoning)
Add authorization checks Add authentication module
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 891 in → 158 out (166 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
mistralai/mistral-medium-3-5:low PASS 100% 911 in → 390 out (398 reasoning)
Add authorization checks Add authentication module
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 884 in → 221 out (205 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
mistralai/mistral-medium-3-5:medium PASS 100% 902 in → 456 out (501 reasoning)
Add authorization checks Add authentication module
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 904 in → 217 out (201 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
mistralai/mistral-medium-3-5:none PASS 100% 892 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 891 in → 17 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 907 in → 329 out (352 reasoning)
Add authorization checks Add authentication module
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 879 in → 173 out (162 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
moonshotai/kimi-k2.6:high PASS 100% 691 in → 149 out (155 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k2.6:high__json_schema PASS 100% 701 in → 288 out (301 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
moonshotai/kimi-k2.6:low PASS 100% 686 in → 184 out (203 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k2.6:low__json_schema PASS 100% 694 in → 318 out (152 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
moonshotai/kimi-k2.6:medium PASS 100% 691 in → 159 out (172 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 708 in → 422 out (112 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": ["Add authorization checks", "Add authentication module"]}
moonshotai/kimi-k2.6:xhigh PASS 100% 701 in → 529 out (520 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 676 in → 384 out (275 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": ["Add authorization checks", "Add authentication module"]}
moonshotai/kimi-k2.7-code:high PASS 100% 682 in → 107 out (98 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 694 in → 143 out (117 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
moonshotai/kimi-k2.7-code:low PASS 100% 699 in → 159 out (150 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k2.7-code:medium PASS 100% 678 in → 43 out (34 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k2.7-code:xhigh PASS 100% 686 in → 66 out (57 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 692 in → 146 out (120 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
moonshotai/kimi-k3:high PASS 100% 751 in → 140 out (118 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k3:high__json_schema PASS 100% 778 in → 90 out (68 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k3:low PASS 100% 753 in → 121 out (99 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k3:low__json_schema PASS 100% 874 in → 118 out (89 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
moonshotai/kimi-k3:max PASS 100% 752 in → 91 out (69 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k3:max__json_schema PASS 100% 765 in → 145 out (123 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k3:medium PASS 100% 761 in → 89 out (67 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k3:medium__json_schema PASS 100% 769 in → 90 out (68 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k3:xhigh PASS 100% 776 in → 96 out (74 reasoning)
Add authorization checks Add authentication module
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 771 in → 102 out (80 reasoning)
Add authorization checks Add authentication module
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 917 in → 267 out (247 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 907 in → 96 out (114 reasoning)
Add authorization checks Add authentication module
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 901 in → 219 out (247 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 898 in → 123 out (145 reasoning)
Add authorization checks Add authentication module
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 910 in → 216 out (230 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
nvidia/nemotron-3-nano-30b-a3b:none FAIL 100% 918 in → 11 out (0 reasoning)
Update README Add authorization checks Add authentication module
Failure: Extra lines: ['Update README']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 919 in → 25 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 918 in → 202 out (249 reasoning)
Add authorization checks Add authentication module
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 913 in → 192 out (211 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 890 in → 133 out (128 reasoning)
Add authorization checks Add authentication module
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 901 in → 97 out (76 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 908 in → 175 out (153 reasoning)
Add authorization checks Add authentication module
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 909 in → 106 out (82 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 924 in → 117 out (107 reasoning)
Add authorization checks Add authentication module
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 933 in → 174 out (140 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 925 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 906 in → 30 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 912 in → 167 out (180 reasoning)
Add authorization checks Add authentication module
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 892 in → 96 out (67 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
openai/gpt-5.4-mini:high PASS 100% 703 in → 56 out (43 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4-mini:high__json_schema PASS 100% 722 in → 84 out (59 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4-mini:low PASS 100% 709 in → 27 out (14 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4-mini:low__json_schema PASS 100% 714 in → 40 out (15 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4-mini:medium PASS 100% 669 in → 52 out (39 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4-mini:medium__json_schema PASS 100% 746 in → 58 out (33 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4-mini:none PASS 100% 689 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4-mini:none__json_schema PASS 100% 738 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4-mini:xhigh PASS 100% 694 in → 110 out (97 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 734 in → 175 out (150 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4-nano:high PASS 100% 687 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4-nano:high__json_schema PASS 100% 733 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4-nano:low PASS 100% 678 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4-nano:low__json_schema PASS 100% 747 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4-nano:medium PASS 100% 694 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4-nano:medium__json_schema PASS 100% 738 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4-nano:none PASS 100% 679 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4-nano:none__json_schema PASS 100% 739 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4-nano:xhigh PASS 100% 677 in → 82 out (69 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 730 in → 551 out (526 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4:high PASS 100% 693 in → 60 out (47 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4:high__json_schema PASS 100% 741 in → 67 out (42 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4:low PASS 100% 713 in → 22 out (9 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4:low__json_schema PASS 100% 725 in → 34 out (9 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4:medium PASS 100% 688 in → 52 out (39 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4:medium__json_schema PASS 100% 721 in → 47 out (22 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4:none PASS 100% 695 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4:none__json_schema PASS 100% 741 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.4:xhigh PASS 100% 692 in → 110 out (97 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.4:xhigh__json_schema PASS 100% 740 in → 1,059 out (1,034 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.5:high PASS 100% 679 in → 59 out (46 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.5:high__json_schema PASS 100% 754 in → 428 out (403 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.5:low PASS 100% 705 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.5:low__json_schema PASS 100% 724 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.5:medium PASS 100% 701 in → 31 out (18 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.5:medium__json_schema PASS 100% 740 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.5:none PASS 100% 680 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.5:none__json_schema PASS 100% 731 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.5:xhigh PASS 100% 692 in → 86 out (73 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.5:xhigh__json_schema PASS 100% 745 in → 119 out (94 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-luna:high PASS 100% 678 in → 29 out (16 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-luna:high__json_schema PASS 100% 739 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-luna:low PASS 100% 698 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-luna:low__json_schema PASS 100% 739 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-luna:max PASS 100% 699 in → 58 out (45 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-luna:max__json_schema PASS 100% 733 in → 69 out (44 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-luna:medium PASS 100% 699 in → 29 out (16 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-luna:medium__json_schema PASS 100% 727 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-luna:none PASS 100% 687 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-luna:none__json_schema PASS 100% 739 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-luna:xhigh PASS 100% 682 in → 51 out (38 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 753 in → 273 out (248 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-sol:high PASS 100% 682 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-sol:high__json_schema PASS 100% 725 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-sol:low PASS 100% 693 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-sol:low__json_schema PASS 100% 734 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-sol:max PASS 100% 705 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-sol:max__json_schema PASS 100% 741 in → 84 out (59 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-sol:medium PASS 100% 713 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-sol:medium__json_schema PASS 100% 725 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-sol:none PASS 100% 678 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-sol:none__json_schema PASS 100% 737 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-sol:xhigh PASS 100% 700 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 741 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-terra:high PASS 100% 690 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-terra:high__json_schema PASS 100% 750 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-terra:low PASS 100% 699 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-terra:low__json_schema PASS 100% 731 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-terra:max PASS 100% 689 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-terra:max__json_schema PASS 100% 753 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-terra:medium PASS 100% 677 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-terra:medium__json_schema PASS 100% 722 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-terra:none PASS 100% 668 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-terra:none__json_schema PASS 100% 730 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-5.6-terra:xhigh PASS 100% 707 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 748 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-oss-120b:high PASS 100% 758 in → 210 out (248 reasoning)
Add authorization checks Add authentication module
openai/gpt-oss-120b:high__json_schema PASS 100% 749 in → 314 out (368 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":[ "Add authorization checks", "Add authentication module" ]}
openai/gpt-oss-120b:low PASS 100% 738 in → 45 out (36 reasoning)
Add authorization checks Add authentication module
openai/gpt-oss-120b:low__json_schema PASS 100% 738 in → 58 out (35 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
openai/gpt-oss-120b:medium PASS 100% 770 in → 99 out (106 reasoning)
Add authorization checks Add authentication module
openai/gpt-oss-120b:medium__json_schema PASS 100% 740 in → 111 out (111 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
openai/gpt-oss-120b:xhigh PASS 100% 734 in → 301 out (371 reasoning)
Add authorization checks Add authentication module
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 738 in → 253 out (294 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
openai/gpt-oss-20b:high PASS 100% 751 in → 148 out (174 reasoning)
Add authorization checks Add authentication module
openai/gpt-oss-20b:high__json_schema PASS 100% 762 in → 531 out (522 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
openai/gpt-oss-20b:low PASS 100% 737 in → 42 out (32 reasoning)
Add authorization checks Add authentication module
openai/gpt-oss-20b:low__json_schema PASS 100% 763 in → 55 out (28 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
openai/gpt-oss-20b:medium PASS 100% 750 in → 129 out (143 reasoning)
Add authorization checks Add authentication module
openai/gpt-oss-20b:medium__json_schema PASS 100% 760 in → 100 out (97 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
openai/gpt-oss-20b:xhigh PASS 100% 744 in → 473 out (530 reasoning)
Add authorization checks Add authentication module
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 748 in → 334 out (315 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
poolside/laguna-m.1:high PASS 100% 893 in → 250 out (240 reasoning)
Add authorization checks Add authentication module
poolside/laguna-m.1:high__json_schema PASS 100% 908 in → 294 out (272 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-m.1:low PASS 100% 872 in → 272 out (262 reasoning)
Add authorization checks Add authentication module
poolside/laguna-m.1:low__json_schema PASS 100% 881 in → 357 out (326 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
poolside/laguna-m.1:medium PASS 100% 889 in → 459 out (449 reasoning)
Add authorization checks Add authentication module
poolside/laguna-m.1:medium__json_schema PASS 100% 878 in → 295 out (278 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-m.1:none PASS 100% 880 in → 9 out (0 reasoning)
Add authorization checks Add authentication module
poolside/laguna-m.1:none__json_schema PASS 100% 888 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-m.1:xhigh PASS 100% 894 in → 587 out (577 reasoning)
Add authorization checks Add authentication module
poolside/laguna-m.1:xhigh__json_schema PASS 100% 899 in → 288 out (271 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-xs-2.1:high PASS 100% 888 in → 285 out (277 reasoning)
Add authorization checks Add authentication module
poolside/laguna-xs-2.1:high__json_schema PASS 100% 907 in → 311 out (294 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-xs-2.1:low PASS 100% 895 in → 434 out (426 reasoning)
Add authorization checks Add authentication module
poolside/laguna-xs-2.1:low__json_schema PASS 100% 896 in → 328 out (311 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-xs-2.1:medium PASS 100% 913 in → 404 out (396 reasoning)
Add authorization checks Add authentication module
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 886 in → 378 out (358 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-xs-2.1:none PASS 100% 864 in → 7 out (0 reasoning)
Add authorization checks Add authentication module
poolside/laguna-xs-2.1:none__json_schema PASS 100% 895 in → 19 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-xs-2.1:xhigh PASS 100% 907 in → 528 out (520 reasoning)
Add authorization checks Add authentication module
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 904 in → 320 out (303 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-xs.2:high PASS 100% 909 in → 156 out (146 reasoning)
Add authorization checks Add authentication module
poolside/laguna-xs.2:high__json_schema PASS 100% 952 in → 129 out (109 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-xs.2:low PASS 100% 914 in → 226 out (216 reasoning)
Add authorization checks Add authentication module
poolside/laguna-xs.2:low__json_schema PASS 100% 920 in → 131 out (101 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
poolside/laguna-xs.2:medium PASS 100% 946 in → 141 out (131 reasoning)
Add authorization checks Add authentication module
poolside/laguna-xs.2:medium__json_schema PASS 100% 931 in → 152 out (135 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-xs.2:none PASS 100% 940 in → 10 out (0 reasoning)
Add authorization checks Add authentication module
poolside/laguna-xs.2:none__json_schema PASS 100% 954 in → 16 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
poolside/laguna-xs.2:xhigh PASS 100% 911 in → 173 out (163 reasoning)
Add authorization checks Add authentication module
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 918 in → 153 out (136 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
qwen/qwen3.6-27b:high PASS 100% 904 in → 555 out (603 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-27b:low PASS 100% 896 in → 766 out (790 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-27b:low__json_schema PASS 100% 915 in → 32 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
qwen/qwen3.6-27b:medium PASS 100% 883 in → 902 out (957 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-27b:medium__json_schema PASS 100% 907 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
qwen/qwen3.6-27b:none PASS 100% 899 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-27b:none__json_schema PASS 100% 883 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 910 in → 505 out (539 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 896 in → 719 out (732 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 875 in → 824 out (835 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 910 in → 694 out (705 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 901 in → 807 out (846 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 899 in → 430 out (427 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 905 in → 326 out (343 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 904 in → 885 out (896 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 889 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 912 in → 21 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": ["Add authorization checks", "Add authentication module"] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 905 in → 710 out (698 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 889 in → 240 out (226 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
qwen/qwen3.6-flash:high PASS 100% 910 in → 608 out (596 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-flash:low PASS 100% 908 in → 825 out (813 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-flash:medium PASS 100% 915 in → 692 out (680 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-flash:none PASS 100% 888 in → 7 out (0 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.6-flash:xhigh PASS 100% 903 in → 647 out (635 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-flash:high PASS 100% 897 in → 743 out (731 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-flash:low PASS 100% 891 in → 441 out (429 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-flash:max PASS 100% 896 in → 467 out (455 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-flash:medium PASS 100% 906 in → 313 out (301 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-flash:none PASS 100% 892 in → 7 out (0 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-flash:xhigh PASS 100% 892 in → 693 out (681 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-max:high PASS 100% 908 in → 128 out (116 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-max:low PASS 100% 897 in → 149 out (137 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-max:medium PASS 100% 888 in → 165 out (153 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-max:none PASS 100% 889 in → 7 out (0 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-max:xhigh PASS 100% 905 in → 148 out (136 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-plus:high PASS 100% 905 in → 211 out (199 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-plus:low PASS 100% 895 in → 129 out (117 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-plus:medium PASS 100% 899 in → 368 out (356 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-plus:none PASS 100% 907 in → 7 out (0 reasoning)
Add authorization checks Add authentication module
qwen/qwen3.7-plus:xhigh PASS 100% 897 in → 180 out (168 reasoning)
Add authorization checks Add authentication module
tencent/hy3:high PASS 100% 708 in → 188 out (179 reasoning)
Add authorization checks Add authentication module
tencent/hy3:high__json_schema PASS 100% 703 in → 195 out (177 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
tencent/hy3:low PASS 100% 706 in → 226 out (217 reasoning)
Add authorization checks Add authentication module
tencent/hy3:low__json_schema PASS 100% 717 in → 221 out (209 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": ["Add authorization checks", "Add authentication module"] }
tencent/hy3:medium PASS 100% 707 in → 215 out (206 reasoning)
Add authorization checks Add authentication module
tencent/hy3:medium__json_schema PASS 100% 717 in → 236 out (232 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
tencent/hy3:none PASS 100% 726 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
tencent/hy3:none__json_schema PASS 100% 710 in → 25 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
tencent/hy3:xhigh PASS 100% 693 in → 304 out (313 reasoning)
Add authorization checks Add authentication module
tencent/hy3:xhigh__json_schema PASS 100% 722 in → 286 out (292 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": [ "Add authorization checks", "Add authentication module" ]}
thinkingmachines/inkling-small:high PASS 100% 711 in → 50 out (46 reasoning)
Add authorization checks Add authentication module
thinkingmachines/inkling-small:high__json_schema PASS 100% 702 in → 273 out (244 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
thinkingmachines/inkling-small:low PASS 100% 685 in → 64 out (51 reasoning)
Add authorization checks Add authentication module
thinkingmachines/inkling-small:low__json_schema PASS 100% 693 in → 69 out (47 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
thinkingmachines/inkling-small:max PASS 100% 684 in → 233 out (228 reasoning)
Add authorization checks Add authentication module
thinkingmachines/inkling-small:max__json_schema PASS 100% 725 in → 115 out (86 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
thinkingmachines/inkling-small:medium PASS 100% 681 in → 165 out (152 reasoning)
Add authorization checks Add authentication module
thinkingmachines/inkling-small:medium__json_schema PASS 100% 700 in → 90 out (66 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
thinkingmachines/inkling-small:xhigh PASS 100% 681 in → 192 out (204 reasoning)
Add authorization checks Add authentication module
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 689 in → 139 out (110 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
thinkingmachines/inkling:high PASS 100% 683 in → 180 out (162 reasoning)
Add authorization checks Add authentication module
thinkingmachines/inkling:low PASS 100% 692 in → 39 out (35 reasoning)
Add authorization checks Add authentication module
thinkingmachines/inkling:low__json_schema PASS 100% 692 in → 65 out (37 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
thinkingmachines/inkling:max PASS 100% 694 in → 178 out (165 reasoning)
Add authorization checks Add authentication module
thinkingmachines/inkling:max__json_schema PASS 100% 707 in → 171 out (150 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
thinkingmachines/inkling:medium PASS 100% 708 in → 124 out (111 reasoning)
Add authorization checks Add authentication module
thinkingmachines/inkling:medium__json_schema PASS 100% 708 in → 153 out (122 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
thinkingmachines/inkling:none PASS 100% 702 in → 11 out (0 reasoning)
Add authorization checks Add authentication module
thinkingmachines/inkling:xhigh PASS 100% 701 in → 149 out (160 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.3:high PASS 100% 872 in → 158 out (151 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.3:high__json_schema PASS 100% 928 in → 418 out (405 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
x-ai/grok-4.3:low PASS 100% 889 in → 155 out (148 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.3:low__json_schema PASS 100% 941 in → 318 out (295 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
x-ai/grok-4.3:max PASS 100% 868 in → 249 out (242 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.3:max__json_schema PASS 100% 931 in → 396 out (373 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
x-ai/grok-4.3:medium PASS 100% 864 in → 399 out (392 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.3:medium__json_schema PASS 100% 950 in → 389 out (376 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
x-ai/grok-4.3:none PASS 100% 849 in → 7 out (0 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.3:none__json_schema PASS 100% 933 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
x-ai/grok-4.3:xhigh PASS 100% 862 in → 185 out (178 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.3:xhigh__json_schema PASS 100% 918 in → 467 out (454 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
x-ai/grok-4.5:high PASS 100% 894 in → 89 out (82 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.5:high__json_schema PASS 100% 983 in → 244 out (228 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
x-ai/grok-4.5:low PASS 100% 907 in → 85 out (78 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.5:low__json_schema PASS 100% 978 in → 119 out (103 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
x-ai/grok-4.5:max PASS 100% 930 in → 125 out (118 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.5:max__json_schema PASS 100% 991 in → 322 out (306 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
x-ai/grok-4.5:medium PASS 100% 885 in → 139 out (132 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.5:medium__json_schema PASS 100% 993 in → 269 out (243 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
x-ai/grok-4.5:xhigh PASS 100% 895 in → 157 out (150 reasoning)
Add authorization checks Add authentication module
x-ai/grok-4.5:xhigh__json_schema PASS 100% 978 in → 277 out (261 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list":["Add authorization checks","Add authentication module"]}
z-ai/glm-4.7-flash:high PASS 100% 710 in → 508 out (462 reasoning)
Add authorization checks Add authentication module
z-ai/glm-4.7-flash:low PASS 100% 733 in → 478 out (438 reasoning)
Add authorization checks Add authentication module
z-ai/glm-4.7-flash:medium PASS 100% 739 in → 491 out (452 reasoning)
Add authorization checks Add authentication module
z-ai/glm-4.7-flash:none PASS 100% 746 in → 7 out (0 reasoning)
Add authorization checks Add authentication module
z-ai/glm-4.7-flash:none__json_schema PASS 100% 733 in → 17 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
z-ai/glm-4.7-flash:xhigh PASS 100% 729 in → 459 out (426 reasoning)
Add authorization checks Add authentication module
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 747 in → 585 out (550 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) {"commit_message_list": ["Add authorization checks", "Add authentication module"]}
z-ai/glm-5.2:high PASS 100% 751 in → 257 out (231 reasoning)
Add authorization checks Add authentication module
z-ai/glm-5.2:high__json_schema PASS 100% 740 in → 256 out (235 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
z-ai/glm-5.2:low PASS 100% 714 in → 301 out (298 reasoning)
Add authorization checks Add authentication module
z-ai/glm-5.2:low__json_schema PASS 100% 732 in → 23 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
z-ai/glm-5.2:medium PASS 100% 731 in → 205 out (186 reasoning)
Add authorization checks Add authentication module
z-ai/glm-5.2:medium__json_schema PASS 100% 736 in → 247 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
z-ai/glm-5.2:none PASS 100% 750 in → 8 out (0 reasoning)
Add authorization checks Add authentication module
z-ai/glm-5.2:none__json_schema PASS 100% 750 in → 29 out (0 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
z-ai/glm-5.2:xhigh PASS 100% 741 in → 268 out (248 reasoning)
Add authorization checks Add authentication module
z-ai/glm-5.2:xhigh__json_schema PASS 100% 758 in → 139 out (142 reasoning)
Add authorization checks Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
arcee-ai/trinity-mini:high FAIL 50% 686 in → 358 out (404 reasoning)
Addauthorization checks Add authentication module
Failure: Missing lines: ['Add authorization checks']; Extra lines: ['Addauthorization checks']
arcee-ai/trinity-mini:medium FAIL 50% 676 in → 388 out (456 reasoning)
Addauthorization checks Add authentication module
Failure: Missing lines: ['Add authorization checks']; Extra lines: ['Addauthorization checks']
arcee-ai/trinity-mini:xhigh FAIL 50% 675 in → 398 out (466 reasoning)
Addauthorization checks Add authentication module
Failure: Missing lines: ['Add authorization checks']; Extra lines: ['Addauthorization checks']
nvidia/nemotron-3-nano-30b-a3b:high FAIL 50% 913 in → 93 out (112 reasoning)
Addauthorization checks Add authentication module
Failure: Missing lines: ['Add authorization checks']; Extra lines: ['Addauthorization checks']
thinkingmachines/inkling-small:none FAIL 50% 701 in → 10 out (0 reasoning)
Add authorization checks Add authentication module<|end_message|>
Failure: Missing lines: ['Add authentication module']; Extra lines: ['Add authentication module<|end_message|>']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: {{ "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
79946c5e95355fa372c7f1be48fe65c76e512822 Test User test@test.com 2026-06-30 Add authorization checks b6f33232e2b1173dad954137d983984acf77e1a5 Test User test@test.com 2026-06-30 Add authentication module
JSON Schema Structured Output
(raw) { "commit_message_list": [ "79946c5e95355fa372c7f1be48fe65c76e512822 Test User test@test.com 2026-06-30 Add authorization checks", "b6f33232e2b1173dad954137d983984acf77e1a5 Test User test@test.com 2026-06-30 Add authentication module" ] }
Failure: Missing lines: ['Add authentication module', 'Add authorization checks']; Extra lines: ['79946c5e95355fa372c7f1be48fe65c76e512822 Test User test@test.com 2026-06-30 Add authorization checks', 'b6f33232e2b1173dad954137d983984acf77e1a5 Test User test@test.com 2026-06-30 Add authentication module']
Invalid JSON. Output: Add authorization checks Add authentication module
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Add authorization checks Add authentication module
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b92aab1e-fa3b-9f1c-a760-9e5281cca9c7","request_id":"b92aab1e-fa3b-9f1c-a760-9e5281cca9c7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-49adf87c-0661-94ac-9da6-552c9e856077","request_id":"49adf87c-0661-94ac-9da6-552c9e856077"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8ac1f9fb-3d07-9231-9e3f-2b7e55e15362","request_id":"8ac1f9fb-3d07-9231-9e3f-2b7e55e15362"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8644e595-2c65-984a-b548-eda57ae61aec","request_id":"8644e595-2c65-984a-b548-eda57ae61aec"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-371294f2-38f8-9472-81a0-3a2c8103f290","request_id":"371294f2-38f8-9472-81a0-3a2c8103f290"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4c2df6a4-536f-974d-b9a4-3532cef03807"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c82b552d-50a6-9654-95e9-68b288849e8a"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-261a9393-43b8-95a4-8ff6-0d02f2a369f4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f17d310e-8cf0-9498-8c36-b9c2d2991cc3"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-06f12e53-823a-96a6-a881-d1c78832ac7a"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3eb41b88-b05c-94be-b3c4-f55277c7d704"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a0bea53d-1fec-9b9f-be93-8b21f32a345c","request_id":"a0bea53d-1fec-9b9f-be93-8b21f32a345c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e6be1ad7-a6b7-97ce-b34d-69b2af4e4b0b","request_id":"e6be1ad7-a6b7-97ce-b34d-69b2af4e4b0b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-346dd913-b868-9ffd-a434-68859c82d7d0","request_id":"346dd913-b868-9ffd-a434-68859c82d7d0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3bd67425-c96c-9a05-98bb-8848acfe5f9f","request_id":"3bd67425-c96c-9a05-98bb-8848acfe5f9f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5cf1a73e-80d3-982c-8293-fe241c9f022d","request_id":"5cf1a73e-80d3-982c-8293-fe241c9f022d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-12ac5383-ecdf-9649-b7dc-0027cd6a3173","request_id":"12ac5383-ecdf-9649-b7dc-0027cd6a3173"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6d44984a-dbd3-9c76-b862-a697f7b6aa2b","request_id":"6d44984a-dbd3-9c76-b862-a697f7b6aa2b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-04348627-715b-9e23-bc20-973ec703da83","request_id":"04348627-715b-9e23-bc20-973ec703da83"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f6d112ea-6d5f-9ce7-9257-4e8a1ba867dc","request_id":"f6d112ea-6d5f-9ce7-9257-4e8a1ba867dc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-680465f4-63f3-94d7-a7d0-5115d1bfa01d","request_id":"680465f4-63f3-94d7-a7d0-5115d1bfa01d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: Add authorization checks Add authentication module{ "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Add authorization checks Add authentication module
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Add authorization checks Add authentication module{ "commit_message_list": [ "Add authorization checks", "Add authentication module" ] }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Add authorization checks Add authentication module
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Add authorization checks Add authentication module
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)