f012 — commit_messages

← commit_messages
Documentation update
Tests ability to generate a commit message for a documentation update (README). Evaluates recognizing documentation-only changes.
easy commit-message documentation docs readme
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf '# Project\nThis is a project.\n' > README.md
  5. 05 git add README.md
  6. 06 git commit -m 'Initial commit'
  7. 07 printf '# Project\nThis is a project.\n\n## Installation\nRun `pip install` to install dependencies.\n' > README.md
  8. 08 git add README.md
Prompt
Generate a concise commit message (max 50 characters) describing the changes in this diff. Focus on the action taken (e.g., 'Add', 'Fix', 'Update').
Expected
Add installation section to README
Fixture Metrics

Scoped model quality, cost, API time, and token usage for commit_messages/f012.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
google/gemini-3.6-flash:xhigh__json_schema PASS 97.5% 131 in → 239 out (228 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
deepseek/deepseek-v4-flash:medium PASS 95.7% 120 in → 128 out (121 reasoning)
Add installation instructions
moonshotai/kimi-k2.7-code:high PASS 95.3% 119 in → 310 out (260 reasoning)
Add installation instructions to README
google/gemini-3.5-flash-lite:medium PASS 95% 131 in → 245 out (240 reasoning)
Add installation instructions to README
thinkingmachines/inkling-small:medium PASS 95% 130 in → 107 out (114 reasoning)
Add installation instructions to README
z-ai/glm-4.7-flash:low PASS 95% 120 in → 378 out (418 reasoning)
Add installation instructions to README
google/gemma-4-26b-a4b-it:medium__json_schema PASS 94.7% 146 in → 362 out (357 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
openai/gpt-5.4-mini:xhigh PASS 94.3% 121 in → 228 out (217 reasoning)
Add installation instructions to README
openai/gpt-5.6-sol:xhigh PASS 94% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-terra:high__json_schema PASS 94% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
qwen/qwen3.6-27b:xhigh__json_schema PASS 93.7% 135 in → 1,460 out (1,408 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
anthropic/claude-fable-5:low PASS 93.3% 170 in → 9 out (0 reasoning)
Add installation section to README
anthropic/claude-opus-4.6:low PASS 93.3% 142 in → 8 out (0 reasoning)
Add installation section to README
anthropic/claude-sonnet-4.6:none PASS 93.3% 142 in → 8 out (0 reasoning)
Add installation instructions to README
arcee-ai/trinity-mini:low PASS 93.3% 119 in → 199 out (230 reasoning)
Add installation instructions
arcee-ai/trinity-mini:medium PASS 93.3% 119 in → 238 out (284 reasoning)
Add installation instructions
deepseek/deepseek-v4-flash-0731:medium PASS 93.3% 120 in → 245 out (222 reasoning)
Add installation instructions to README
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 93.3% 120 in → 161 out (164 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 93.3% 199 in → 503 out (603 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions"}
deepseek/deepseek-v4-pro:high PASS 93.3% 120 in → 46 out (39 reasoning)
Add installation instructions to README
google/gemini-3-flash-preview:medium__json_schema PASS 93.3% 193 in → 291 out (280 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
google/gemini-3.5-flash-lite:high__json_schema PASS 93.3% 193 in → 480 out (463 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
google/gemini-3.5-flash:xhigh__json_schema PASS 93.3% 193 in → 456 out (444 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
google/gemma-4-26b-a4b-it:medium PASS 93.3% 146 in → 540 out (534 reasoning)
Add installation instructions to README
google/gemma-4-31b-it:low PASS 93.3% 146 in → 194 out (1 reasoning)
Add installation section to README
google/gemma-4-31b-it:medium__json_schema PASS 93.3% 146 in → 233 out (200 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
minimax/minimax-m2.7:medium__json_schema PASS 93.3% 285 in → 169 out (156 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
minimax/minimax-m3:medium PASS 93.3% 235 in → 119 out (112 reasoning)
Add installation section to README
openai/gpt-5.4-mini:high__json_schema PASS 93.3% 159 in → 162 out (144 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions"}
openai/gpt-5.4-mini:none__json_schema PASS 93.3% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.4:xhigh PASS 93.3% 121 in → 181 out (170 reasoning)
Add installation instructions to README
openai/gpt-5.4:xhigh__json_schema PASS 93.3% 159 in → 177 out (157 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
openai/gpt-5.5:high__json_schema PASS 93.3% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-sol:medium PASS 93.3% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-sol:none__json_schema PASS 93.3% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
qwen/qwen3.6-35b-a3b:high__json_schema PASS 93.3% 135 in → 1,517 out (1,391 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
qwen/qwen3.7-flash:low PASS 93.3% 135 in → 1,092 out (1,082 reasoning)
Add installation instructions to README
thinkingmachines/inkling-small:medium__json_schema PASS 93.3% 130 in → 189 out (173 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
thinkingmachines/inkling:none PASS 93.3% 128 in → 9 out (0 reasoning)
Add installation section to README
x-ai/grok-4.3:max PASS 93.3% 295 in → 406 out (401 reasoning)
Add installation section to README
z-ai/glm-4.7-flash:high PASS 93.3% 120 in → 420 out (444 reasoning)
Add installation instructions
anthropic/claude-sonnet-5:xhigh__json_schema PASS 93% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
moonshotai/kimi-k2.6:medium PASS 93% 118 in → 199 out (192 reasoning)
Add installation section to README
openai/gpt-5.5:xhigh PASS 93% 121 in → 154 out (143 reasoning)
Add installation instructions to README
anthropic/claude-opus-4.6:none PASS 92.7% 142 in → 8 out (0 reasoning)
Add installation section to README
anthropic/claude-opus-4.7:medium__json_schema PASS 92.7% 396 in → 23 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-opus-4.8:high__json_schema PASS 92.7% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
anthropic/claude-sonnet-4.6:low__json_schema PASS 92.7% 312 in → 14 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-sonnet-4.6:max PASS 92.7% 142 in → 48 out (33 reasoning)
Add installation instructions to README
anthropic/claude-sonnet-5:high PASS 92.7% 170 in → 11 out (0 reasoning)
Add installation instructions to README
google/gemini-3-flash-preview:medium PASS 92.7% 130 in → 193 out (188 reasoning)
Add installation instructions to README
openai/gpt-5.4:medium PASS 92.7% 121 in → 51 out (40 reasoning)
Add installation instructions to README
openai/gpt-oss-20b:high__json_schema PASS 92.7% 180 in → 519 out (516 reasoning)
Add installation instructions to README.md
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README.md" }
qwen/qwen3.6-27b:high__json_schema PASS 92.7% 135 in → 1,400 out (1,287 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
qwen/qwen3.6-35b-a3b:low__json_schema PASS 92.7% 135 in → 1,301 out (1,175 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
qwen/qwen3.6-flash:medium PASS 92.7% 135 in → 1,154 out (1,144 reasoning)
Add installation instructions to README
thinkingmachines/inkling-small:low__json_schema PASS 92.7% 130 in → 60 out (44 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
google/gemma-4-26b-a4b-it:high__json_schema PASS 92.5% 146 in → 533 out (545 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
tencent/hy3:xhigh__json_schema PASS 92.5% 127 in → 1,469 out (1,169 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
x-ai/grok-4.5:xhigh__json_schema PASS 92.5% 395 in → 244 out (232 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add Installation section to README"}
anthropic/claude-opus-4.8:medium__json_schema PASS 92.3% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
google/gemma-4-31b-it:xhigh__json_schema PASS 92.3% 146 in → 233 out (221 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
minimax/minimax-m2.7:low PASS 92.3% 149 in → 190 out (226 reasoning)
Add Installation section to README
minimax/minimax-m3:low PASS 92.3% 287 in → 135 out (164 reasoning)
Add installation instructions to README
mistralai/mistral-medium-3-5:low__json_schema PASS 92.3% 121 in → 226 out (256 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions"}
openai/gpt-5.4:low__json_schema PASS 92.3% 159 in → 36 out (16 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
poolside/laguna-xs-2.1:high PASS 92.3% 135 in → 289 out (283 reasoning)
Add Installation section to README
qwen/qwen3.7-max:none PASS 92.3% 137 in → 5 out (0 reasoning)
Add installation instructions to README
anthropic/claude-fable-5:low__json_schema PASS 92% 391 in → 19 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
anthropic/claude-haiku-4.5:low PASS 92% 171 in → 205 out (192 reasoning)
Add installation instructions
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 92% 120 in → 17 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
deepseek/deepseek-v4-flash:none PASS 92% 120 in → 6 out (0 reasoning)
Add installation instructions to README
google/gemini-3.1-pro-preview:medium__json_schema PASS 92% 193 in → 103 out (91 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
google/gemini-3.5-flash-lite:medium__json_schema PASS 92% 131 in → 369 out (352 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
google/gemini-3.6-flash:max PASS 92% 131 in → 340 out (335 reasoning)
Add installation instructions to README
moonshotai/kimi-k2.6:xhigh PASS 92% 119 in → 328 out (326 reasoning)
Add installation instructions to README
openai/gpt-5.4-nano:low__json_schema PASS 92% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.5:medium PASS 92% 121 in → 104 out (93 reasoning)
Add installation instructions to README
poolside/laguna-xs.2:high PASS 92% 172 in → 213 out (205 reasoning)
Add installation instructions to README
poolside/laguna-xs.2:none PASS 92% 172 in → 7 out (0 reasoning)
Add installation instructions to README
qwen/qwen3.7-plus:low PASS 92% 135 in → 421 out (411 reasoning)
Add installation instructions to README
x-ai/grok-4.3:none PASS 92% 293 in → 5 out (0 reasoning)
Add installation section to README
anthropic/claude-fable-5:xhigh__json_schema PASS 91.7% 391 in → 19 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
anthropic/claude-haiku-4.5:high PASS 91.7% 171 in → 281 out (239 reasoning)
Add installation instructions
anthropic/claude-opus-4.6:max__json_schema PASS 91.7% 312 in → 46 out (19 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
anthropic/claude-opus-4.7:max__json_schema PASS 91.7% 396 in → 23 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-opus-4.8:low__json_schema PASS 91.7% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-opus-4.8:medium PASS 91.7% 170 in → 11 out (0 reasoning)
Add installation instructions to README
anthropic/claude-sonnet-4.6:max__json_schema PASS 91.7% 312 in → 64 out (42 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-sonnet-4.6:medium PASS 91.7% 142 in → 8 out (0 reasoning)
Add installation instructions to README
anthropic/claude-sonnet-4.6:medium__json_schema PASS 91.7% 312 in → 14 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-sonnet-4.6:none__json_schema PASS 91.7% 312 in → 14 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-sonnet-5:high__json_schema PASS 91.7% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-sonnet-5:low PASS 91.7% 170 in → 11 out (0 reasoning)
Add installation instructions to README
anthropic/claude-sonnet-5:medium PASS 91.7% 170 in → 11 out (0 reasoning)
Add installation instructions to README
arcee-ai/trinity-large-thinking:low PASS 91.7% 119 in → 1,260 out (1,258 reasoning)
Add installation instructions
arcee-ai/trinity-mini:high__json_schema PASS 91.7% 119 in → 197 out (222 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions" }
arcee-ai/trinity-mini:low__json_schema PASS 91.7% 119 in → 418 out (488 reasoning)
Add installation section
JSON Schema Structured Output
(raw) { "commit_message":"Add installation section" }
arcee-ai/trinity-mini:xhigh PASS 91.7% 119 in → 276 out (325 reasoning)
Add installation guide
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 91.7% 120 in → 58 out (49 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
deepseek/deepseek-v4-flash:none__json_schema PASS 91.7% 188 in → 13 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 91.7% 120 in → 13 out (0 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions" }
google/gemini-3-flash-preview:high PASS 91.7% 130 in → 227 out (222 reasoning)
Add installation instructions to README
google/gemini-3-flash-preview:low PASS 91.7% 130 in → 231 out (226 reasoning)
Add installation instructions to README
google/gemini-3-flash-preview:low__json_schema PASS 91.7% 193 in → 291 out (280 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
google/gemini-3-flash-preview:xhigh__json_schema PASS 91.7% 193 in → 247 out (235 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
google/gemini-3.1-flash-lite-preview:high PASS 91.7% 130 in → 340 out (335 reasoning)
Add installation instructions to README
google/gemini-3.1-flash-lite-preview:medium PASS 91.7% 131 in → 221 out (216 reasoning)
Add installation instructions to README
google/gemini-3.1-pro-preview:low__json_schema PASS 91.7% 193 in → 108 out (91 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
google/gemini-3.5-flash-lite:max PASS 91.7% 130 in → 319 out (314 reasoning)
Add installation instructions to README
google/gemini-3.5-flash-lite:xhigh PASS 91.7% 131 in → 318 out (313 reasoning)
Add installation instructions to README
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 91.7% 131 in → 345 out (335 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions"}
google/gemini-3.5-flash:high PASS 91.7% 130 in → 322 out (317 reasoning)
Add installation instructions to README
google/gemini-3.5-flash:high__json_schema PASS 91.7% 193 in → 397 out (385 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
google/gemini-3.5-flash:low__json_schema PASS 91.7% 193 in → 83 out (72 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
google/gemini-3.5-flash:medium__json_schema PASS 91.7% 193 in → 345 out (334 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
google/gemini-3.6-flash:max__json_schema PASS 91.7% 193 in → 318 out (307 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
google/gemini-3.6-flash:medium__json_schema PASS 91.7% 193 in → 188 out (177 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
google/gemini-3.6-flash:xhigh PASS 91.7% 131 in → 341 out (336 reasoning)
Add installation instructions to README
google/gemma-4-26b-a4b-it:none__json_schema PASS 91.7% 143 in → 15 out (0 reasoning)
Add installation section to README.md
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README.md"}
google/gemma-4-26b-a4b-it:xhigh PASS 91.7% 146 in → 277 out (278 reasoning)
Add installation instructions to README
google/gemma-4-31b-it:medium PASS 91.7% 146 in → 157 out (158 reasoning)
Add installation instructions to README
google/gemma-4-31b-it:xhigh PASS 91.7% 146 in → 231 out (218 reasoning)
Add installation instructions to README
liquid/lfm-2-24b-a2b PASS 91.7% 132 in → 4 out
Add installation instructions
minimax/minimax-m2.5:high PASS 91.7% 149 in → 128 out (147 reasoning)
Add Installation section to README
minimax/minimax-m2.5:low PASS 91.7% 149 in → 120 out (148 reasoning)
Add Installation section to README
minimax/minimax-m2.5:low__json_schema PASS 91.7% 149 in → 132 out (118 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
minimax/minimax-m2.5:medium__json_schema PASS 91.7% 164 in → 136 out (144 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
minimax/minimax-m2.5:xhigh__json_schema PASS 91.7% 149 in → 100 out (86 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
minimax/minimax-m3:low__json_schema PASS 91.7% 274 in → 15 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
minimax/minimax-m3:medium__json_schema PASS 91.7% 235 in → 12 out (11 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
minimax/minimax-m3:xhigh PASS 91.7% 287 in → 54 out (63 reasoning)
Add installation instructions to README
minimax/minimax-m3:xhigh__json_schema PASS 91.7% 274 in → 15 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
mistralai/mistral-medium-3-5:high PASS 91.7% 133 in → 342 out (411 reasoning)
Add installation instructions
mistralai/mistral-medium-3-5:medium PASS 91.7% 133 in → 265 out (319 reasoning)
Add installation instructions
mistralai/mistral-medium-3-5:none PASS 91.7% 133 in → 6 out (0 reasoning)
"Add installation instructions"
moonshotai/kimi-k2.6:high PASS 91.7% 119 in → 272 out (281 reasoning)
Add installation instructions to README
moonshotai/kimi-k2.6:medium__json_schema PASS 91.7% 119 in → 566 out (554 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
moonshotai/kimi-k2.6:xhigh__json_schema PASS 91.7% 119 in → 303 out (341 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 91.7% 134 in → 134 out (113 reasoning)
Add Installation section to README
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 91.7% 134 in → 304 out (229 reasoning)
Add Installation section
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section" }
openai/gpt-5.4-mini:low PASS 91.7% 121 in → 26 out (15 reasoning)
Add installation instructions to README
openai/gpt-5.4:high PASS 91.7% 121 in → 105 out (94 reasoning)
Add installation instructions to README
openai/gpt-5.5:low PASS 91.7% 121 in → 24 out (13 reasoning)
Add installation instructions to README
openai/gpt-5.5:none__json_schema PASS 91.7% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.5:xhigh__json_schema PASS 91.7% 159 in → 75 out (57 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions"}
openai/gpt-5.6-luna:max__json_schema PASS 91.7% 159 in → 114 out (94 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-luna:xhigh__json_schema PASS 91.7% 159 in → 16 out (0 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions"}
openai/gpt-5.6-sol:max__json_schema PASS 91.7% 159 in → 61 out (41 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-terra:none PASS 91.7% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-terra:none__json_schema PASS 91.7% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-oss-120b:medium__json_schema PASS 91.7% 169 in → 197 out (161 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add Installation section to README"}
openai/gpt-oss-20b:low PASS 91.7% 184 in → 37 out (23 reasoning)
Add installation section to README.md
poolside/laguna-m.1:low__json_schema PASS 91.7% 135 in → 188 out (170 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
poolside/laguna-m.1:medium__json_schema PASS 91.7% 135 in → 253 out (236 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
poolside/laguna-xs-2.1:high__json_schema PASS 91.7% 135 in → 372 out (359 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
poolside/laguna-xs-2.1:none PASS 91.7% 135 in → 12 out (0 reasoning)
Add installation instructions to README </think>Add installation instructions to README
poolside/laguna-xs.2:xhigh PASS 91.7% 172 in → 355 out (347 reasoning)
Add Installation section to README
poolside/laguna-xs.2:xhigh__json_schema PASS 91.7% 172 in → 197 out (184 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
qwen/qwen3.6-27b:low PASS 91.7% 135 in → 1,380 out (1,139 reasoning)
Add installation instructions to README
qwen/qwen3.6-27b:none PASS 91.7% 137 in → 5 out (0 reasoning)
Add installation instructions to README
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 91.7% 135 in → 1,373 out (1,112 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
qwen/qwen3.7-flash:xhigh PASS 91.7% 135 in → 978 out (968 reasoning)
Add installation instructions to README
qwen/qwen3.7-plus:none PASS 91.7% 137 in → 5 out (0 reasoning)
Add installation instructions to README
tencent/hy3:high PASS 91.7% 127 in → 1,178 out (900 reasoning)
Add installation instructions to README
tencent/hy3:medium__json_schema PASS 91.7% 127 in → 623 out (417 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section to README" }
thinkingmachines/inkling:max PASS 91.7% 130 in → 181 out (170 reasoning)
Add installation instructions to README
thinkingmachines/inkling:xhigh PASS 91.7% 130 in → 113 out (128 reasoning)
Add installation section to README
x-ai/grok-4.3:low PASS 91.7% 301 in → 274 out (269 reasoning)
Add installation section to README
x-ai/grok-4.3:xhigh__json_schema PASS 91.7% 355 in → 408 out (398 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
x-ai/grok-4.5:low__json_schema PASS 91.7% 395 in → 90 out (78 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
x-ai/grok-4.5:medium__json_schema PASS 91.7% 395 in → 222 out (210 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
z-ai/glm-4.7-flash:none PASS 91.7% 120 in → 4 out (0 reasoning)
Add installation instructions
z-ai/glm-5.2:high PASS 91.7% 127 in → 175 out (183 reasoning)
Add installation instructions to README
anthropic/claude-haiku-4.5:medium PASS 91.3% 171 in → 318 out (274 reasoning)
Add installation instructions
anthropic/claude-opus-4.7:low__json_schema PASS 91.3% 396 in → 23 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-sonnet-4.6:xhigh PASS 91.3% 142 in → 8 out (0 reasoning)
Add installation instructions to README
anthropic/claude-sonnet-5:medium__json_schema PASS 91.3% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
arcee-ai/trinity-mini:medium__json_schema PASS 91.3% 119 in → 199 out (224 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions" }
google/gemini-3.1-flash-lite-preview:low PASS 91.3% 131 in → 113 out (108 reasoning)
Add installation instructions to README
minimax/minimax-m2.5:xhigh PASS 91.3% 149 in → 162 out (204 reasoning)
Add installation instructions to README
mistralai/mistral-medium-3-5:high__json_schema PASS 91.3% 121 in → 226 out (257 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions"}
openai/gpt-5.5:medium__json_schema PASS 91.3% 159 in → 46 out (26 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
poolside/laguna-m.1:high PASS 91.3% 135 in → 292 out (284 reasoning)
Add installation section to README
qwen/qwen3.6-27b:high PASS 91.3% 135 in → 1,070 out (1,023 reasoning)
Add installation section to README
qwen/qwen3.6-35b-a3b:none PASS 91.3% 137 in → 6 out (0 reasoning)
Add installation instructions to README
thinkingmachines/inkling-small:xhigh PASS 91.3% 130 in → 242 out (231 reasoning)
Add installation section to README
thinkingmachines/inkling:max__json_schema PASS 91.3% 130 in → 147 out (131 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
x-ai/grok-4.5:max__json_schema PASS 91.3% 395 in → 294 out (282 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
z-ai/glm-4.7-flash:low__json_schema PASS 91.3% 120 in → 536 out (558 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions"}
z-ai/glm-4.7-flash:medium PASS 91.3% 120 in → 297 out (321 reasoning)
Add installation instructions
z-ai/glm-5.2:low PASS 91.3% 127 in → 172 out (183 reasoning)
Add installation instructions to README
z-ai/glm-5.2:medium PASS 91.3% 127 in → 203 out (197 reasoning)
Add install instructions to README
z-ai/glm-5.2:xhigh PASS 91.3% 120 in → 350 out (335 reasoning)
Add installation section to README
anthropic/claude-haiku-4.5:none PASS 91% 142 in → 8 out (0 reasoning)
Add installation instructions to README
anthropic/claude-opus-4.6:medium__json_schema PASS 91% 312 in → 14 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
anthropic/claude-opus-4.7:max PASS 91% 175 in → 84 out (45 reasoning)
Add Installation section to README
anthropic/claude-opus-4.7:medium PASS 91% 175 in → 13 out (0 reasoning)
Add installation instructions to README
anthropic/claude-opus-4.7:xhigh PASS 91% 175 in → 13 out (0 reasoning)
Add installation instructions to README
anthropic/claude-opus-4.8:low PASS 91% 170 in → 11 out (0 reasoning)
Add installation instructions to README
anthropic/claude-opus-4.8:none PASS 91% 170 in → 11 out (0 reasoning)
Add installation instructions to README
anthropic/claude-sonnet-4.6:high__json_schema PASS 91% 312 in → 14 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
deepseek/deepseek-v4-flash-0731:max PASS 91% 212 in → 261 out (189 reasoning)
Add installation instructions to README
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 91% 207 in → 337 out (323 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
deepseek/deepseek-v4-flash-0731:xhigh PASS 91% 199 in → 81 out (78 reasoning)
Add installation instructions to README
deepseek/deepseek-v4-pro:high__json_schema PASS 91% 120 in → 15 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
google/gemini-3.1-pro-preview:medium PASS 91% 130 in → 287 out (282 reasoning)
Add installation instructions to README
google/gemma-4-26b-a4b-it:high PASS 91% 145 in → 428 out (411 reasoning)
Add installation instructions to README
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 91% 146 in → 597 out (549 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
google/gemma-4-31b-it:low__json_schema PASS 91% 146 in → 257 out (233 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
google/gemma-4-31b-it:none__json_schema PASS 91% 143 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
ibm-granite/granite-4.1-8b__json_schema PASS 91% 133 in → 13 out
Add installation instructions
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions" }
minimax/minimax-m2.7:low__json_schema PASS 91% 285 in → 236 out (224 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
minimax/minimax-m2.7:xhigh PASS 91% 149 in → 186 out (218 reasoning)
Add Installation section
minimax/minimax-m2.7:xhigh__json_schema PASS 91% 149 in → 17 out (16 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
mistralai/devstral-2512__json_schema PASS 91% 121 in → 12 out
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
mistralai/mistral-medium-3-5:low PASS 91% 133 in → 5,699 out (5,669 reasoning)
Add installation section
moonshotai/kimi-k2.6:low__json_schema PASS 91% 118 in → 722 out (365 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
moonshotai/kimi-k2.7-code:high__json_schema PASS 91% 206 in → 134 out (122 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
moonshotai/kimi-k2.7-code:xhigh PASS 91% 119 in → 62 out (52 reasoning)
Add installation section to README (35)
nvidia/nemotron-3-super-120b-a12b:high PASS 91% 134 in → 331 out (264 reasoning)
Add Installation section to README
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 91% 134 in → 16 out (0 reasoning)
Add Installation section to README.md
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section to README.md" }
openai/gpt-5.4-mini:high PASS 91% 121 in → 87 out (78 reasoning)
Add installation instructions
openai/gpt-5.4-mini:xhigh__json_schema PASS 91% 159 in → 118 out (100 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions"}
openai/gpt-5.4-nano:high__json_schema PASS 91% 159 in → 18 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
openai/gpt-5.4:medium__json_schema PASS 91% 159 in → 98 out (78 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.5:none PASS 91% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-luna:high PASS 91% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-luna:low PASS 91% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-luna:low__json_schema PASS 91% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-luna:max PASS 91% 121 in → 142 out (131 reasoning)
Add installation instructions to README
openai/gpt-5.6-luna:medium__json_schema PASS 91% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-luna:none PASS 91% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-sol:max PASS 91% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-terra:medium__json_schema PASS 91% 159 in → 44 out (24 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-terra:xhigh PASS 91% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-terra:xhigh__json_schema PASS 91% 159 in → 52 out (32 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-oss-120b:high__json_schema PASS 91% 182 in → 518 out (502 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
poolside/laguna-m.1:low PASS 91% 135 in → 370 out (362 reasoning)
Add installation instructions to README
poolside/laguna-m.1:medium PASS 91% 135 in → 256 out (247 reasoning)
Add installation instructions to README.md
poolside/laguna-m.1:none PASS 91% 135 in → 7 out (0 reasoning)
Add installation instructions to README
poolside/laguna-xs-2.1:medium PASS 91% 135 in → 222 out (216 reasoning)
Add Installation section to README
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 91% 135 in → 899 out (849 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
thinkingmachines/inkling-small:xhigh__json_schema PASS 91% 130 in → 148 out (132 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
z-ai/glm-5.2:medium__json_schema PASS 91% 306 in → 109 out (93 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
z-ai/glm-5.2:xhigh__json_schema PASS 91% 129 in → 514 out (499 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
anthropic/claude-opus-4.6:xhigh__json_schema PASS 90.7% 312 in → 14 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
anthropic/claude-opus-4.7:high PASS 90.7% 175 in → 13 out (0 reasoning)
Add installation instructions to README
deepseek/deepseek-v4-flash:high PASS 90.7% 120 in → 156 out (148 reasoning)
Add installation section to README
deepseek/deepseek-v4-flash:low PASS 90.7% 120 in → 216 out (208 reasoning)
Add installation instructions to README
openai/gpt-5.4-nano:low PASS 90.7% 121 in → 9 out (0 reasoning)
Add installation section to README
qwen/qwen3.6-35b-a3b:low PASS 90.7% 135 in → 1,239 out (1,115 reasoning)
Add installation instructions to README
qwen/qwen3.7-flash:medium PASS 90.7% 135 in → 1,158 out (1,148 reasoning)
Add installation instructions to README
thinkingmachines/inkling-small:high PASS 90.7% 130 in → 411 out (301 reasoning)
Add installation instructions to README
thinkingmachines/inkling:medium PASS 90.7% 130 in → 179 out (167 reasoning)
Add installation instructions to README
z-ai/glm-4.7-flash:xhigh PASS 90.7% 120 in → 604 out (659 reasoning)
Add installation instructions to README
anthropic/claude-haiku-4.5:high__json_schema PASS 90.3% 341 in → 409 out (334 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add Installation section to README"}
anthropic/claude-opus-4.6:high PASS 90.3% 142 in → 8 out (0 reasoning)
Add installation section to README
anthropic/claude-opus-4.6:low__json_schema PASS 90.3% 312 in → 14 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
anthropic/claude-opus-4.7:low PASS 90.3% 175 in → 13 out (0 reasoning)
Add installation instructions to README
anthropic/claude-opus-4.7:xhigh__json_schema PASS 90.3% 396 in → 23 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-opus-4.8:high PASS 90.3% 170 in → 11 out (0 reasoning)
Add installation instructions to README
arcee-ai/trinity-large-thinking:xhigh PASS 90.3% 119 in → 697 out (692 reasoning)
Add installation section to README
google/gemini-3.6-flash:low PASS 90.3% 131 in → 186 out (181 reasoning)
Add installation instructions to README
google/gemma-4-31b-it:high PASS 90.3% 146 in → 208 out (189 reasoning)
Add installation instructions to README
minimax/minimax-m2.7:high PASS 90.3% 149 in → 178 out (216 reasoning)
Add Installation section to README
openai/gpt-5.4-nano:none__json_schema PASS 90.3% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
poolside/laguna-xs-2.1:xhigh__json_schema PASS 90.3% 135 in → 197 out (184 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add Installation section to README"}
x-ai/grok-4.3:high PASS 90.3% 295 in → 447 out (442 reasoning)
Add installation section to README
anthropic/claude-fable-5:high__json_schema PASS 90% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
anthropic/claude-fable-5:medium__json_schema PASS 90% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
anthropic/claude-haiku-4.5:medium__json_schema PASS 90% 341 in → 278 out (221 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
anthropic/claude-opus-4.6:high__json_schema PASS 90% 312 in → 14 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
anthropic/claude-opus-4.6:max PASS 90% 142 in → 40 out (19 reasoning)
Add installation section to README
anthropic/claude-opus-4.6:medium PASS 90% 142 in → 8 out (0 reasoning)
Add installation section to README
anthropic/claude-opus-4.8:max PASS 90% 170 in → 129 out (41 reasoning)
Add installation section to README
anthropic/claude-opus-4.8:max__json_schema PASS 90% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
anthropic/claude-opus-4.8:none__json_schema PASS 90% 391 in → 18 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
anthropic/claude-opus-4.8:xhigh__json_schema PASS 90% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 90% 312 in → 14 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-sonnet-5:none PASS 90% 170 in → 11 out (0 reasoning)
Add installation instructions to README
anthropic/claude-sonnet-5:xhigh PASS 90% 170 in → 11 out (0 reasoning)
Add installation instructions to README
arcee-ai/trinity-mini:xhigh__json_schema PASS 90% 119 in → 204 out (226 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions" }
deepseek/deepseek-v4-flash-0731:low PASS 90% 120 in → 78 out (70 reasoning)
Add installation instructions to README
deepseek/deepseek-v4-flash:high__json_schema PASS 90% 188 in → 143 out (152 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
deepseek/deepseek-v4-pro:low__json_schema PASS 90% 122 in → 199 out (183 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
deepseek/deepseek-v4-pro:medium__json_schema PASS 90% 122 in → 199 out (183 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
google/gemini-3-flash-preview:high__json_schema PASS 90% 193 in → 206 out (195 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 90% 131 in → 126 out (109 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 90% 193 in → 355 out (344 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
google/gemini-3.1-pro-preview:high PASS 90% 130 in → 377 out (372 reasoning)
Add installation instructions to README
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 90% 193 in → 138 out (126 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
google/gemini-3.5-flash-lite:high PASS 90% 130 in → 401 out (396 reasoning)
Add installation instructions to README
google/gemini-3.5-flash-lite:max__json_schema PASS 90% 193 in → 325 out (313 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
google/gemini-3.5-flash:low PASS 90% 130 in → 213 out (208 reasoning)
Add installation instructions to README
google/gemini-3.5-flash:xhigh PASS 90% 130 in → 365 out (360 reasoning)
Add installation instructions to README
google/gemini-3.6-flash:high PASS 90% 131 in → 331 out (326 reasoning)
Add installation instructions to README
inclusionai/ling-2.6-flash PASS 90% 145 in → 6 out
Add installation instructions to README
minimax/minimax-m2.5:medium PASS 90% 152 in → 97 out (91 reasoning)
Add Installation section to README
minimax/minimax-m2.7:high__json_schema PASS 90% 285 in → 355 out (343 reasoning)
Add Installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add Installation instructions to README"}
mistralai/mistral-medium-3-5:none__json_schema PASS 90% 121 in → 12 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 90% 121 in → 180 out (208 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions"}
moonshotai/kimi-k2.7-code:low__json_schema PASS 90% 119 in → 146 out (131 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
moonshotai/kimi-k2.7-code:medium PASS 90% 119 in → 70 out (79 reasoning)
Add installation instructions to README
moonshotai/kimi-k2.7-code:medium__json_schema PASS 90% 119 in → 199 out (184 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 90% 134 in → 232 out (211 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 90% 134 in → 152 out (150 reasoning)
Add Installation section
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section" }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 90% 134 in → 12 out (0 reasoning)
Update README with installation instructions
JSON Schema Structured Output
(raw) {"commit_message": "Update README with installation instructions"}
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 90% 134 in → 198 out (170 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section to README" }
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 90% 134 in → 178 out (133 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section to README" }
openai/gpt-5.4-mini:medium__json_schema PASS 90% 159 in → 50 out (32 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions"}
openai/gpt-5.4-nano:medium PASS 90% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.4-nano:medium__json_schema PASS 90% 159 in → 18 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
openai/gpt-5.4-nano:xhigh PASS 90% 121 in → 68 out (57 reasoning)
Add installation instructions to README
openai/gpt-5.4:high__json_schema PASS 90% 159 in → 99 out (79 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.4:none PASS 90% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.4:none__json_schema PASS 90% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-luna:high__json_schema PASS 90% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-luna:xhigh PASS 90% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-sol:low PASS 90% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-sol:low__json_schema PASS 90% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-oss-120b:low__json_schema PASS 90% 169 in → 38 out (24 reasoning)
Update README with installation section
JSON Schema Structured Output
(raw) {"commit_message":"Update README with installation section"}
openai/gpt-oss-20b:high PASS 90% 169 in → 519 out (508 reasoning)
Add installation section to README
openai/gpt-oss-20b:low__json_schema PASS 90% 182 in → 53 out (34 reasoning)
Update README with installation instructions
JSON Schema Structured Output
(raw) {"commit_message":"Update README with installation instructions"}
openai/gpt-oss-20b:xhigh PASS 90% 169 in → 402 out (364 reasoning)
Add installation instructions to README.
openai/gpt-oss-20b:xhigh__json_schema PASS 90% 169 in → 313 out (264 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
poolside/laguna-m.1:none__json_schema PASS 90% 135 in → 10 out (0 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions"}
poolside/laguna-m.1:xhigh__json_schema PASS 90% 135 in → 851 out (834 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
poolside/laguna-xs-2.1:low__json_schema PASS 90% 135 in → 315 out (302 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
poolside/laguna-xs-2.1:medium__json_schema PASS 90% 135 in → 194 out (181 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add Installation section to README"}
poolside/laguna-xs-2.1:none__json_schema PASS 90% 135 in → 17 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
poolside/laguna-xs-2.1:xhigh PASS 90% 135 in → 257 out (251 reasoning)
Add Installation section to README
poolside/laguna-xs.2:low PASS 90% 172 in → 290 out (282 reasoning)
Add Installation section to README
poolside/laguna-xs.2:low__json_schema PASS 90% 172 in → 212 out (199 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
qwen/qwen3.6-27b:medium PASS 90% 135 in → 1,408 out (1,324 reasoning)
Add installation instructions to README
qwen/qwen3.6-35b-a3b:medium PASS 90% 135 in → 1,218 out (1,133 reasoning)
Add installation instructions to README
qwen/qwen3.6-35b-a3b:xhigh PASS 90% 135 in → 1,540 out (1,530 reasoning)
Add installation section to README
qwen/qwen3.6-flash:low PASS 90% 135 in → 335 out (325 reasoning)
Add Installation section to README
qwen/qwen3.7-flash:none PASS 90% 137 in → 5 out (0 reasoning)
Add installation instructions to README
qwen/qwen3.7-max:high PASS 90% 135 in → 816 out (806 reasoning)
Add installation instructions to README
thinkingmachines/inkling-small:high__json_schema PASS 90% 130 in → 109 out (92 reasoning)
Add installation instructions to README.md
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README.md"}
thinkingmachines/inkling-small:none PASS 90% 128 in → 8 out (0 reasoning)
Add installation section to README
thinkingmachines/inkling:high PASS 90% 130 in → 148 out (137 reasoning)
Add installation section to README
thinkingmachines/inkling:low PASS 90% 130 in → 90 out (79 reasoning)
Add installation section to README
x-ai/grok-4.3:max__json_schema PASS 90% 355 in → 479 out (469 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
x-ai/grok-4.3:medium__json_schema PASS 90% 361 in → 406 out (396 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
x-ai/grok-4.3:xhigh PASS 90% 295 in → 363 out (358 reasoning)
Add Installation section to README
x-ai/grok-4.5:high__json_schema PASS 90% 395 in → 301 out (289 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
x-ai/grok-4.5:low PASS 90% 323 in → 115 out (110 reasoning)
Add installation section to README
anthropic/claude-fable-5:max__json_schema PASS 89.7% 391 in → 99 out (73 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
deepseek/deepseek-v4-pro:low PASS 89.7% 120 in → 208 out (227 reasoning)
Add installation section to README
deepseek/deepseek-v4-pro:xhigh PASS 89.7% 120 in → 534 out (354 reasoning)
Add installation instructions
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 89.7% 131 in → 204 out (187 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
google/gemma-4-26b-a4b-it:low PASS 89.7% 146 in → 467 out (414 reasoning)
Add installation section to README
minimax/minimax-m2.7:medium PASS 89.7% 133 in → 121 out (138 reasoning)
Add README installation section
minimax/minimax-m3:high__json_schema PASS 89.7% 235 in → 14 out (13 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
qwen/qwen3.7-plus:xhigh PASS 89.7% 135 in → 556 out (546 reasoning)
Add installation instructions to README
anthropic/claude-haiku-4.5:xhigh PASS 89.3% 171 in → 418 out (359 reasoning)
Add installation instructions
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 89.3% 341 in → 257 out (242 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions"}
anthropic/claude-opus-4.6:xhigh PASS 89.3% 142 in → 8 out (0 reasoning)
Add installation section to README
anthropic/claude-opus-4.7:none PASS 89.3% 175 in → 13 out (0 reasoning)
Add installation instructions to README
anthropic/claude-opus-4.7:none__json_schema PASS 89.3% 396 in → 23 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
anthropic/claude-opus-4.8:xhigh PASS 89.3% 170 in → 11 out (0 reasoning)
Add installation instructions to README
anthropic/claude-sonnet-4.6:high PASS 89.3% 142 in → 8 out (0 reasoning)
Add installation instructions to README
anthropic/claude-sonnet-4.6:low PASS 89.3% 142 in → 8 out (0 reasoning)
Add installation instructions to README
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 89.3% 120 in → 107 out (101 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 89.3% 122 in → 276 out (262 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
deepseek/deepseek-v4-pro:none PASS 89.3% 120 in → 4 out (0 reasoning)
Add installation instructions
deepseek/deepseek-v4-pro:none__json_schema PASS 89.3% 122 in → 15 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
google/gemini-3-flash-preview:xhigh PASS 89.3% 130 in → 185 out (180 reasoning)
Add installation instructions to README
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 89.3% 193 in → 273 out (261 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 89.3% 131 in → 307 out (302 reasoning)
Add installation instructions to README
google/gemini-3.1-pro-preview:low PASS 89.3% 130 in → 328 out (323 reasoning)
Add installation instructions to README
google/gemini-3.1-pro-preview:xhigh PASS 89.3% 130 in → 507 out (500 reasoning)
Add installation instructions to README.md
google/gemini-3.5-flash-lite:low PASS 89.3% 131 in → 5 out (0 reasoning)
Add installation instructions to README
google/gemini-3.6-flash:low__json_schema PASS 89.3% 193 in → 13 out (0 reasoning)
Add Installation section to README.md
JSON Schema Structured Output
(raw) {"commit_message":"Add Installation section to README.md"}
google/gemini-3.6-flash:medium PASS 89.3% 131 in → 301 out (296 reasoning)
Add installation instructions to README
google/gemma-4-26b-a4b-it:none PASS 89.3% 143 in → 8 out (0 reasoning)
Add installation section to README.md
minimax/minimax-m3:high PASS 89.3% 287 in → 120 out (143 reasoning)
Add installation section to README
mistralai/mistral-medium-3-5:medium__json_schema PASS 89.3% 121 in → 167 out (188 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions"}
moonshotai/kimi-k2.6:high__json_schema PASS 89.3% 119 in → 465 out (450 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section to README" }
moonshotai/kimi-k2.6:low PASS 89.3% 119 in → 513 out (506 reasoning)
Add installation instructions to README
moonshotai/kimi-k2.7-code:low PASS 89.3% 119 in → 195 out (188 reasoning)
Add installation instructions to README
nvidia/nemotron-3-nano-30b-a3b:low PASS 89.3% 134 in → 642 out (538 reasoning)
Add Installation instructions
nvidia/nemotron-3-nano-30b-a3b:medium PASS 89.3% 134 in → 76 out (78 reasoning)
Add Installation guide
nvidia/nemotron-3-super-120b-a12b:low PASS 89.3% 134 in → 376 out (265 reasoning)
Add Installation section to README
openai/gpt-5.4-mini:low__json_schema PASS 89.3% 159 in → 33 out (15 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions"}
openai/gpt-5.4-mini:medium PASS 89.3% 121 in → 51 out (42 reasoning)
Add installation instructions
openai/gpt-5.4-nano:xhigh__json_schema PASS 89.3% 159 in → 77 out (57 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.4:low PASS 89.3% 121 in → 23 out (12 reasoning)
Add installation instructions to README
openai/gpt-5.6-luna:medium PASS 89.3% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-luna:none__json_schema PASS 89.3% 159 in → 16 out (0 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions"}
openai/gpt-5.6-sol:high PASS 89.3% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-sol:high__json_schema PASS 89.3% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-sol:medium__json_schema PASS 89.3% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-sol:none PASS 89.3% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-terra:high PASS 89.3% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-terra:low__json_schema PASS 89.3% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-oss-120b:high PASS 89.3% 186 in → 445 out (431 reasoning)
Add Installation section to README
openai/gpt-oss-120b:medium PASS 89.3% 169 in → 68 out (63 reasoning)
Add Installation section to README
openai/gpt-oss-120b:xhigh PASS 89.3% 180 in → 427 out (292 reasoning)
Add Installation section to README
poolside/laguna-m.1:high__json_schema PASS 89.3% 135 in → 307 out (296 reasoning)
Add installation guide
JSON Schema Structured Output
(raw) {"commit_message": "Add installation guide"}
qwen/qwen3.6-27b:low__json_schema PASS 89.3% 135 in → 1,515 out (1 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
qwen/qwen3.6-27b:none__json_schema PASS 89.3% 137 in → 17 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
qwen/qwen3.6-flash:none PASS 89.3% 137 in → 5 out (0 reasoning)
Add installation instructions to README
qwen/qwen3.7-flash:high PASS 89.3% 135 in → 875 out (865 reasoning)
Add installation instructions to README
qwen/qwen3.7-flash:max PASS 89.3% 135 in → 1,300 out (1,290 reasoning)
Add installation instructions to README
qwen/qwen3.7-max:medium PASS 89.3% 135 in → 659 out (649 reasoning)
Add installation instructions to README
qwen/qwen3.7-max:xhigh PASS 89.3% 135 in → 1,299 out (1,289 reasoning)
Add installation instructions to README
tencent/hy3:high__json_schema PASS 89.3% 127 in → 1,312 out (1,116 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
tencent/hy3:low PASS 89.3% 127 in → 777 out (770 reasoning)
Add installation section to README
thinkingmachines/inkling-small:low PASS 89.3% 130 in → 47 out (36 reasoning)
Add installation instructions to README
thinkingmachines/inkling-small:max PASS 89.3% 130 in → 204 out (193 reasoning)
Add installation section to README
x-ai/grok-4.3:none__json_schema PASS 89.3% 353 in → 10 out (0 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add Installation section to README"}
z-ai/glm-4.7-flash:none__json_schema PASS 89.3% 120 in → 15 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
z-ai/glm-5.2:none PASS 89.3% 121 in → 6 out (0 reasoning)
Add installation instructions to README
z-ai/glm-5.2:none__json_schema PASS 89.3% 121 in → 14 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 89% 119 in → 174 out (140 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section to README" }
openai/gpt-5.4-nano:none PASS 89% 121 in → 9 out (0 reasoning)
Add installation instructions to README
anthropic/claude-sonnet-5:none__json_schema PASS 88.7% 391 in → 20 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
google/gemini-3.5-flash-lite:low__json_schema PASS 88.7% 193 in → 12 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
openai/gpt-5.5:high PASS 88.7% 121 in → 84 out (73 reasoning)
Add installation instructions to README
qwen/qwen3.7-max:low PASS 88.7% 135 in → 539 out (529 reasoning)
Add installation instructions to README
anthropic/claude-fable-5:high PASS 88.3% 170 in → 9 out (0 reasoning)
Add installation section to README
anthropic/claude-opus-4.6:none__json_schema PASS 88.3% 312 in → 14 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
arcee-ai/trinity-large-thinking:medium PASS 88.3% 119 in → 181 out (176 reasoning)
Add installation instructions
deepseek/deepseek-v4-flash-0731:high PASS 88.3% 199 in → 99 out (92 reasoning)
Add installation instructions to README
google/gemini-3.5-flash:medium PASS 88.3% 130 in → 359 out (354 reasoning)
Add installation instructions to README
google/gemini-3.6-flash:high__json_schema PASS 88.3% 131 in → 322 out (311 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
google/gemma-4-26b-a4b-it:low__json_schema PASS 88.3% 146 in → 399 out (387 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
minimax/minimax-m2.5:high__json_schema PASS 88.3% 149 in → 150 out (167 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions" }
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 88.3% 134 in → 421 out (342 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add Installation section to README"}
nvidia/nemotron-3-super-120b-a12b:medium PASS 88.3% 134 in → 287 out (253 reasoning)
Add Installation section to README
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 88.3% 134 in → 215 out (179 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section to README" }
nvidia/nemotron-3-super-120b-a12b:none PASS 88.3% 134 in → 7 out (0 reasoning)
Add Installation section to README.md
openai/gpt-5.5:low__json_schema PASS 88.3% 159 in → 16 out (0 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions"}
openai/gpt-5.6-terra:medium PASS 88.3% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-oss-120b:xhigh__json_schema PASS 88.3% 180 in → 514 out (332 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section to README" }
openai/gpt-oss-20b:medium PASS 88.3% 180 in → 104 out (103 reasoning)
Add installation section to README.
openai/gpt-oss-20b:medium__json_schema PASS 88.3% 184 in → 108 out (107 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
poolside/laguna-xs.2:medium__json_schema PASS 88.3% 172 in → 309 out (296 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add Installation section to README"}
qwen/qwen3.6-35b-a3b:high PASS 88.3% 135 in → 1,757 out (1,481 reasoning)
Add installation instructions to README
qwen/qwen3.7-plus:high PASS 88.3% 135 in → 342 out (332 reasoning)
Add installation instructions to README
qwen/qwen3.7-plus:medium PASS 88.3% 135 in → 442 out (432 reasoning)
Add installation instructions to README
tencent/hy3:low__json_schema PASS 88.3% 127 in → 458 out (391 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation section to README" }
tencent/hy3:medium PASS 88.3% 127 in → 460 out (453 reasoning)
Add Installation section to README
tencent/hy3:none__json_schema PASS 88.3% 130 in → 14 out (0 reasoning)
Update README with installation section
JSON Schema Structured Output
(raw) { "commit_message": "Update README with installation section" }
tencent/hy3:xhigh PASS 88.3% 127 in → 719 out (540 reasoning)
Add Installation section to README
x-ai/grok-4.3:low__json_schema PASS 88.3% 361 in → 228 out (218 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation section to README"}
x-ai/grok-4.5:high PASS 88.3% 323 in → 140 out (135 reasoning)
Add installation section to README
x-ai/grok-4.5:xhigh PASS 88.3% 323 in → 153 out (148 reasoning)
Add installation section to README
z-ai/glm-4.7-flash:high__json_schema PASS 88.3% 120 in → 521 out (512 reasoning)
Add installation
JSON Schema Structured Output
(raw) {"commit_message": "Add installation"}
tencent/hy3:none PASS 88% 130 in → 6 out (0 reasoning)
Add Installation section to README
anthropic/claude-haiku-4.5:low__json_schema PASS 87.7% 341 in → 207 out (221 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
deepseek/deepseek-v4-flash-0731:none PASS 87.7% 120 in → 7 out (0 reasoning)
Update README with installation instructions
deepseek/deepseek-v4-pro:medium PASS 87.7% 120 in → 315 out (310 reasoning)
Add installation instructions
google/gemma-4-31b-it:high__json_schema PASS 87.7% 145 in → 168 out (144 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
mistralai/devstral-2512 PASS 87.7% 121 in → 8 out
"Add installation instructions to README"
mistralai/mistral-medium-3-5:xhigh PASS 87.7% 133 in → 516 out (568 reasoning)
Add installation instructions
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 87.7% 134 in → 491 out (418 reasoning)
Add Installation section to README
openai/gpt-5.4-mini:none PASS 87.7% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-terra:low PASS 87.7% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-5.6-terra:max__json_schema PASS 87.7% 159 in → 70 out (50 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
poolside/laguna-xs-2.1:low PASS 87.7% 135 in → 326 out (320 reasoning)
Add Installation section to README
poolside/laguna-xs.2:medium PASS 87.7% 172 in → 336 out (328 reasoning)
Add installation instructions to README
qwen/qwen3.6-27b:xhigh PASS 87.7% 135 in → 783 out (765 reasoning)
Add installation instructions to README
x-ai/grok-4.3:medium PASS 87.7% 301 in → 487 out (482 reasoning)
Add installation section to README
anthropic/claude-fable-5:medium PASS 87.3% 170 in → 11 out (0 reasoning)
`Add installation section to README`
ibm-granite/granite-4.1-8b PASS 87% 133 in → 6 out
Add Installation instructions to README
anthropic/claude-haiku-4.5:none__json_schema PASS 86.7% 311 in → 15 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
anthropic/claude-sonnet-5:low__json_schema PASS 86.7% 391 in → 21 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
nvidia/nemotron-3-nano-30b-a3b:none PASS 86.7% 134 in → 6 out (0 reasoning)
Update README with installation section
openai/gpt-5.6-sol:xhigh__json_schema PASS 86.7% 159 in → 18 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
openai/gpt-5.6-terra:max PASS 86.7% 121 in → 9 out (0 reasoning)
Add installation instructions to README
openai/gpt-oss-120b:low PASS 86.7% 182 in → 34 out (23 reasoning)
Add Installation section to README.md
x-ai/grok-4.3:high__json_schema PASS 86.7% 355 in → 495 out (485 reasoning)
Add Installation section to README
JSON Schema Structured Output
(raw) {"commit_message":"Add Installation section to README"}
x-ai/grok-4.5:medium PASS 86% 323 in → 137 out (132 reasoning)
Add Installation section to README
thinkingmachines/inkling-small:max__json_schema PASS 85.3% 130 in → 213 out (197 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
z-ai/glm-5.2:low__json_schema PASS 85% 306 in → 120 out (108 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
qwen/qwen3.6-flash:high PASS 83.9% 135 in → 1,381 out (1,371 reasoning)
Add installation instructions to README
anthropic/claude-opus-4.7:high__json_schema PASS 83.3% 396 in → 23 out (0 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message":"Add installation instructions to README"}
qwen/qwen3.6-35b-a3b:none__json_schema PASS 83.3% 137 in → 12 out (0 reasoning)
Add installation instructions
JSON Schema Structured Output
(raw) { "commit_message": "Add installation instructions" }
google/gemini-3.1-pro-preview:high__json_schema PASS 81.7% 193 in → 364 out (352 reasoning)
Add installation instructions to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
arcee-ai/trinity-mini:high PASS 81% 119 in → 363 out (411 reasoning)
Add installationinstructions
poolside/laguna-xs.2:high__json_schema PASS 76.7% 172 in → 192 out (177 reasoning)
Add Installation section with pip install command
JSON Schema Structured Output
(raw) {"commit_message": "Add Installation section with pip install command"}
openai/gpt-5.4-nano:high PASS 76.3% 121 in → 9 out (0 reasoning)
Add installation section to README
deepseek/deepseek-v4-flash:xhigh PASS 75.7% 199 in → 421 out (468 reasoning)
AddInstallation section to README
nvidia/nemotron-3-nano-30b-a3b:high PASS 72.3% 134 in → 74 out (79 reasoning)
Add Installation section with pip install instructions.
google/gemma-4-31b-it:none PASS 68.3% 143 in → 8 out (0 reasoning)
Update README.md with installation instructions
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 63.3% 134 in → 96 out (92 reasoning)
Add Installation section
JSON Schema Structured Output
(raw) { "commit_message": "Add Installation section" }
poolside/laguna-m.1:xhigh PASS 63.3% 135 in → 200 out (192 reasoning)
Add installation instructions to README
poolside/laguna-xs.2:none__json_schema PASS 63.3% 172 in → 12 out (0 reasoning)
Add installation section to README
JSON Schema Structured Output
(raw) {"commit_message": "Add installation section to README"}
qwen/qwen3.6-flash:xhigh PASS 63.3% 135 in → 815 out (805 reasoning)
Add installation instructions to README
x-ai/grok-4.5:max PASS 63.3% 323 in → 131 out (126 reasoning)
Add installation section to README
anthropic/claude-fable-5:xhigh PASS 62% 170 in → 81 out (13 reasoning)
``` Add installation section to README ``` This commit message is 36 characters, starts with an action verb ("Add"), and concisely describes the change: a new "Installation" section with setup instructions was added to the README file.
arcee-ai/trinity-large-thinking:high PASS 61.7% 119 in → 730 out (727 reasoning)
Add installation section to README
inclusionai/ling-2.6-flash__json_schema PASS 54.3% 145 in → 16 out
[Fixed] Update README installation instructions
JSON Schema Structured Output
(raw) { "commit_message": "[Fixed] Update README installation instructions" }
moonshotai/kimi-k3:max FAIL 38.3% 196 in → 306 out (258 reasoning)
**Commit message:** ``` Add installation section to README ``` (34 characters — describes the addition of the Installation section with `pip install` instructions to the README.)
anthropic/claude-fable-5:max FAIL 34% 170 in → 357 out (68 reasoning)
``` Add installation instructions to README ``` This message (40 chars) uses the imperative "Add" and clearly describes the new Installation section added to the README.
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:low__json_schema FAIL 0%
Invalid JSON. Output: Add installation instructions to README
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:medium__json_schema FAIL 0%
Invalid JSON. Output: Add installation instructions to README
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: Add installation instructions
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k3:high FAIL 0% 196 in → 63 out (43 reasoning)
Add installation instructions to README
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:high__json_schema FAIL 0% 196 in → 269 out (224 reasoning)
**Commit message:** ``` Add installation instructions to README ``` (39 characters — describes the action of adding a new Installation section to the README file.)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low FAIL 0% 196 in → 223 out (203 reasoning)
Add installation instructions to README
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low__json_schema FAIL 0% 196 in → 102 out (82 reasoning)
Add installation section to README
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max__json_schema FAIL 0% 196 in → 248 out (203 reasoning)
**Add installation instructions to README** (39 characters — starts with the action verb "Add", clearly describes the new installation section added to the README.)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium FAIL 0% 196 in → 233 out (213 reasoning)
Add installation instructions to README
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium__json_schema FAIL 0% 196 in → 125 out (103 reasoning)
**Add installation instructions to README**
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh FAIL 0% 196 in → 282 out (238 reasoning)
**Commit message:** ``` Add installation instructions to README ``` (39 characters — describes the addition of the new Installation section with pip install instructions.)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh__json_schema FAIL 0% 196 in → 152 out (130 reasoning)
**Add installation instructions to README**
Failure: Scoring error: llm_judge requires a judge client
qwen/qwen3.6-27b:medium__json_schema FAIL 0%
Invalid JSON. Output: Add installation instructions to README
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-44f7748a-0310-934a-a784-983661524180","request_id":"44f7748a-0310-934a-a784-983661524180"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-65ef7dca-ad36-9786-a07a-ff54d39fbb6f","request_id":"65ef7dca-ad36-9786-a07a-ff54d39fbb6f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b747f919-488d-9ab7-9f34-efbd0efb0e4d","request_id":"b747f919-488d-9ab7-9f34-efbd0efb0e4d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c5c892bf-2114-97b7-ae73-cb2e84677ebd","request_id":"c5c892bf-2114-97b7-ae73-cb2e84677ebd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4fe3798a-cb66-900e-b13b-c13c62324d22","request_id":"4fe3798a-cb66-900e-b13b-c13c62324d22"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-300181a8-d708-9eaa-b21b-05e8d7fb2cd6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6fee7d2d-1338-9ccf-a0f1-ec902450a886"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3ca4c061-0c5b-9e31-9ddc-81036c72824b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a6aa36a9-07f3-9fe4-9afd-6ffcddd525fb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0% 137 in → 5 out (0 reasoning)
Add installation instructions to README
Failure: Scoring error: llm_judge requires a judge client
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f95bbae1-1517-95e7-a9e8-c9b80422fe72"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ed1f3576-f324-9dfc-8519-ab54e9fd1fa5","request_id":"ed1f3576-f324-9dfc-8519-ab54e9fd1fa5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-70ffa3d1-e28a-9c21-96e4-c00ddd828347","request_id":"70ffa3d1-e28a-9c21-96e4-c00ddd828347"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-68b4d6f9-948c-9d07-b613-0529d0a70298","request_id":"68b4d6f9-948c-9d07-b613-0529d0a70298"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a8f0dc04-a5f4-978f-8148-b24ad1af798a","request_id":"a8f0dc04-a5f4-978f-8148-b24ad1af798a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-77949219-6a7a-9d66-86ac-23069a485722","request_id":"77949219-6a7a-9d66-86ac-23069a485722"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-de49e4ca-6401-9fd2-9b01-ddd9c0abe0f0","request_id":"de49e4ca-6401-9fd2-9b01-ddd9c0abe0f0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3092e239-6410-9188-9f11-47beba7ea931","request_id":"3092e239-6410-9188-9f11-47beba7ea931"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3d2581a5-0e1d-987c-a51b-12eabcf788d0","request_id":"3d2581a5-0e1d-987c-a51b-12eabcf788d0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-faaa04c3-8b7b-979d-8f77-7a97f0cd3748","request_id":"faaa04c3-8b7b-979d-8f77-7a97f0cd3748"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2529900b-1f69-9c8c-bb82-0d62c0cbc40a","request_id":"2529900b-1f69-9c8c-bb82-0d62c0cbc40a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: Add installation instructions to README{"commit_message": "Add installation instructions to README"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: Add installation section to README
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: Add installation instructions to README
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: Add installation instructions to README
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: Add installation instructions to README{"commit_message": "Add installation instructions to README"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Add installation section to README
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)