f002 — commit_messages

← commit_messages
Multiple files added
Tests ability to generate a commit message summarizing multiple new files (config.py, main.py, utils.py). Evaluates grouping related file additions.
easy commit-message multi-file add summarization
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'config' > config.py
  5. 05 echo 'main' > main.py
  6. 06 echo 'utils' > utils.py
  7. 07 git add config.py main.py utils.py
Prompt
Generate a concise commit message (max 50 characters) describing the changes in this diff. Focus on the action taken (e.g., 'Add', 'Fix', 'Update').
Expected
Add initial project files
Fixture Metrics

Scoped model quality, cost, API time, and token usage for commit_messages/f002.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
openai/gpt-5.4:medium PASS 95.3% 195 in → 87 out (73 reasoning)
Add config, main, and utils modules
openai/gpt-5.4-mini:medium__json_schema PASS 95% 233 in → 68 out (45 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
deepseek/deepseek-v4-flash:low__json_schema PASS 94.3% 196 in → 189 out (181 reasoning)
Add config, main, utils
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils"}
anthropic/claude-sonnet-4.6:max PASS 93.7% 238 in → 72 out (51 reasoning)
Add initial project structure files
google/gemini-3.1-pro-preview:high__json_schema PASS 93.7% 308 in → 421 out (406 reasoning)
Add main, config, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add main, config, and utils files"}
anthropic/claude-haiku-4.5:none__json_schema PASS 93.3% 407 in → 18 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
anthropic/claude-opus-4.6:xhigh PASS 93.3% 238 in → 11 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-opus-4.8:high__json_schema PASS 93.3% 497 in → 23 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils modules"}
anthropic/claude-opus-4.8:none PASS 93.3% 276 in → 13 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-opus-4.8:none__json_schema PASS 93.3% 497 in → 21 out (0 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
deepseek/deepseek-v4-flash:high PASS 93.3% 196 in → 244 out (234 reasoning)
Add config, main, utils
deepseek/deepseek-v4-pro:none__json_schema PASS 93.3% 387 in → 15 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
mistralai/devstral-2512__json_schema PASS 93.3% 226 in → 15 out
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
openai/gpt-5.6-luna:xhigh__json_schema PASS 93.3% 233 in → 21 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils modules"}
poolside/laguna-xs-2.1:xhigh__json_schema PASS 93.3% 237 in → 765 out (750 reasoning)
Add config, main, utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils files"}
qwen/qwen3.6-flash:none PASS 93.3% 235 in → 8 out (0 reasoning)
Add config, main, and utils files
tencent/hy3:none__json_schema PASS 93.3% 204 in → 15 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
thinkingmachines/inkling-small:xhigh PASS 93.3% 204 in → 245 out (242 reasoning)
Add config, main, and utils files
thinkingmachines/inkling:low PASS 93.3% 204 in → 61 out (46 reasoning)
Add config, main, and utils files
deepseek/deepseek-v4-flash:medium PASS 93% 196 in → 259 out (252 reasoning)
Add config, main, utils
anthropic/claude-opus-4.6:high PASS 92.7% 238 in → 11 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-sonnet-4.6:high__json_schema PASS 92.7% 408 in → 17 out (0 reasoning)
Add main, config, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add main, config, and utils modules"}
google/gemma-4-31b-it:medium__json_schema PASS 92.7% 264 in → 377 out (365 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
openai/gpt-5.6-luna:max__json_schema PASS 92.7% 233 in → 73 out (50 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils modules"}
openai/gpt-oss-120b:low__json_schema PASS 92.7% 254 in → 191 out (108 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils modules" }
poolside/laguna-m.1:medium PASS 92.7% 237 in → 379 out (368 reasoning)
Add config, main, and utils modules
qwen/qwen3.7-max:none PASS 92.7% 235 in → 8 out (0 reasoning)
Add config, main, and utils modules
google/gemma-4-26b-a4b-it:high__json_schema PASS 92.5% 261 in → 414 out (354 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
thinkingmachines/inkling-small:low PASS 92.5% 204 in → 55 out (51 reasoning)
Add initial project files
anthropic/claude-opus-4.8:low__json_schema PASS 92.3% 497 in → 23 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
nvidia/nemotron-3-super-120b-a12b:high PASS 92.3% 239 in → 338 out (310 reasoning)
Add config, main, utils files
openai/gpt-oss-120b:medium PASS 92.3% 256 in → 226 out (169 reasoning)
Add config, main, and utils modules
z-ai/glm-5.2:xhigh__json_schema PASS 92.3% 204 in → 14 out (13 reasoning)
Add initial project structure
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project structure" }
deepseek/deepseek-v4-pro:high__json_schema PASS 92% 198 in → 266 out (247 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils modules" }
moonshotai/kimi-k2.7-code:high PASS 92% 197 in → 201 out (179 reasoning)
Add config, main, and utils files
poolside/laguna-xs-2.1:xhigh PASS 92% 237 in → 315 out (306 reasoning)
Add config, main, and utils modules
z-ai/glm-4.7-flash:xhigh PASS 92% 197 in → 767 out (752 reasoning)
Add config, main, utils
anthropic/claude-fable-5:max PASS 91.7% 276 in → 385 out (80 reasoning)
``` Add config, main, and utils modules ``` This message (36 characters) captures the action of adding three new placeholder Python files to the project.
anthropic/claude-fable-5:medium PASS 91.7% 276 in → 61 out (0 reasoning)
``` Add initial project files ``` This commit message captures that three new placeholder files (`config.py`, `main.py`, `utils.py`) were created, marking the initial project structure setup.
anthropic/claude-opus-4.6:medium PASS 91.7% 238 in → 11 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-opus-4.8:medium PASS 91.7% 276 in → 13 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-opus-4.8:xhigh__json_schema PASS 91.7% 497 in → 23 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
google/gemini-3.5-flash-lite:high PASS 91.7% 245 in → 363 out (359 reasoning)
Add initial project files
google/gemini-3.5-flash-lite:low__json_schema PASS 91.7% 308 in → 11 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
google/gemini-3.5-flash:medium__json_schema PASS 91.7% 308 in → 332 out (317 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
google/gemini-3.6-flash:high__json_schema PASS 91.7% 308 in → 412 out (398 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
google/gemma-4-26b-a4b-it:high PASS 91.7% 261 in → 498 out (466 reasoning)
Add initial project files
google/gemma-4-26b-a4b-it:medium PASS 91.7% 260 in → 357 out (314 reasoning)
Add initial project files
google/gemma-4-26b-a4b-it:medium__json_schema PASS 91.7% 260 in → 466 out (421 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
google/gemma-4-31b-it:low__json_schema PASS 91.7% 264 in → 376 out (364 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
ibm-granite/granite-4.1-8b__json_schema PASS 91.7% 221 in → 16 out
Add three new module stub files
JSON Schema Structured Output
(raw) { "commit_message": "Add three new module stub files" }
minimax/minimax-m3:xhigh__json_schema PASS 91.7% 313 in → 17 out (16 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils files" }
openai/gpt-5.4-mini:high PASS 91.7% 195 in → 175 out (161 reasoning)
Add config, main, and utils files
openai/gpt-5.4-mini:none PASS 91.7% 195 in → 12 out (0 reasoning)
Add config, main, and utils files
openai/gpt-5.6-luna:low__json_schema PASS 91.7% 233 in → 21 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils modules"}
openai/gpt-5.6-terra:max PASS 91.7% 195 in → 91 out (76 reasoning)
Add initial config, main, and utils modules
openai/gpt-5.6-terra:xhigh__json_schema PASS 91.7% 233 in → 21 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils modules"}
openai/gpt-oss-120b:xhigh__json_schema PASS 91.7% 258 in → 390 out (291 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils modules" }
poolside/laguna-m.1:low PASS 91.7% 237 in → 287 out (276 reasoning)
Add config, main, and utils modules
qwen/qwen3.6-35b-a3b:medium PASS 91.7% 233 in → 2,219 out (2,004 reasoning)
Add initial placeholder files
qwen/qwen3.6-flash:high PASS 91.7% 233 in → 1,634 out (1,621 reasoning)
Add config, main, and utils files
tencent/hy3:none PASS 91.7% 204 in → 9 out (0 reasoning)
Add config, main, and utils modules
thinkingmachines/inkling:none PASS 91.7% 202 in → 11 out (0 reasoning)
Add config, main, and utils modules
x-ai/grok-4.5:low PASS 91.7% 408 in → 168 out (164 reasoning)
Add initial project files
anthropic/claude-fable-5:max__json_schema PASS 91.3% 497 in → 244 out (89 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
openai/gpt-5.4:xhigh PASS 91.3% 195 in → 229 out (215 reasoning)
Add config, main, and utils modules
poolside/laguna-xs-2.1:high PASS 91.3% 237 in → 848 out (840 reasoning)
Add config, main, and utils
qwen/qwen3.6-flash:low PASS 91.3% 233 in → 1,378 out (1,369 reasoning)
Add project skeleton files
thinkingmachines/inkling-small:medium PASS 91.3% 204 in → 171 out (98 reasoning)
Add config, main, and utils files
anthropic/claude-haiku-4.5:high__json_schema PASS 91% 437 in → 426 out (377 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
anthropic/claude-opus-4.6:max PASS 91% 238 in → 52 out (30 reasoning)
Add initial project files
anthropic/claude-opus-4.8:xhigh PASS 91% 276 in → 13 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-sonnet-5:high PASS 91% 276 in → 12 out (0 reasoning)
Add config, main, and utils files
anthropic/claude-sonnet-5:medium PASS 91% 276 in → 12 out (0 reasoning)
Add config, main, and utils files
arcee-ai/trinity-mini:medium__json_schema PASS 91% 195 in → 223 out (221 reasoning)
Add config, main, utils
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, utils" }
deepseek/deepseek-v4-flash:medium__json_schema PASS 91% 196 in → 347 out (320 reasoning)
Add config, main, utils
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils"}
google/gemini-3-flash-preview:high__json_schema PASS 91% 246 in → 443 out (428 reasoning)
Add main, config, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add main, config, and utils files"}
google/gemini-3-flash-preview:medium__json_schema PASS 91% 308 in → 363 out (353 reasoning)
Add core project files
JSON Schema Structured Output
(raw) {"commit_message":"Add core project files"}
google/gemini-3.1-pro-preview:low__json_schema PASS 91% 308 in → 100 out (84 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
google/gemini-3.5-flash-lite:xhigh PASS 91% 245 in → 385 out (381 reasoning)
Add initial project files
google/gemini-3.6-flash:low PASS 91% 246 in → 216 out (203 reasoning)
Add config.py, main.py, and utils.py
google/gemini-3.6-flash:medium__json_schema PASS 91% 246 in → 312 out (298 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
google/gemma-4-31b-it:xhigh PASS 91% 261 in → 246 out (1 reasoning)
Add initial project files
minimax/minimax-m2.5:medium PASS 91% 227 in → 168 out (162 reasoning)
Add initial project files
minimax/minimax-m2.7:high PASS 91% 229 in → 264 out (259 reasoning)
Add initial project files
openai/gpt-5.4-mini:low__json_schema PASS 91% 233 in → 45 out (26 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
openai/gpt-5.4-mini:medium PASS 91% 195 in → 142 out (128 reasoning)
Add config, main, and utils files
openai/gpt-5.4-nano:xhigh__json_schema PASS 91% 233 in → 495 out (473 reasoning)
Add config, main, utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, utils modules"}
openai/gpt-5.5:high PASS 91% 195 in → 93 out (79 reasoning)
Add config, main, and utils modules
openai/gpt-5.6-luna:high__json_schema PASS 91% 233 in → 21 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils modules"}
openai/gpt-5.6-terra:medium PASS 91% 195 in → 12 out (0 reasoning)
Add config, main, and utils modules
openai/gpt-oss-120b:high__json_schema PASS 91% 256 in → 17 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils modules" }
qwen/qwen3.6-27b:medium PASS 91% 233 in → 489 out (410 reasoning)
Add config, main, and utils files
qwen/qwen3.6-27b:xhigh PASS 91% 233 in → 1,343 out (1,334 reasoning)
Add initial project files
qwen/qwen3.6-35b-a3b:high PASS 91% 233 in → 1,456 out (1,202 reasoning)
Add config, main, and utils files
qwen/qwen3.6-35b-a3b:none__json_schema PASS 91% 235 in → 14 out (0 reasoning)
Add initial project skeleton files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project skeleton files" }
qwen/qwen3.7-flash:low PASS 91% 233 in → 1,114 out (1,101 reasoning)
Add config, main, and utils files
thinkingmachines/inkling-small:low__json_schema PASS 91% 204 in → 171 out (152 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
thinkingmachines/inkling-small:none PASS 91% 204 in → 7 out (0 reasoning)
Add initial project files<|end_message|>
z-ai/glm-4.7-flash:high PASS 91% 197 in → 1,166 out (1,162 reasoning)
Add config, main, utils
anthropic/claude-fable-5:low__json_schema PASS 90.7% 497 in → 18 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
anthropic/claude-fable-5:medium__json_schema PASS 90.7% 497 in → 18 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
anthropic/claude-sonnet-4.6:medium PASS 90.7% 238 in → 8 out (0 reasoning)
Add initial project structure files
anthropic/claude-sonnet-5:medium__json_schema PASS 90.3% 497 in → 22 out (0 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
google/gemini-3.5-flash-lite:medium PASS 90.3% 246 in → 305 out (301 reasoning)
Add initial project files
minimax/minimax-m3:medium__json_schema PASS 90.3% 352 in → 15 out (0 reasoning)
Add initial project structure files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project structure files" }
openai/gpt-5.4-mini:xhigh__json_schema PASS 90.3% 233 in → 723 out (700 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
qwen/qwen3.7-max:xhigh PASS 90.3% 233 in → 1,062 out (1,053 reasoning)
Add initial Python files
thinkingmachines/inkling:xhigh PASS 90.3% 204 in → 187 out (187 reasoning)
Add initial project files
z-ai/glm-5.2:none PASS 90.3% 197 in → 9 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-fable-5:xhigh__json_schema PASS 90% 497 in → 18 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
anthropic/claude-opus-4.6:max__json_schema PASS 90% 408 in → 61 out (34 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
anthropic/claude-opus-4.6:none PASS 90% 238 in → 11 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-opus-4.6:xhigh__json_schema PASS 90% 408 in → 13 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
anthropic/claude-sonnet-4.6:high PASS 90% 238 in → 8 out (0 reasoning)
Add initial project structure files
anthropic/claude-sonnet-5:xhigh__json_schema PASS 90% 497 in → 22 out (0 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
arcee-ai/trinity-large-thinking:medium PASS 90% 195 in → 564 out (555 reasoning)
Add config, main, utils files
deepseek/deepseek-v4-pro:low__json_schema PASS 90% 387 in → 118 out (102 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
google/gemini-3-flash-preview:low PASS 90% 245 in → 328 out (315 reasoning)
Add config.py, main.py, and utils.py
google/gemini-3.1-flash-lite-preview:medium PASS 90% 245 in → 241 out (237 reasoning)
Add initial project files
google/gemma-4-31b-it:high__json_schema PASS 90% 261 in → 247 out (217 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
google/gemma-4-31b-it:medium PASS 90% 261 in → 349 out (319 reasoning)
Add initial project files
minimax/minimax-m3:xhigh PASS 90% 365 in → 47 out (50 reasoning)
Add initial project files
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 90% 239 in → 18 out (0 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils files" }
openai/gpt-5.4:high PASS 90% 195 in → 148 out (134 reasoning)
Add config, main, and utils modules
openai/gpt-5.4:low PASS 90% 195 in → 30 out (16 reasoning)
Add config, main, and utils files
openai/gpt-5.5:low PASS 90% 195 in → 8 out (0 reasoning)
Add initial project files
openai/gpt-5.5:none PASS 90% 195 in → 8 out (0 reasoning)
Add initial Python files
openai/gpt-5.6-luna:medium__json_schema PASS 90% 233 in → 21 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils modules"}
openai/gpt-5.6-luna:none__json_schema PASS 90% 233 in → 21 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils modules"}
openai/gpt-oss-20b:low__json_schema PASS 90% 256 in → 61 out (29 reasoning)
Add config.py, main.py, utils.py with placeholders
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, utils.py with placeholders" }
qwen/qwen3.6-35b-a3b:low__json_schema PASS 90% 233 in → 441 out (354 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils files" }
qwen/qwen3.6-flash:medium PASS 90% 233 in → 1,933 out (1,924 reasoning)
Add initial project files
qwen/qwen3.7-flash:max PASS 90% 233 in → 1,109 out (1,096 reasoning)
Add main, config, and utils files
qwen/qwen3.7-flash:none PASS 90% 235 in → 8 out (0 reasoning)
Add config, main, and utils files
tencent/hy3:xhigh__json_schema PASS 90% 201 in → 984 out (814 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils files" }
z-ai/glm-5.2:low__json_schema PASS 90% 204 in → 323 out (289 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
anthropic/claude-haiku-4.5:medium PASS 89.7% 267 in → 403 out (376 reasoning)
Add config, main, and utils modules
google/gemma-4-26b-a4b-it:none PASS 89.7% 258 in → 9 out (0 reasoning)
Add config, main, and utils files
openai/gpt-5.4:none PASS 89.7% 195 in → 12 out (0 reasoning)
Add config, main, and utils files
openai/gpt-5.6-sol:medium__json_schema PASS 89.7% 233 in → 21 out (0 reasoning)
Add config, main, and utility modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utility modules"}
poolside/laguna-m.1:high__json_schema PASS 89.7% 237 in → 420 out (400 reasoning)
Add config, main, utils modules
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, utils modules" }
anthropic/claude-fable-5:high__json_schema PASS 89.3% 497 in → 18 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
anthropic/claude-opus-4.7:medium__json_schema PASS 89.3% 502 in → 20 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
anthropic/claude-opus-4.8:max__json_schema PASS 89.3% 497 in → 23 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
anthropic/claude-sonnet-4.6:medium__json_schema PASS 89.3% 408 in → 13 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 89.3% 196 in → 140 out (129 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 89.3% 308 in → 164 out (153 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
google/gemini-3.5-flash-lite:high__json_schema PASS 89.3% 308 in → 504 out (494 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
google/gemini-3.6-flash:low__json_schema PASS 89.3% 308 in → 14 out (0 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
google/gemma-4-26b-a4b-it:xhigh PASS 89.3% 261 in → 669 out (632 reasoning)
Add initial project files
moonshotai/kimi-k2.6:high__json_schema PASS 89.3% 197 in → 260 out (245 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
openai/gpt-5.4-mini:high__json_schema PASS 89.3% 233 in → 153 out (130 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
openai/gpt-5.4-nano:none__json_schema PASS 89.3% 233 in → 18 out (0 reasoning)
Add initial Python scaffold files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial Python scaffold files"}
openai/gpt-5.5:medium__json_schema PASS 89.3% 233 in → 17 out (0 reasoning)
Add initial Python files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial Python files"}
openai/gpt-5.5:xhigh PASS 89.3% 195 in → 74 out (64 reasoning)
Add initial Python files
poolside/laguna-m.1:medium__json_schema PASS 89.3% 237 in → 293 out (274 reasoning)
Add config, main, utils modules
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, utils modules" }
poolside/laguna-xs.2:medium__json_schema PASS 89.3% 274 in → 299 out (283 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
qwen/qwen3.6-35b-a3b:none PASS 89.3% 235 in → 9 out (0 reasoning)
Add config, main, and utils files
qwen/qwen3.7-plus:xhigh PASS 89.3% 233 in → 1,067 out (1,054 reasoning)
Add config, main, and utils files
thinkingmachines/inkling-small:medium__json_schema PASS 89.3% 204 in → 180 out (161 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
x-ai/grok-4.5:medium PASS 89.3% 408 in → 239 out (231 reasoning)
Add initial config, main, utils files
z-ai/glm-4.7-flash:low PASS 89.3% 197 in → 552 out (521 reasoning)
Add config, main, utils
z-ai/glm-5.2:medium PASS 89.3% 204 in → 339 out (333 reasoning)
Add initial project files
anthropic/claude-sonnet-5:low PASS 89% 276 in → 14 out (0 reasoning)
Add initial config, main, and utils files
anthropic/claude-sonnet-5:xhigh PASS 89% 276 in → 12 out (0 reasoning)
Add config, main, and utils files
arcee-ai/trinity-large-thinking:high PASS 89% 195 in → 688 out (680 reasoning)
Add config.py, main.py, utils.py
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 89% 196 in → 359 out (282 reasoning)
Add config, main, utils
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils"}
deepseek/deepseek-v4-pro:low PASS 89% 196 in → 205 out (196 reasoning)
Add config, main, and utils modules
google/gemini-3.1-flash-lite-preview:high PASS 89% 245 in → 356 out (352 reasoning)
Add initial project files
anthropic/claude-opus-4.8:medium__json_schema PASS 88.7% 497 in → 22 out (0 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
arcee-ai/trinity-mini:low PASS 88.7% 195 in → 361 out (373 reasoning)
Add config, main, utils
google/gemini-3.5-flash:low__json_schema PASS 88.7% 308 in → 139 out (119 reasoning)
Add main, config, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add main, config, and utils files" }
moonshotai/kimi-k2.6:low__json_schema PASS 88.7% 196 in → 370 out (288 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
openai/gpt-5.4-mini:xhigh PASS 88.7% 195 in → 460 out (447 reasoning)
Add config, main, and utils
anthropic/claude-haiku-4.5:low__json_schema PASS 88.3% 437 in → 747 out (406 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
anthropic/claude-opus-4.6:high__json_schema PASS 88.3% 408 in → 13 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
anthropic/claude-opus-4.6:low PASS 88.3% 238 in → 11 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-opus-4.6:none__json_schema PASS 88.3% 408 in → 13 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
anthropic/claude-opus-4.7:low__json_schema PASS 88.3% 502 in → 20 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
anthropic/claude-sonnet-4.6:low PASS 88.3% 238 in → 8 out (0 reasoning)
Add initial project structure files
arcee-ai/trinity-mini:high__json_schema PASS 88.3% 195 in → 366 out (383 reasoning)
Add config, main, utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, utils files" }
deepseek/deepseek-v4-flash-0731:high PASS 88.3% 281 in → 107 out (102 reasoning)
Add initial project files
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 88.3% 478 in → 162 out (150 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
deepseek/deepseek-v4-flash-0731:none PASS 88.3% 196 in → 5 out (0 reasoning)
Add initial project files
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 88.3% 196 in → 12 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
deepseek/deepseek-v4-pro:high PASS 88.3% 196 in → 371 out (365 reasoning)
Add initial project structure
deepseek/deepseek-v4-pro:medium__json_schema PASS 88.3% 196 in → 484 out (378 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils modules"}
deepseek/deepseek-v4-pro:none PASS 88.3% 196 in → 5 out (0 reasoning)
Add initial project structure files
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 88.3% 465 in → 211 out (195 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
google/gemini-3.1-flash-lite-preview:low PASS 88.3% 246 in → 138 out (134 reasoning)
Add initial project files
google/gemini-3.5-flash-lite:low PASS 88.3% 246 in → 4 out (0 reasoning)
Add initial project files
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 88.3% 246 in → 567 out (557 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
google/gemma-4-26b-a4b-it:low__json_schema PASS 88.3% 261 in → 534 out (463 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
google/gemma-4-26b-a4b-it:none__json_schema PASS 88.3% 258 in → 21 out (0 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils files" }
google/gemma-4-31b-it:high PASS 88.3% 261 in → 314 out (1 reasoning)
Add initial project files
google/gemma-4-31b-it:none PASS 88.3% 258 in → 5 out (0 reasoning)
Add initial project files
google/gemma-4-31b-it:none__json_schema PASS 88.3% 258 in → 14 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
moonshotai/kimi-k2.6:high PASS 88.3% 197 in → 413 out (407 reasoning)
Add initial project files
moonshotai/kimi-k2.6:xhigh__json_schema PASS 88.3% 197 in → 16 out (0 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial config, main, and utils files"}
moonshotai/kimi-k2.7-code:low__json_schema PASS 88.3% 197 in → 120 out (103 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial config, main, and utils files"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 88.3% 239 in → 313 out (228 reasoning)
Add config, main, utils
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 88.3% 239 in → 293 out (214 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, utils.py" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 88.3% 239 in → 438 out (325 reasoning)
Add config, main, utils
openai/gpt-5.4-mini:low PASS 88.3% 195 in → 49 out (35 reasoning)
Add config, main, and utils files
openai/gpt-5.5:none__json_schema PASS 88.3% 233 in → 17 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
openai/gpt-5.5:xhigh__json_schema PASS 88.3% 233 in → 84 out (65 reasoning)
Add initial Python modules
JSON Schema Structured Output
(raw) {"commit_message":"Add initial Python modules"}
openai/gpt-5.6-sol:none__json_schema PASS 88.3% 233 in → 21 out (0 reasoning)
Add config, main, and utility modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utility modules"}
openai/gpt-5.6-sol:xhigh PASS 88.3% 195 in → 12 out (0 reasoning)
Add config, main, and utility modules
openai/gpt-5.6-terra:low__json_schema PASS 88.3% 233 in → 21 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils modules"}
openai/gpt-oss-120b:low PASS 88.3% 256 in → 64 out (43 reasoning)
Add config, main, and utils modules
poolside/laguna-m.1:xhigh PASS 88.3% 237 in → 491 out (479 reasoning)
Add config, main, and utils modules.
qwen/qwen3.6-27b:none PASS 88.3% 235 in → 9 out (0 reasoning)
Add config, main, and utils files
qwen/qwen3.6-27b:none__json_schema PASS 88.3% 235 in → 20 out (0 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils files" }
qwen/qwen3.6-flash:xhigh PASS 88.3% 233 in → 1,469 out (1,456 reasoning)
Add config, main, and utils files
qwen/qwen3.7-max:medium PASS 88.3% 233 in → 1,252 out (1,243 reasoning)
Add initial Python modules
qwen/qwen3.7-plus:high PASS 88.3% 233 in → 855 out (846 reasoning)
Add initial project files
qwen/qwen3.7-plus:medium PASS 88.3% 233 in → 703 out (694 reasoning)
Add initial project structure
tencent/hy3:high__json_schema PASS 88.3% 201 in → 801 out (727 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
thinkingmachines/inkling-small:max__json_schema PASS 88.3% 204 in → 546 out (527 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
thinkingmachines/inkling-small:xhigh__json_schema PASS 88.3% 204 in → 244 out (223 reasoning)
Add config.py, main.py, and utils.py
JSON Schema Structured Output
(raw) {"commit_message": "Add config.py, main.py, and utils.py"}
thinkingmachines/inkling:medium PASS 88.3% 204 in → 230 out (219 reasoning)
Add initial project files
x-ai/grok-4.5:high PASS 88.3% 408 in → 244 out (234 reasoning)
Add config.py, main.py, and utils.py
x-ai/grok-4.5:high__json_schema PASS 88.3% 480 in → 559 out (548 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
x-ai/grok-4.5:xhigh PASS 88.3% 408 in → 411 out (407 reasoning)
Add initial Python files
z-ai/glm-5.2:low PASS 88.3% 204 in → 351 out (337 reasoning)
Add initial project files
anthropic/claude-opus-4.8:low PASS 88% 276 in → 13 out (0 reasoning)
Add config, main, and utils modules
minimax/minimax-m2.7:medium__json_schema PASS 88% 363 in → 320 out (304 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, utils.py"}
mistralai/mistral-medium-3-5:low PASS 88% 238 in → 924 out (1,004 reasoning)
Add config, main, utils
openai/gpt-5.4-nano:low PASS 88% 195 in → 13 out (0 reasoning)
Add initial config, main, and utils files
openai/gpt-5.4:none__json_schema PASS 88% 233 in → 21 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils modules"}
qwen/qwen3.6-35b-a3b:low PASS 88% 233 in → 1,192 out (1,180 reasoning)
Add config, main, and utils
z-ai/glm-4.7-flash:low__json_schema PASS 88% 197 in → 861 out (775 reasoning)
Add config, main, utils
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils"}
anthropic/claude-haiku-4.5:low PASS 87.7% 267 in → 351 out (235 reasoning)
``` Add config, main, and utils modules ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 87.7% 437 in → 341 out (270 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
anthropic/claude-opus-4.6:low__json_schema PASS 87.7% 408 in → 13 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
anthropic/claude-opus-4.6:medium__json_schema PASS 87.7% 408 in → 13 out (0 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
anthropic/claude-opus-4.8:max PASS 87.7% 276 in → 177 out (76 reasoning)
`Add config, main, and utils modules`
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 87.7% 408 in → 14 out (0 reasoning)
Add initial project structure files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project structure files"}
arcee-ai/trinity-large-thinking:low PASS 87.7% 195 in → 1,008 out (1,002 reasoning)
Add config, main, utils
deepseek/deepseek-v4-flash-0731:xhigh PASS 87.7% 196 in → 125 out (112 reasoning)
Add config, main, and utils
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 87.7% 281 in → 58 out (49 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
google/gemini-3-flash-preview:xhigh__json_schema PASS 87.7% 246 in → 461 out (447 reasoning)
Add main, config, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add main, config, and utils files"}
google/gemini-3.1-pro-preview:medium__json_schema PASS 87.7% 308 in → 429 out (409 reasoning)
Add main.py, config.py, and utils.py
JSON Schema Structured Output
(raw) {"commit_message": "Add main.py, config.py, and utils.py"}
google/gemini-3.5-flash-lite:max PASS 87.7% 246 in → 356 out (352 reasoning)
Add initial project files
google/gemini-3.5-flash:low PASS 87.7% 245 in → 169 out (165 reasoning)
Add initial project files
google/gemini-3.6-flash:max__json_schema PASS 87.7% 246 in → 471 out (457 reasoning)
Add main, config, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add main, config, and utils files"}
google/gemma-4-26b-a4b-it:low PASS 87.7% 261 in → 577 out (572 reasoning)
Add initial project files
minimax/minimax-m2.7:low__json_schema PASS 87.7% 363 in → 188 out (176 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
mistralai/mistral-medium-3-5:high PASS 87.7% 238 in → 6,655 out (6,464 reasoning)
Add config, main, utils
moonshotai/kimi-k2.6:medium PASS 87.7% 197 in → 747 out (836 reasoning)
Add initial project files
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 87.7% 239 in → 319 out (267 reasoning)
Add config, main, utils
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils"}
nvidia/nemotron-3-super-120b-a12b:none PASS 87.7% 239 in → 9 out (0 reasoning)
Add config, main, and utils files
openai/gpt-5.4-nano:medium__json_schema PASS 87.7% 233 in → 22 out (0 reasoning)
Add new config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add new config, main, and utils files"}
openai/gpt-5.6-sol:high__json_schema PASS 87.7% 233 in → 21 out (0 reasoning)
Add config, main, and utility modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utility modules"}
openai/gpt-5.6-terra:low PASS 87.7% 195 in → 12 out (0 reasoning)
Add config, main, and utility files
poolside/laguna-m.1:xhigh__json_schema PASS 87.7% 237 in → 1,245 out (1,226 reasoning)
Add config, main, utils modules
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, utils modules" }
poolside/laguna-xs-2.1:low PASS 87.7% 237 in → 241 out (233 reasoning)
Add config, main, utils files
poolside/laguna-xs-2.1:medium PASS 87.7% 237 in → 1,351 out (1,343 reasoning)
Add config, main, utils modules
poolside/laguna-xs-2.1:none__json_schema PASS 87.7% 237 in → 15 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
poolside/laguna-xs.2:medium PASS 87.7% 274 in → 282 out (271 reasoning)
Add config, main, and utils files
poolside/laguna-xs.2:none__json_schema PASS 87.7% 274 in → 20 out (0 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils modules" }
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 87.7% 233 in → 1,152 out (1,036 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils files" }
qwen/qwen3.7-max:high PASS 87.7% 233 in → 949 out (940 reasoning)
Add initial Python modules
qwen/qwen3.7-plus:low PASS 87.7% 233 in → 917 out (908 reasoning)
Add initial project files
qwen/qwen3.7-plus:none PASS 87.7% 235 in → 9 out (0 reasoning)
Add initial config, main, and utils files
thinkingmachines/inkling-small:high__json_schema PASS 87.7% 204 in → 783 out (763 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message": "Add config.py, main.py, utils.py"}
thinkingmachines/inkling:max PASS 87.7% 204 in → 509 out (373 reasoning)
Add config, main, and utils files
z-ai/glm-5.2:high__json_schema PASS 87.7% 383 in → 130 out (118 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
z-ai/glm-5.2:xhigh PASS 87.7% 204 in → 249 out (258 reasoning)
Add initial project files
ibm-granite/granite-4.1-8b PASS 87.3% 221 in → 12 out
Add new core files (config, main, utils)
openai/gpt-oss-20b:high__json_schema PASS 87.3% 243 in → 170 out (142 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, utils.py"}
openai/gpt-oss-20b:medium__json_schema PASS 87.3% 254 in → 380 out (264 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, utils.py"}
qwen/qwen3.6-27b:high__json_schema PASS 87.3% 233 in → 1,532 out (1,239 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, utils.py" }
x-ai/grok-4.3:xhigh PASS 87.3% 371 in → 716 out (707 reasoning)
Add config.py, main.py, utils.py
anthropic/claude-sonnet-5:low__json_schema PASS 87% 497 in → 24 out (0 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial config, main, and utils files"}
inclusionai/ling-2.6-flash PASS 87% 243 in → 11 out
Add initial project files: config, main, utils
openai/gpt-5.4-nano:none PASS 87% 195 in → 13 out (0 reasoning)
Add initial config, main, and utils files
poolside/laguna-m.1:none__json_schema PASS 87% 237 in → 24 out (0 reasoning)
Add new files: config.py, main.py, utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add new files: config.py, main.py, utils.py" }
poolside/laguna-xs.2:low__json_schema PASS 87% 274 in → 530 out (515 reasoning)
Add config, main, and utils
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils"}
anthropic/claude-fable-5:low PASS 86.7% 276 in → 15 out (0 reasoning)
Add initial project files: config, main, utils
anthropic/claude-fable-5:xhigh PASS 86.7% 276 in → 86 out (40 reasoning)
``` Add initial config, main, and utils modules ``` This commit message (45 characters) captures that three new Python files were created, using the conventional "Add" prefix for new file additions.
anthropic/claude-opus-4.7:low PASS 86.7% 281 in → 10 out (0 reasoning)
Add initial project files
anthropic/claude-opus-4.8:high PASS 86.7% 276 in → 13 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-sonnet-4.6:xhigh PASS 86.7% 238 in → 8 out (0 reasoning)
Add initial project structure files
anthropic/claude-sonnet-5:high__json_schema PASS 86.7% 497 in → 24 out (0 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial config, main, and utils files"}
deepseek/deepseek-v4-flash:none PASS 86.7% 196 in → 5 out (0 reasoning)
Add initial skeleton files
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 86.7% 275 in → 484 out (396 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, utils.py"}
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 86.7% 308 in → 340 out (329 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
google/gemini-3.5-flash:high__json_schema PASS 86.7% 308 in → 490 out (475 reasoning)
Add main, config, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add main, config, and utils files"}
google/gemma-4-31b-it:xhigh__json_schema PASS 86.7% 261 in → 273 out (221 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
minimax/minimax-m3:low PASS 86.7% 365 in → 50 out (51 reasoning)
Add initial project structure files
mistralai/mistral-medium-3-5:low__json_schema PASS 86.7% 226 in → 1,729 out (1,781 reasoning)
Add config, main, utils
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils"}
mistralai/mistral-medium-3-5:xhigh PASS 86.7% 238 in → 3,688 out (3,791 reasoning)
Add config.py, main.py, utils.py
moonshotai/kimi-k2.7-code:xhigh PASS 86.7% 197 in → 219 out (213 reasoning)
Add initial project files
nvidia/nemotron-3-nano-30b-a3b:low PASS 86.7% 239 in → 403 out (279 reasoning)
Add config.py, main.py, utils.py
openai/gpt-5.4:low__json_schema PASS 86.7% 233 in → 39 out (16 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
openai/gpt-5.6-luna:high PASS 86.7% 195 in → 13 out (0 reasoning)
Add initial config, main, and utils files
openai/gpt-5.6-sol:low__json_schema PASS 86.7% 233 in → 21 out (0 reasoning)
Add config, main, and utility modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utility modules"}
openai/gpt-5.6-sol:medium PASS 86.7% 195 in → 12 out (0 reasoning)
Add config, main, and utility modules
openai/gpt-5.6-terra:none__json_schema PASS 86.7% 233 in → 21 out (0 reasoning)
Add config, main, and utility modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utility modules"}
openai/gpt-5.6-terra:xhigh PASS 86.7% 195 in → 12 out (0 reasoning)
Add config, main, and utility modules
openai/gpt-oss-20b:medium PASS 86.7% 254 in → 271 out (181 reasoning)
Add config, main, utils modules.
poolside/laguna-m.1:low__json_schema PASS 86.7% 237 in → 607 out (587 reasoning)
Add config, main, utils modules
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, utils modules" }
poolside/laguna-xs.2:high__json_schema PASS 86.7% 274 in → 288 out (273 reasoning)
Add config, main, utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils files"}
x-ai/grok-4.3:medium__json_schema PASS 86.7% 437 in → 345 out (327 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, utils.py" }
x-ai/grok-4.5:max__json_schema PASS 86.7% 480 in → 493 out (477 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, utils.py"}
x-ai/grok-4.5:xhigh__json_schema PASS 86.7% 480 in → 310 out (298 reasoning)
Add initial project structure files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project structure files"}
z-ai/glm-5.2:high PASS 86.7% 204 in → 94 out (76 reasoning)
Add initial project files
deepseek/deepseek-v4-flash:none__json_schema PASS 86.4% 196 in → 17 out (0 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial config, main, and utils files"}
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 86.3% 239 in → 397 out (317 reasoning)
Add config, main, utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils files"}
openai/gpt-5.6-terra:max__json_schema PASS 86.3% 233 in → 78 out (54 reasoning)
Add initial config, main, and utility files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial config, main, and utility files"}
openai/gpt-oss-20b:xhigh__json_schema PASS 86.3% 256 in → 15 out (0 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 86% 437 in → 335 out (247 reasoning)
Add config, main, and utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils modules"}
anthropic/claude-opus-4.7:high PASS 86% 281 in → 15 out (0 reasoning)
Add config, main, and utils modules
anthropic/claude-opus-4.7:max PASS 86% 281 in → 161 out (70 reasoning)
Add initial config, main, and utils files
anthropic/claude-sonnet-4.6:none PASS 86% 238 in → 8 out (0 reasoning)
Add initial project structure files
anthropic/claude-sonnet-5:none PASS 86% 276 in → 14 out (0 reasoning)
Add initial config, main, and utils files
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 86% 196 in → 174 out (163 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
google/gemini-3-flash-preview:high PASS 86% 245 in → 500 out (492 reasoning)
Add main, config, and utils files
google/gemini-3.6-flash:xhigh__json_schema PASS 86% 308 in → 403 out (389 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
minimax/minimax-m2.5:xhigh__json_schema PASS 86% 227 in → 178 out (167 reasoning)
Add config.py, main.py, and utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, and utils.py" }
minimax/minimax-m2.7:medium PASS 86% 226 in → 167 out (164 reasoning)
Add config.py, main.py, utils.py
mistralai/mistral-medium-3-5:high__json_schema PASS 86% 226 in → 944 out (992 reasoning)
Add project files
JSON Schema Structured Output
(raw) {"commit_message": "Add project files"}
mistralai/mistral-medium-3-5:none PASS 86% 238 in → 4 out (0 reasoning)
Add new files
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 86% 239 in → 139 out (126 reasoning)
Add config, main, utils
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, utils" }
openai/gpt-5.5:high__json_schema PASS 86% 233 in → 18 out (0 reasoning)
Add initial Python module files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial Python module files"}
openai/gpt-5.6-luna:xhigh PASS 86% 195 in → 13 out (0 reasoning)
Add initial config, main, and utility files
qwen/qwen3.6-27b:low__json_schema PASS 86% 233 in → 1,730 out (1,421 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils files" }
qwen/qwen3.7-flash:medium PASS 86% 233 in → 1,736 out (1,724 reasoning)
Add config, main, and utils
tencent/hy3:high PASS 86% 201 in → 840 out (829 reasoning)
Add config.py, main.py and utils.py
tencent/hy3:medium__json_schema PASS 86% 201 in → 1,072 out (809 reasoning)
Add config.py, main.py and utils.py
JSON Schema Structured Output
(raw) {"commit_message": "Add config.py, main.py and utils.py"}
x-ai/grok-4.3:medium PASS 86% 377 in → 604 out (595 reasoning)
Add config.py, main.py, utils.py
z-ai/glm-5.2:medium__json_schema PASS 86% 383 in → 138 out (126 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
google/gemini-3.6-flash:medium PASS 85.7% 246 in → 379 out (366 reasoning)
Add config.py, main.py, and utils.py
moonshotai/kimi-k2.6:medium__json_schema PASS 85.7% 196 in → 603 out (473 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
openai/gpt-5.4:xhigh__json_schema PASS 85.7% 233 in → 447 out (422 reasoning)
Add config.py, main.py, and utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, and utils.py"}
openai/gpt-5.6-sol:none PASS 85.7% 195 in → 12 out (0 reasoning)
Add config, main, and utility modules
deepseek/deepseek-v4-flash-0731:low PASS 85.3% 196 in → 210 out (200 reasoning)
Add config.py, main.py, utils.py
google/gemini-3.6-flash:max PASS 85.3% 245 in → 464 out (451 reasoning)
Add config.py, main.py, and utils.py
minimax/minimax-m2.5:xhigh PASS 85.3% 227 in → 169 out (157 reasoning)
Add config.py, main.py, and utils.py
minimax/minimax-m2.7:low PASS 85.3% 227 in → 154 out (142 reasoning)
Add config.py, main.py, and utils.py
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 85.3% 226 in → 491 out (508 reasoning)
Add config, main, utils
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils"}
openai/gpt-5.6-luna:low PASS 85.3% 195 in → 13 out (0 reasoning)
Add initial config, main, and utility files
poolside/laguna-xs-2.1:high__json_schema PASS 85.3% 237 in → 371 out (356 reasoning)
Add config, main, utils modules
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils modules"}
anthropic/claude-opus-4.7:high__json_schema PASS 85% 502 in → 27 out (0 reasoning)
Add initial project files: config, main, utils
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files: config, main, utils"}
anthropic/claude-opus-4.7:none PASS 85% 281 in → 16 out (0 reasoning)
Add initial config, main, and utils files
anthropic/claude-opus-4.7:none__json_schema PASS 85% 502 in → 27 out (0 reasoning)
Add initial project files: config, main, utils
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files: config, main, utils"}
deepseek/deepseek-v4-flash:high__json_schema PASS 85% 196 in → 261 out (252 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message": "Add config.py, main.py, utils.py"}
google/gemini-3.5-flash-lite:max__json_schema PASS 85% 246 in → 404 out (393 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial project files"}
google/gemini-3.6-flash:high PASS 85% 245 in → 388 out (375 reasoning)
Add config.py, main.py, and utils.py
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 85% 261 in → 588 out (503 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils files" }
google/gemma-4-31b-it:low PASS 85% 260 in → 215 out (194 reasoning)
Add initial project structure
inclusionai/ling-2.6-flash__json_schema PASS 85% 243 in → 23 out
Add initial project files (config, main, utils)
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files (config, main, utils)" }
minimax/minimax-m2.5:high PASS 85% 227 in → 208 out (206 reasoning)
Add config.py, main.py, and utils.py
minimax/minimax-m2.5:high__json_schema PASS 85% 252 in → 312 out (290 reasoning)
Add config.py, main.py, and utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, and utils.py" }
minimax/minimax-m2.5:low__json_schema PASS 85% 252 in → 208 out (208 reasoning)
Add config.py, main.py, and utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, and utils.py" }
minimax/minimax-m2.5:medium__json_schema PASS 85% 227 in → 215 out (196 reasoning)
Add config.py, main.py, and utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, and utils.py" }
minimax/minimax-m2.7:high__json_schema PASS 85% 363 in → 243 out (232 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files"}
minimax/minimax-m3:high__json_schema PASS 85% 313 in → 18 out (17 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial config, main, and utils files" }
mistralai/mistral-medium-3-5:medium__json_schema PASS 85% 226 in → 4,781 out (4,938 reasoning)
Add config main utils
JSON Schema Structured Output
(raw) {"commit_message": "Add config main utils"}
moonshotai/kimi-k2.6:low PASS 85% 197 in → 392 out (386 reasoning)
Add initial Python files
moonshotai/kimi-k2.7-code:medium PASS 85% 197 in → 262 out (256 reasoning)
Add initial Python files
moonshotai/kimi-k2.7-code:medium__json_schema PASS 85% 284 in → 209 out (197 reasoning)
Add initial Python module files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial Python module files"}
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 85% 239 in → 382 out (278 reasoning)
Add config.py, main.py, utils.py
openai/gpt-5.4:medium__json_schema PASS 85% 233 in → 123 out (99 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial config, main, and utils files"}
openai/gpt-5.6-luna:max PASS 85% 195 in → 141 out (126 reasoning)
Add initial config, main, and utility files
openai/gpt-5.6-luna:none PASS 85% 195 in → 13 out (0 reasoning)
Add initial config, main, and utility files
openai/gpt-oss-120b:high PASS 85% 256 in → 585 out (570 reasoning)
Add config.py, main.py, utils.py
poolside/laguna-xs-2.1:low__json_schema PASS 85% 237 in → 440 out (423 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message": "Add config.py, main.py, utils.py"}
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 85% 233 in → 1,178 out (1,065 reasoning)
Add initial project files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial project files" }
thinkingmachines/inkling:high PASS 85% 204 in → 291 out (269 reasoning)
Add config.py, main.py, and utils.py
x-ai/grok-4.3:none PASS 85% 369 in → 9 out (0 reasoning)
Add initial config, main, and utils files
x-ai/grok-4.3:none__json_schema PASS 85% 429 in → 16 out (0 reasoning)
Add new files config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add new files config.py, main.py, utils.py"}
x-ai/grok-4.3:xhigh__json_schema PASS 85% 431 in → 452 out (438 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, utils.py"}
x-ai/grok-4.5:medium__json_schema PASS 85% 480 in → 340 out (325 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
z-ai/glm-4.7-flash:none__json_schema PASS 85% 197 in → 14 out (0 reasoning)
Add config, main, utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, utils files"}
minimax/minimax-m3:high PASS 84.7% 365 in → 68 out (70 reasoning)
Add initial config, main, and utils files
openai/gpt-5.5:low__json_schema PASS 84.7% 233 in → 18 out (0 reasoning)
Add initial Python module files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial Python module files"}
openai/gpt-5.6-luna:medium PASS 84.7% 195 in → 13 out (0 reasoning)
Add initial config, main, and utility files
openai/gpt-5.6-sol:high PASS 84.7% 195 in → 12 out (0 reasoning)
Add config, main, and utility modules
poolside/laguna-m.1:high PASS 84.7% 237 in → 391 out (379 reasoning)
Add config, main, and utils modules.
tencent/hy3:low PASS 84.7% 201 in → 679 out (667 reasoning)
Add config.py, main.py, and utils.py
x-ai/grok-4.3:max PASS 84.7% 371 in → 548 out (539 reasoning)
Add config.py, main.py, utils.py
z-ai/glm-5.2:none__json_schema PASS 84.7% 198 in → 16 out (0 reasoning)
Add config, main, and utils
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils" }
anthropic/claude-opus-4.7:xhigh__json_schema PASS 84.3% 502 in → 27 out (0 reasoning)
Add initial project files: config, main, utils
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files: config, main, utils"}
anthropic/claude-sonnet-4.6:low__json_schema PASS 84.3% 408 in → 14 out (0 reasoning)
Add initial project structure files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project structure files"}
deepseek/deepseek-v4-flash-0731:max PASS 84.3% 288 in → 419 out (409 reasoning)
Add initial config, main, and utils files
deepseek/deepseek-v4-flash:xhigh PASS 84.3% 275 in → 941 out (930 reasoning)
Add config.py, main.py, utils.py
deepseek/deepseek-v4-pro:medium PASS 84.3% 196 in → 301 out (289 reasoning)
Add config.py, main.py, and utils.py
google/gemini-3.5-flash-lite:medium__json_schema PASS 84.3% 308 in → 15 out (0 reasoning)
Add initial files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial files" }
moonshotai/kimi-k2.7-code:high__json_schema PASS 84.3% 197 in → 286 out (271 reasoning)
Add initial Python files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial Python files" }
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 84.3% 239 in → 408 out (331 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, utils.py" }
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 84.3% 239 in → 616 out (521 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, utils.py" }
openai/gpt-5.4-nano:medium PASS 84.3% 195 in → 13 out (0 reasoning)
Add initial config, main, and utils files
openai/gpt-5.6-sol:max__json_schema PASS 84.3% 233 in → 136 out (113 reasoning)
Add config, main, and utility modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utility modules"}
openai/gpt-oss-120b:medium__json_schema PASS 84.3% 243 in → 248 out (203 reasoning)
Add config.py, main.py, and utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, and utils.py" }
openai/gpt-oss-20b:xhigh PASS 84.3% 256 in → 323 out (306 reasoning)
Add config.py, main.py, utils.py
qwen/qwen3.7-max:low PASS 84.3% 233 in → 982 out (972 reasoning)
Add initial Python project files
thinkingmachines/inkling-small:max PASS 84.3% 204 in → 474 out (348 reasoning)
Add config.py, main.py, and utils.py
x-ai/grok-4.3:low__json_schema PASS 84.3% 437 in → 254 out (240 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, utils.py"}
liquid/lfm-2-24b-a2b PASS 84% 214 in → 4 out
Add new modules
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 84% 239 in → 410 out (276 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, utils.py" }
openai/gpt-5.4-nano:low__json_schema PASS 84% 233 in → 22 out (0 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial config, main, and utils files"}
qwen/qwen3.6-27b:low PASS 84% 233 in → 1,173 out (926 reasoning)
Add config.py, main.py, utils.py
tencent/hy3:medium PASS 84% 201 in → 890 out (879 reasoning)
Add config.py, main.py, utils.py
x-ai/grok-4.5:max PASS 83.7% 408 in → 172 out (163 reasoning)
Add initial config, main, and utils files
arcee-ai/trinity-mini:high PASS 83.3% 195 in → 266 out (284 reasoning)
Add config,main, utils files
google/gemini-3-flash-preview:low__json_schema PASS 83.3% 308 in → 377 out (363 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
google/gemini-3-flash-preview:xhigh PASS 83.3% 245 in → 501 out (493 reasoning)
Add main, config, and utils files
mistralai/mistral-medium-3-5:none__json_schema PASS 83.3% 226 in → 11 out (0 reasoning)
Add new source files
JSON Schema Structured Output
(raw) {"commit_message": "Add new source files"}
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 83.3% 239 in → 21 out (0 reasoning)
Add basic config.py, main.py, and utils.py files
JSON Schema Structured Output
(raw) {"commit_message": "Add basic config.py, main.py, and utils.py files"}
openai/gpt-5.5:medium PASS 83.3% 195 in → 46 out (35 reasoning)
Add initial Python module files
openai/gpt-5.6-terra:medium__json_schema PASS 83.3% 233 in → 21 out (0 reasoning)
Add config, main, and utility modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utility modules"}
poolside/laguna-xs-2.1:medium__json_schema PASS 83.3% 237 in → 946 out (929 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message": "Add config.py, main.py, utils.py"}
poolside/laguna-xs-2.1:none PASS 83.3% 237 in → 9 out (0 reasoning)
Add config, main, and utils module files
poolside/laguna-xs.2:high PASS 83.3% 274 in → 288 out (277 reasoning)
Add config, main, and utils files
poolside/laguna-xs.2:none PASS 83.3% 274 in → 11 out (0 reasoning)
Add initial config, main, and utils files
qwen/qwen3.6-27b:medium__json_schema PASS 83.3% 233 in → 1,886 out (1 reasoning)
Add config.py, main.py, and utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, and utils.py" }
qwen/qwen3.6-27b:xhigh__json_schema PASS 83.3% 233 in → 1,011 out (754 reasoning)
Add config.py, main.py, and utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, and utils.py" }
qwen/qwen3.7-flash:high PASS 83.3% 233 in → 1,459 out (1,445 reasoning)
Add config, main.py, and utils.py
x-ai/grok-4.3:high PASS 83.3% 371 in → 589 out (580 reasoning)
Add config.py, main.py, utils.py
x-ai/grok-4.3:max__json_schema PASS 83.3% 431 in → 448 out (434 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, utils.py"}
x-ai/grok-4.5:low__json_schema PASS 83.3% 480 in → 231 out (215 reasoning)
Add initial main, config, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial main, config, and utils files"}
arcee-ai/trinity-large-thinking:xhigh PASS 83% 195 in → 571 out (560 reasoning)
Add config.py, main.py, utils.py
openai/gpt-oss-20b:low PASS 83% 243 in → 63 out (40 reasoning)
Add config.py, main.py, and utils.py modules.
google/gemini-3.1-pro-preview:high PASS 82.7% 245 in → 650 out (637 reasoning)
Add main.py, config.py, and utils.py
google/gemini-3.1-pro-preview:low PASS 82.7% 245 in → 399 out (385 reasoning)
Add main.py, config.py, and utils.py files
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 82.7% 308 in → 210 out (191 reasoning)
Add config.py, main.py, and utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, and utils.py"}
moonshotai/kimi-k2.6:xhigh PASS 82.7% 197 in → 974 out (938 reasoning)
Add config, main, and utils files
openai/gpt-5.4:high__json_schema PASS 82.7% 233 in → 156 out (132 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add initial config, main, and utils files"}
openai/gpt-5.6-sol:low PASS 82.7% 195 in → 12 out (0 reasoning)
Add config, main, and utility modules
openai/gpt-5.6-sol:max PASS 82.7% 195 in → 100 out (86 reasoning)
Add config, main, and utility modules
tencent/hy3:xhigh PASS 82.7% 201 in → 1,116 out (1,104 reasoning)
Add config.py, main.py, and utils.py
x-ai/grok-4.3:high__json_schema PASS 82.7% 431 in → 502 out (488 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message":"Add config.py, main.py, utils.py"}
x-ai/grok-4.3:low PASS 82.7% 377 in → 321 out (312 reasoning)
Add config.py, main.py, utils.py
z-ai/glm-4.7-flash:none PASS 82.7% 197 in → 3 out (0 reasoning)
Add module structure
z-ai/glm-4.7-flash:medium PASS 82.5% 197 in → 696 out (662 reasoning)
Add new files
openai/gpt-5.4-nano:high__json_schema PASS 82.3% 233 in → 187 out (164 reasoning)
Add initial config, main, utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add initial config, main, utils modules"}
openai/gpt-5.6-sol:xhigh__json_schema PASS 82.3% 233 in → 21 out (0 reasoning)
Add config, main, and utility modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utility modules"}
openai/gpt-oss-120b:xhigh PASS 82.3% 256 in → 283 out (275 reasoning)
Add config, main, and utils modules
google/gemini-3.5-flash:xhigh PASS 82% 245 in → 645 out (632 reasoning)
Add main.py, config.py, and utils.py
minimax/minimax-m2.7:xhigh PASS 82% 227 in → 288 out (240 reasoning)
Add config.py, main.py, utils.py
arcee-ai/trinity-mini:medium PASS 81.7% 195 in → 319 out (333 reasoning)
Add config main utils
deepseek/deepseek-v4-flash:low PASS 81.7% 196 in → 142 out (134 reasoning)
Add config, main, utils
google/gemini-3.1-pro-preview:xhigh PASS 81.7% 245 in → 734 out (721 reasoning)
Add main.py, config.py, and utils.py
google/gemini-3.5-flash:high PASS 81.7% 245 in → 559 out (546 reasoning)
Add main.py, config.py, and utils.py
google/gemini-3.6-flash:xhigh PASS 81.7% 246 in → 467 out (454 reasoning)
Add main.py, config.py, and utils.py
minimax/minimax-m3:low__json_schema PASS 81.7% 313 in → 19 out (18 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial config, main, and utils files" }
moonshotai/kimi-k2.7-code:low PASS 81.7% 197 in → 251 out (239 reasoning)
Add config.py, main.py, and utils.py
openai/gpt-5.4-nano:xhigh PASS 81.7% 195 in → 207 out (196 reasoning)
Add config/main/utils modules
openai/gpt-oss-20b:high PASS 81.7% 243 in → 205 out (188 reasoning)
Add config.py, main.py, utils.py
poolside/laguna-xs.2:xhigh PASS 81.7% 274 in → 255 out (249 reasoning)
Add core files
poolside/laguna-xs.2:xhigh__json_schema PASS 81.7% 274 in → 734 out (717 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) {"commit_message": "Add config.py, main.py, utils.py"}
qwen/qwen3.6-27b:high PASS 81.7% 233 in → 1,476 out (1,281 reasoning)
Add main.py, config.py, utils.py
openai/gpt-5.6-terra:high__json_schema PASS 81.3% 233 in → 21 out (0 reasoning)
Add config, main, and utility modules
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utility modules"}
thinkingmachines/inkling:none__json_schema PASS 81.3% 202 in → 19 out (0 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add initial config, main, and utils files" }
arcee-ai/trinity-mini:xhigh__json_schema PASS 81% 195 in → 313 out (324 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, utils.py" }
google/gemini-3-flash-preview:medium PASS 81% 245 in → 460 out (447 reasoning)
Add config.py, main.py, and utils.py
openai/gpt-5.6-terra:high PASS 81% 195 in → 13 out (0 reasoning)
Add initial config, main, and utils files
deepseek/deepseek-v4-pro:xhigh PASS 80.7% 196 in → 318 out (331 reasoning)
Add config.py,main.py, utils.py
deepseek/deepseek-v4-flash-0731:medium PASS 80% 275 in → 153 out (143 reasoning)
Add config.py, main.py, utils.py
google/gemini-3.1-pro-preview:medium PASS 80% 245 in → 523 out (510 reasoning)
Add config.py, main.py, and utils.py
openai/gpt-5.4-nano:high PASS 80% 195 in → 13 out (0 reasoning)
Add initial config, main, and utils files
poolside/laguna-m.1:none PASS 80% 237 in → 11 out (0 reasoning)
Add initial config, main, and utils files
qwen/qwen3.6-35b-a3b:high__json_schema PASS 80% 233 in → 1,115 out (895 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) { "commit_message": "Add config, main, and utils files" }
anthropic/claude-opus-4.7:max__json_schema PASS 79.3% 502 in → 27 out (0 reasoning)
Add initial project files: config, main, utils
JSON Schema Structured Output
(raw) {"commit_message":"Add initial project files: config, main, utils"}
tencent/hy3:low__json_schema PASS 79.3% 201 in → 786 out (615 reasoning)
Add config.py, main.py and utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py and utils.py" }
anthropic/claude-sonnet-4.6:none__json_schema PASS 78.3% 408 in → 17 out (0 reasoning)
Add main, config, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add main, config, and utils modules"}
thinkingmachines/inkling-small:high PASS 78.3% 204 in → 437 out (421 reasoning)
Add config.py, main.py, and utils.py
arcee-ai/trinity-mini:xhigh PASS 77.7% 195 in → 210 out (228 reasoning)
Add new files
openai/gpt-5.6-terra:none PASS 77.7% 195 in → 13 out (0 reasoning)
Add initial config, main, and utility files
mistralai/mistral-medium-3-5:medium PASS 77.3% 238 in → 5,840 out (6,073 reasoning)
Add config.py, main.py, utils.py
anthropic/claude-haiku-4.5:none PASS 76.7% 238 in → 14 out (0 reasoning)
Add initial project files (config, main, utils)
google/gemini-3.5-flash:medium PASS 76.7% 245 in → 495 out (482 reasoning)
Add config.py, main.py, and utils.py
google/gemini-3.5-flash:xhigh__json_schema PASS 76.7% 308 in → 545 out (530 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
minimax/minimax-m2.5:low PASS 76.7% 227 in → 284 out (324 reasoning)
feat: add initial project files main.py, config.py, and utils.py
anthropic/claude-opus-4.7:xhigh PASS 75% 281 in → 17 out (0 reasoning)
Add initial config, main, and utils modules
mistralai/devstral-2512 PASS 73.3% 226 in → 11 out
"Add config, main, and utils files"
anthropic/claude-sonnet-4.6:max__json_schema PASS 72.7% 408 in → 84 out (56 reasoning)
Add main, config, and utils modules
JSON Schema Structured Output
(raw) {"commit_message":"Add main, config, and utils modules"}
anthropic/claude-sonnet-5:none__json_schema PASS 72.7% 497 in → 23 out (0 reasoning)
Add initial config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add initial config, main, and utils files"}
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 71.7% 246 in → 294 out (280 reasoning)
Initialize project files
JSON Schema Structured Output
(raw) { "commit_message": "Initialize project files" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 66% 245 in → 316 out (313 reasoning)
Initialize project files
openai/gpt-5.4-mini:none__json_schema PASS 66% 233 in → 21 out (0 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message":"Add config, main, and utils files"}
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 64.3% 239 in → 129 out (116 reasoning)
Addconfig.py, main.py, and utils.py files.
poolside/laguna-xs.2:low PASS 63.3% 274 in → 317 out (305 reasoning)
Add config, main, and utils modules
qwen/qwen3.6-35b-a3b:xhigh PASS 61.7% 233 in → 1,324 out (1,115 reasoning)
Add config, main, and utils files
minimax/minimax-m3:medium PASS 61.3% 365 in → 50 out (46 reasoning)
Init project structure with config, main, utils
arcee-ai/trinity-mini:low__json_schema PASS 61% 195 in → 280 out (293 reasoning)
Addconfig, main, utils files
JSON Schema Structured Output
(raw) { "commit_message": "Addconfig, main, utils files" }
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 60% 246 in → 124 out (114 reasoning)
Initialize project structure
JSON Schema Structured Output
(raw) {"commit_message": "Initialize project structure"}
thinkingmachines/inkling:high__json_schema PASS 60% 204 in → 652 out (633 reasoning)
Add config, main, and utils files
JSON Schema Structured Output
(raw) {"commit_message": "Add config, main, and utils files"}
anthropic/claude-opus-4.7:medium PASS 58.3% 281 in → 16 out (0 reasoning)
Add initial config, main, and utils files
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 56.7% 239 in → 529 out (327 reasoning)
Add config.py, main.py, utils.py
JSON Schema Structured Output
(raw) { "commit_message": "Add config.py, main.py, utils.py" }
nvidia/nemotron-3-super-120b-a12b:low PASS 56.7% 239 in → 123 out (107 reasoning)
Add config.py, main.py, utils.py
anthropic/claude-haiku-4.5:high FAIL 48.3% 267 in → 1,424 out (452 reasoning)
# Add config, main, and utils files **Character count: 33/50**
nvidia/nemotron-3-nano-30b-a3b:none FAIL 38.3% 239 in → 11 out (0 reasoning)
Add config and utils modules + update main entry point
anthropic/claude-haiku-4.5:xhigh FAIL 6.7% 267 in → 594 out (397 reasoning)
# Add initial project files This commit message (25 characters) clearly indicates that three new files (config.py, main.py, utils.py) are being added to the project, following the imperative mood convention.
anthropic/claude-fable-5:high FAIL 5% 276 in → 90 out (0 reasoning)
**Commit message:** ``` Add initial project files ``` This message is concise (26 characters), starts with the action verb "Add", and accurately describes the creation of three new placeholder files (`config.py`, `main.py`, `utils.py`) that scaffold the project structure.
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: Add new modules
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:xhigh__json_schema FAIL 0%
Invalid JSON. Output: ```json {"commit_message": "Add config, main, and utils files"} ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k3:high FAIL 0% 274 in → 220 out (160 reasoning)
**Commit message:** ``` Add initial config, main, and utils files ``` (38 characters) Alternative options: - `Add initial project scaffolding` (31 chars) - `Add core project modules` (24 chars)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:high__json_schema FAIL 0% 274 in → 138 out (117 reasoning)
**Add initial project structure**
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low FAIL 0% 274 in → 210 out (155 reasoning)
``` Add initial project files ``` (25 characters) Alternative options: - `Add config, main, and utils modules` (35 chars) - `Add initial project scaffolding` (31 chars)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low__json_schema FAIL 0% 274 in → 322 out (265 reasoning)
**Commit message:** ``` Add initial config, main, and utils files ``` *(41 characters)* This message uses the imperative action "Add" and clearly identifies the three new files introduced in the diff.
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max FAIL 0% 274 in → 174 out (137 reasoning)
**Add initial project files (config, main, utils)** Alternative shorter option: `Add initial project scaffolding`
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max__json_schema FAIL 0% 274 in → 223 out (168 reasoning)
**Commit message:** ``` Add initial project files ``` Alternative options: - `Add config, main, and utils modules` (35 chars) - `Add initial project scaffolding` (31 chars)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium FAIL 0% 274 in → 191 out (140 reasoning)
**Add config, main, and utils files** Alternative options: - `Add initial project files` (25 chars) - `Initialize project with core modules` (36 chars)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium__json_schema FAIL 0% 274 in → 343 out (276 reasoning)
**Commit message:** ``` Add initial config, main, and utils files ``` *(35 characters)* This message follows the imperative convention, starts with the action verb "Add", and accurately summarizes the diff, which creates three new Python files with placeholder content.
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh FAIL 0% 274 in → 132 out (113 reasoning)
Add initial project files
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh__json_schema FAIL 0% 274 in → 350 out (296 reasoning)
**Commit message:** ``` Add config, main, and utils files ``` (33 characters) This message clearly states the action (Add) and identifies the three new files created in the diff.
Failure: Scoring error: llm_judge requires a judge client
nvidia/nemotron-3-super-120b-a12b:medium FAIL 0% 239 in → 304 out (211 reasoning)
Add config, main, and utils files
Failure: Scoring error: llm_judge requires a judge client
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c75a6f51-b9ce-975b-985d-0084889cdd49","request_id":"c75a6f51-b9ce-975b-985d-0084889cdd49"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1bbaad62-4cce-964f-ac73-4eea476f268b","request_id":"1bbaad62-4cce-964f-ac73-4eea476f268b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c2528d70-d405-97c2-9c20-e32af94746a9","request_id":"c2528d70-d405-97c2-9c20-e32af94746a9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f44798e3-2894-9abf-8d29-9f7d4bfd5f65","request_id":"f44798e3-2894-9abf-8d29-9f7d4bfd5f65"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-614de3cf-9989-9629-86d9-5976022e4e6b","request_id":"614de3cf-9989-9629-86d9-5976022e4e6b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8339a895-d285-94b2-876f-1c2256731dc4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7ee4457f-e88f-900e-97f3-7122ad894378"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6908b5aa-4e21-9cd5-ac1b-fe47fad5f832"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d1bdeece-232b-9def-ad0c-d0132cdb460e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d11dc081-62f5-9f44-b8ba-ee23c7691e36"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh FAIL 0% 233 in → 712 out (700 reasoning)
Add config, main, and utils
Failure: Scoring error: llm_judge requires a judge client
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b8e7b410-8376-9fc8-b353-1a065da13b2d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d7d04221-be7b-989d-a333-49a3d825bafc","request_id":"d7d04221-be7b-989d-a333-49a3d825bafc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e2ae5902-e2ef-9371-a491-a06945840a1d","request_id":"e2ae5902-e2ef-9371-a491-a06945840a1d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1f3c07ac-517e-99a2-b400-81b8f2702625","request_id":"1f3c07ac-517e-99a2-b400-81b8f2702625"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9c1cdac8-f73e-9bca-aa31-277d7367162b","request_id":"9c1cdac8-f73e-9bca-aa31-277d7367162b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-404c2ce0-0b26-9185-a137-f3f4447d5e61","request_id":"404c2ce0-0b26-9185-a137-f3f4447d5e61"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4e187f76-8400-9a30-9cf4-9bcd9242cc0e","request_id":"4e187f76-8400-9a30-9cf4-9bcd9242cc0e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8c4a2e4d-40cb-9191-a146-2ac02f94ebf9","request_id":"8c4a2e4d-40cb-9191-a146-2ac02f94ebf9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7c650312-f6f5-95e5-8d21-a2d0bec3458e","request_id":"7c650312-f6f5-95e5-8d21-a2d0bec3458e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0e4edd1d-c9df-9084-8b53-03aed5850e73","request_id":"0e4edd1d-c9df-9084-8b53-03aed5850e73"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-58b3e80d-c962-98fd-82fd-b1193d387060","request_id":"58b3e80d-c962-98fd-82fd-b1193d387060"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: Add new Python files{"commit_message": "Add new Python files"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: Add config, main, and utils files
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:max__json_schema FAIL 0%
Invalid JSON. Output: Add initial project files
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: Add config, main, and utils files
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Add config, main, and utils files
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output: Add config, main, utils
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)