f005 — commit_messages

← commit_messages
File modification
Tests ability to generate a commit message for a single-file content modification. Evaluates describing incremental changes.
easy commit-message modification update
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'original content' > app.py
  5. 05 git add app.py
  6. 06 git commit -m 'Initial commit'
  7. 07 echo 'updated content' > app.py
  8. 08 git add app.py
Prompt
Generate a concise commit message (max 50 characters) describing the changes in this diff. Focus on the action taken (e.g., 'Add', 'Fix', 'Update').
Expected
Update app.py with new content
Fixture Metrics

Scoped model quality, cost, API time, and token usage for commit_messages/f005.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
x-ai/grok-4.3:low PASS 95% 280 in → 330 out (326 reasoning)
Update app.py content
minimax/minimax-m2.7:low__json_schema PASS 94.7% 265 in → 262 out (243 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
anthropic/claude-opus-4.7:none__json_schema PASS 94.3% 364 in → 20 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
arcee-ai/trinity-mini:low__json_schema PASS 94% 98 in → 443 out (486 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
google/gemini-3.1-flash-lite-preview:high PASS 94% 109 in → 320 out (315 reasoning)
Update app.py content
moonshotai/kimi-k2.6:xhigh__json_schema PASS 94% 98 in → 443 out (481 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
openai/gpt-5.6-terra:xhigh PASS 94% 96 in → 34 out (25 reasoning)
Update app content
z-ai/glm-4.7-flash:medium PASS 94% 96 in → 345 out (329 reasoning)
Update app.py
anthropic/claude-haiku-4.5:none PASS 93.3% 117 in → 8 out (0 reasoning)
Update app.py content
anthropic/claude-opus-4.7:none PASS 93.3% 143 in → 10 out (0 reasoning)
Update app.py content
anthropic/claude-sonnet-4.6:low PASS 93.3% 117 in → 10 out (0 reasoning)
`Update app.py content`
anthropic/claude-sonnet-4.6:low__json_schema PASS 93.3% 287 in → 14 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
arcee-ai/trinity-large-thinking:xhigh PASS 93.3% 98 in → 126 out (119 reasoning)
Update app.py content
google/gemini-3.5-flash-lite:low__json_schema PASS 93.3% 110 in → 17 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
minimax/minimax-m2.5:high PASS 93.3% 129 in → 124 out (139 reasoning)
Update app.py content
minimax/minimax-m2.7:high PASS 93.3% 128 in → 213 out (227 reasoning)
Update app.py content
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 93.3% 103 in → 286 out (294 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 93.3% 116 in → 133 out (112 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
openai/gpt-oss-120b:xhigh__json_schema PASS 93.3% 155 in → 167 out (171 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
poolside/laguna-m.1:medium PASS 93.3% 113 in → 175 out (168 reasoning)
Update app.py content
poolside/laguna-m.1:xhigh__json_schema PASS 93.3% 113 in → 265 out (248 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
poolside/laguna-xs-2.1:xhigh__json_schema PASS 93.3% 113 in → 311 out (300 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
qwen/qwen3.6-flash:high PASS 93.3% 109 in → 1,390 out (1,381 reasoning)
Update app.py content
thinkingmachines/inkling-small:high PASS 93.3% 105 in → 91 out (88 reasoning)
Update app.py content
google/gemma-4-31b-it:medium PASS 92.7% 125 in → 168 out (147 reasoning)
Update app.py
inclusionai/ling-2.6-flash__json_schema PASS 92.7% 120 in → 10 out
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
minimax/minimax-m2.5:low__json_schema PASS 92.7% 129 in → 149 out (145 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 92.7% 116 in → 15 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update content in app.py" }
openai/gpt-5.4-mini:medium PASS 92.7% 96 in → 62 out (53 reasoning)
Update app.py
openai/gpt-5.6-sol:none PASS 92.7% 96 in → 7 out (0 reasoning)
Update app content
qwen/qwen3.6-35b-a3b:high__json_schema PASS 92.7% 109 in → 1,070 out (941 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
thinkingmachines/inkling:low PASS 92.7% 105 in → 38 out (28 reasoning)
Update app.py content
x-ai/grok-4.3:none PASS 92.7% 272 in → 4 out (0 reasoning)
Update app.py content
anthropic/claude-sonnet-4.6:medium PASS 92% 117 in → 10 out (0 reasoning)
`Update app.py content`
anthropic/claude-sonnet-4.6:medium__json_schema PASS 92% 287 in → 14 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
google/gemini-3.5-flash:xhigh PASS 92% 109 in → 372 out (367 reasoning)
Update app.py content
minimax/minimax-m3:low PASS 92% 267 in → 37 out (39 reasoning)
Update app.py content
mistralai/mistral-medium-3-5:none__json_schema PASS 92% 103 in → 11 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
openai/gpt-5.4-mini:medium__json_schema PASS 92% 134 in → 66 out (48 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-terra:medium__json_schema PASS 92% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
z-ai/glm-4.7-flash:low PASS 92% 96 in → 387 out (370 reasoning)
Update app.py
anthropic/claude-fable-5:high__json_schema PASS 91.7% 359 in → 30 out (16 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
anthropic/claude-fable-5:medium PASS 91.7% 138 in → 9 out (0 reasoning)
Update content in app.py
anthropic/claude-haiku-4.5:medium PASS 91.7% 146 in → 200 out (175 reasoning)
``` Update app.py content ``` (23 characters)
anthropic/claude-opus-4.6:xhigh PASS 91.7% 117 in → 9 out (0 reasoning)
Update content in app.py
anthropic/claude-opus-4.7:max PASS 91.7% 143 in → 73 out (32 reasoning)
Update content in app.py
anthropic/claude-opus-4.7:max__json_schema PASS 91.7% 364 in → 20 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
anthropic/claude-opus-4.8:none PASS 91.7% 138 in → 8 out (0 reasoning)
Update app.py content
anthropic/claude-sonnet-5:low PASS 91.7% 138 in → 8 out (0 reasoning)
Update app.py content
anthropic/claude-sonnet-5:xhigh__json_schema PASS 91.7% 359 in → 18 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
arcee-ai/trinity-mini:medium__json_schema PASS 91.7% 98 in → 321 out (334 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 91.7% 102 in → 143 out (137 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
deepseek/deepseek-v4-flash:medium PASS 91.7% 96 in → 206 out (199 reasoning)
Update app.py
google/gemini-3-flash-preview:high PASS 91.7% 109 in → 331 out (326 reasoning)
Update app.py content
google/gemini-3-flash-preview:high__json_schema PASS 91.7% 172 in → 450 out (438 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
google/gemini-3-flash-preview:low__json_schema PASS 91.7% 172 in → 310 out (299 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
google/gemini-3.5-flash-lite:high__json_schema PASS 91.7% 172 in → 401 out (390 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
google/gemini-3.6-flash:max PASS 91.7% 110 in → 352 out (346 reasoning)
Update content in app.py
google/gemma-4-26b-a4b-it:medium__json_schema PASS 91.7% 125 in → 455 out (415 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
google/gemma-4-26b-a4b-it:none__json_schema PASS 91.7% 122 in → 18 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
google/gemma-4-31b-it:low PASS 91.7% 125 in → 149 out (122 reasoning)
Update app.py
google/gemma-4-31b-it:none__json_schema PASS 91.7% 122 in → 14 out (0 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
google/gemma-4-31b-it:xhigh__json_schema PASS 91.7% 125 in → 238 out (193 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
minimax/minimax-m2.5:low PASS 91.7% 129 in → 101 out (112 reasoning)
Update app.py content
minimax/minimax-m2.7:high__json_schema PASS 91.7% 265 in → 173 out (162 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
minimax/minimax-m2.7:low PASS 91.7% 131 in → 175 out (170 reasoning)
Update app.py content
minimax/minimax-m3:high PASS 91.7% 267 in → 51 out (52 reasoning)
Update app.py content
minimax/minimax-m3:high__json_schema PASS 91.7% 254 in → 14 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
mistralai/mistral-medium-3-5:low__json_schema PASS 91.7% 103 in → 961 out (1,088 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
mistralai/mistral-medium-3-5:medium__json_schema PASS 91.7% 103 in → 130 out (131 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
moonshotai/kimi-k2.6:medium__json_schema PASS 91.7% 98 in → 272 out (301 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
moonshotai/kimi-k2.7-code:low__json_schema PASS 91.7% 98 in → 230 out (218 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 91.7% 116 in → 495 out (342 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 91.7% 116 in → 224 out (192 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 91.7% 116 in → 71 out (75 reasoning)
Update app.py content
nvidia/nemotron-3-super-120b-a12b:none PASS 91.7% 116 in → 3 out (0 reasoning)
Update content
openai/gpt-5.4-mini:low__json_schema PASS 91.7% 134 in → 31 out (13 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.4-mini:xhigh PASS 91.7% 96 in → 83 out (73 reasoning)
Update app.py content
openai/gpt-5.4-nano:high PASS 91.7% 96 in → 50 out (40 reasoning)
Update app.py content
openai/gpt-5.4-nano:medium PASS 91.7% 96 in → 8 out (0 reasoning)
Update app.py content
openai/gpt-5.4-nano:medium__json_schema PASS 91.7% 134 in → 45 out (26 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-5.4:high__json_schema PASS 91.7% 134 in → 82 out (63 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-5.4:low__json_schema PASS 91.7% 134 in → 30 out (11 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-5.4:none__json_schema PASS 91.7% 134 in → 17 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-5.5:high PASS 91.7% 96 in → 42 out (33 reasoning)
Update app content
openai/gpt-5.5:low PASS 91.7% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-5.5:xhigh__json_schema PASS 91.7% 134 in → 74 out (56 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-sol:medium__json_schema PASS 91.7% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-terra:medium PASS 91.7% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-oss-120b:high__json_schema PASS 91.7% 228 in → 143 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
openai/gpt-oss-120b:low PASS 91.7% 168 in → 30 out (0 reasoning)
Update app.py content
openai/gpt-oss-120b:xhigh PASS 91.7% 159 in → 400 out (309 reasoning)
Update app.py content
openai/gpt-oss-20b:high__json_schema PASS 91.7% 144 in → 354 out (322 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
poolside/laguna-m.1:low PASS 91.7% 113 in → 209 out (202 reasoning)
Update app.py content
poolside/laguna-xs-2.1:medium PASS 91.7% 113 in → 166 out (161 reasoning)
Update app.py content
poolside/laguna-xs.2:medium PASS 91.7% 150 in → 168 out (161 reasoning)
Update app.py content
poolside/laguna-xs.2:none__json_schema PASS 91.7% 150 in → 11 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
qwen/qwen3.6-27b:none__json_schema PASS 91.7% 111 in → 16 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
qwen/qwen3.6-27b:xhigh__json_schema PASS 91.7% 109 in → 1,200 out (1,128 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
qwen/qwen3.6-35b-a3b:high PASS 91.7% 109 in → 1,450 out (1,305 reasoning)
Update app.py content
qwen/qwen3.6-flash:low PASS 91.7% 109 in → 969 out (960 reasoning)
Update app.py content
qwen/qwen3.7-plus:low PASS 91.7% 109 in → 779 out (770 reasoning)
Update app.py content
qwen/qwen3.7-plus:xhigh PASS 91.7% 109 in → 964 out (955 reasoning)
Update app.py content
tencent/hy3:low PASS 91.7% 103 in → 390 out (384 reasoning)
Update app.py content
tencent/hy3:xhigh PASS 91.7% 103 in → 776 out (770 reasoning)
Update app.py content
thinkingmachines/inkling-small:max PASS 91.7% 105 in → 71 out (71 reasoning)
Update app.py content
thinkingmachines/inkling-small:max__json_schema PASS 91.7% 105 in → 267 out (252 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
thinkingmachines/inkling-small:medium__json_schema PASS 91.7% 105 in → 240 out (225 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
x-ai/grok-4.5:medium__json_schema PASS 91.7% 374 in → 319 out (308 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
x-ai/grok-4.5:xhigh PASS 91.7% 302 in → 209 out (205 reasoning)
Update app.py content
x-ai/grok-4.5:xhigh__json_schema PASS 91.7% 374 in → 263 out (251 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
z-ai/glm-4.7-flash:xhigh PASS 91.7% 96 in → 372 out (350 reasoning)
Update app.py
z-ai/glm-5.2:low__json_schema PASS 91.7% 96 in → 421 out (368 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
z-ai/glm-5.2:none__json_schema PASS 91.7% 97 in → 13 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 91.3% 316 in → 626 out (573 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
anthropic/claude-sonnet-5:xhigh PASS 91.3% 138 in → 33 out (25 reasoning)
Update content in app.py
google/gemini-3.6-flash:low__json_schema PASS 91.3% 110 in → 12 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
minimax/minimax-m2.5:xhigh__json_schema PASS 91.3% 141 in → 160 out (169 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
moonshotai/kimi-k2.7-code:xhigh PASS 91.3% 98 in → 124 out (118 reasoning)
Update app.py content
openai/gpt-5.6-luna:none PASS 91.3% 96 in → 7 out (0 reasoning)
Update app content
poolside/laguna-xs.2:low PASS 91.3% 150 in → 440 out (434 reasoning)
Update app.py
qwen/qwen3.7-plus:high PASS 91.3% 109 in → 664 out (655 reasoning)
Update app.py content
thinkingmachines/inkling-small:xhigh__json_schema PASS 91.3% 105 in → 198 out (181 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
x-ai/grok-4.5:high__json_schema PASS 91.3% 374 in → 11 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
anthropic/claude-fable-5:max PASS 91% 138 in → 369 out (72 reasoning)
``` Update content in app.py ``` This message (24 characters) uses the imperative mood, identifies the action ("Update"), and specifies the affected file, staying well within the 50-character limit.
anthropic/claude-fable-5:medium__json_schema PASS 91% 359 in → 18 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
anthropic/claude-haiku-4.5:low PASS 91% 146 in → 273 out (221 reasoning)
Update app.py content
anthropic/claude-haiku-4.5:xhigh PASS 91% 146 in → 524 out (478 reasoning)
Update app.py
anthropic/claude-opus-4.7:medium__json_schema PASS 91% 364 in → 20 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
anthropic/claude-opus-4.7:xhigh PASS 91% 143 in → 11 out (0 reasoning)
Update content in app.py
anthropic/claude-opus-4.8:max__json_schema PASS 91% 359 in → 19 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
anthropic/claude-sonnet-4.6:none PASS 91% 117 in → 10 out (0 reasoning)
`Update app.py content`
arcee-ai/trinity-large-thinking:high PASS 91% 98 in → 352 out (345 reasoning)
Update app.py content
arcee-ai/trinity-mini:xhigh__json_schema PASS 91% 98 in → 282 out (315 reasoning)
Update content
JSON Schema Structured Output
(raw) { "commit_message": "Update content" }
deepseek/deepseek-v4-flash-0731:high PASS 91% 175 in → 142 out (138 reasoning)
Update app.py content
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 91% 96 in → 187 out (176 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
deepseek/deepseek-v4-flash-0731:medium PASS 91% 96 in → 107 out (98 reasoning)
Update app.py content
deepseek/deepseek-v4-pro:low__json_schema PASS 91% 287 in → 83 out (71 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
deepseek/deepseek-v4-pro:medium__json_schema PASS 91% 287 in → 199 out (187 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
deepseek/deepseek-v4-pro:none PASS 91% 96 in → 4 out (0 reasoning)
Update app.py content
google/gemini-3-flash-preview:medium PASS 91% 109 in → 344 out (339 reasoning)
Update app.py content
google/gemini-3.5-flash-lite:max PASS 91% 110 in → 338 out (333 reasoning)
Update app.py content
google/gemini-3.5-flash-lite:medium PASS 91% 109 in → 260 out (255 reasoning)
Update app.py content
google/gemini-3.5-flash-lite:medium__json_schema PASS 91% 110 in → 228 out (218 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py"}
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 91% 110 in → 474 out (463 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
google/gemini-3.5-flash:low__json_schema PASS 91% 172 in → 95 out (78 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
google/gemini-3.6-flash:medium PASS 91% 110 in → 312 out (306 reasoning)
Update content in app.py
google/gemini-3.6-flash:xhigh PASS 91% 109 in → 297 out (292 reasoning)
Update app.py content
google/gemma-4-26b-a4b-it:high PASS 91% 125 in → 480 out (415 reasoning)
Update app.py
minimax/minimax-m3:xhigh PASS 91% 267 in → 66 out (70 reasoning)
Update app.py content
mistralai/mistral-medium-3-5:none PASS 91% 115 in → 4 out (0 reasoning)
Update app content
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 91% 185 in → 106 out (94 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 91% 116 in → 124 out (118 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 91% 116 in → 88 out (70 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
openai/gpt-5.4-nano:none PASS 91% 96 in → 8 out (0 reasoning)
Update app.py content
openai/gpt-5.5:medium PASS 91% 96 in → 31 out (22 reasoning)
Update app content
openai/gpt-5.5:medium__json_schema PASS 91% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.5:none__json_schema PASS 91% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-sol:low PASS 91% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-5.6-sol:medium PASS 91% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-5.6-sol:xhigh__json_schema PASS 91% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-terra:high__json_schema PASS 91% 134 in → 39 out (21 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-terra:none PASS 91% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-oss-20b:high PASS 91% 159 in → 256 out (214 reasoning)
Update app.py content
openai/gpt-oss-20b:xhigh PASS 91% 162 in → 340 out (378 reasoning)
Update app.py content
poolside/laguna-m.1:high PASS 91% 113 in → 200 out (193 reasoning)
Update app.py content
poolside/laguna-xs.2:none PASS 91% 150 in → 6 out (0 reasoning)
Update app.py content
qwen/qwen3.6-27b:xhigh PASS 91% 109 in → 1,287 out (1,133 reasoning)
Update app.py content
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 91% 109 in → 1,095 out (990 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
qwen/qwen3.6-35b-a3b:none__json_schema PASS 91% 111 in → 16 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
qwen/qwen3.6-35b-a3b:xhigh PASS 91% 109 in → 2,309 out (2,166 reasoning)
Update app.py content
qwen/qwen3.7-flash:xhigh PASS 91% 109 in → 1,175 out (1,166 reasoning)
Update app.py content
qwen/qwen3.7-plus:medium PASS 91% 109 in → 649 out (640 reasoning)
Update app.py content
tencent/hy3:low__json_schema PASS 91% 103 in → 612 out (585 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py"}
tencent/hy3:medium__json_schema PASS 91% 103 in → 875 out (814 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
tencent/hy3:xhigh__json_schema PASS 91% 103 in → 427 out (419 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
thinkingmachines/inkling-small:xhigh PASS 91% 105 in → 135 out (125 reasoning)
Update app.py content
x-ai/grok-4.5:high PASS 91% 302 in → 199 out (195 reasoning)
Update app.py content
x-ai/grok-4.5:low PASS 91% 302 in → 201 out (197 reasoning)
Update app.py content
x-ai/grok-4.5:low__json_schema PASS 91% 374 in → 237 out (226 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
x-ai/grok-4.5:max PASS 91% 302 in → 193 out (188 reasoning)
Update content in app.py
z-ai/glm-5.2:high PASS 91% 103 in → 321 out (299 reasoning)
Update content in app.py
z-ai/glm-5.2:low PASS 91% 103 in → 156 out (148 reasoning)
Update app.py content
anthropic/claude-opus-4.6:xhigh__json_schema PASS 90.7% 287 in → 15 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 90.7% 287 in → 14 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
moonshotai/kimi-k2.7-code:high__json_schema PASS 90.7% 98 in → 136 out (121 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
poolside/laguna-xs-2.1:high__json_schema PASS 90.7% 113 in → 206 out (195 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 90.3% 172 in → 309 out (296 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
google/gemini-3.5-flash:xhigh__json_schema PASS 90.3% 172 in → 470 out (457 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
moonshotai/kimi-k2.6:low__json_schema PASS 90.3% 97 in → 792 out (302 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
openai/gpt-oss-120b:medium PASS 90.3% 155 in → 82 out (79 reasoning)
Update app.py content
qwen/qwen3.7-max:xhigh PASS 90.3% 109 in → 881 out (871 reasoning)
Update content in app.py
anthropic/claude-fable-5:low__json_schema PASS 90% 359 in → 18 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
anthropic/claude-opus-4.6:high PASS 90% 117 in → 9 out (0 reasoning)
Update content in app.py
anthropic/claude-opus-4.6:high__json_schema PASS 90% 287 in → 15 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-opus-4.6:medium PASS 90% 117 in → 8 out (0 reasoning)
Update app.py content
anthropic/claude-opus-4.6:medium__json_schema PASS 90% 287 in → 15 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-opus-4.7:low__json_schema PASS 90% 364 in → 20 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
anthropic/claude-opus-4.7:xhigh__json_schema PASS 90% 364 in → 20 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
anthropic/claude-opus-4.8:high PASS 90% 138 in → 36 out (28 reasoning)
Update content in app.py
anthropic/claude-opus-4.8:high__json_schema PASS 90% 359 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message": "Update app content"}
anthropic/claude-opus-4.8:medium PASS 90% 138 in → 9 out (0 reasoning)
Update content in app.py
anthropic/claude-opus-4.8:none__json_schema PASS 90% 359 in → 18 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-sonnet-4.6:none__json_schema PASS 90% 287 in → 14 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
arcee-ai/trinity-large-thinking:medium PASS 90% 98 in → 113 out (106 reasoning)
Update app.py content
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 90% 242 in → 284 out (284 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
deepseek/deepseek-v4-flash-0731:none PASS 90% 96 in → 5 out (0 reasoning)
Update app.py content
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 90% 96 in → 11 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
deepseek/deepseek-v4-flash:high PASS 90% 96 in → 187 out (181 reasoning)
Update app.py content
deepseek/deepseek-v4-flash:low PASS 90% 96 in → 262 out (255 reasoning)
Update content in app.py
deepseek/deepseek-v4-pro:medium PASS 90% 96 in → 185 out (189 reasoning)
Update app.py content
google/gemini-3-flash-preview:xhigh__json_schema PASS 90% 110 in → 268 out (256 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
google/gemini-3.1-flash-lite-preview:low PASS 90% 110 in → 118 out (112 reasoning)
Update content in app.py
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 90% 110 in → 254 out (241 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
google/gemini-3.1-pro-preview:low PASS 90% 109 in → 285 out (279 reasoning)
Update content in app.py
google/gemini-3.1-pro-preview:medium PASS 90% 109 in → 292 out (286 reasoning)
Update content in app.py
google/gemini-3.1-pro-preview:xhigh PASS 90% 109 in → 370 out (364 reasoning)
Update content in app.py
google/gemini-3.5-flash-lite:high PASS 90% 110 in → 305 out (299 reasoning)
Update content in app.py
google/gemini-3.5-flash:medium__json_schema PASS 90% 172 in → 290 out (279 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
google/gemini-3.6-flash:high PASS 90% 109 in → 214 out (208 reasoning)
Update content in app.py
google/gemma-4-26b-a4b-it:none PASS 90% 122 in → 7 out (0 reasoning)
Update content in app.py
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 90% 124 in → 567 out (524 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
google/gemma-4-31b-it:none PASS 90% 122 in → 7 out (0 reasoning)
Update content in app.py
ibm-granite/granite-4.1-8b__json_schema PASS 90% 110 in → 16 out
Update content in app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update content in app.py" }
minimax/minimax-m2.5:high__json_schema PASS 90% 129 in → 70 out (70 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
minimax/minimax-m2.5:medium__json_schema PASS 90% 129 in → 161 out (166 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 90% 116 in → 86 out (85 reasoning)
Update app.py content
openai/gpt-5.4-mini:high PASS 90% 96 in → 82 out (72 reasoning)
Update app.py content
openai/gpt-5.4-nano:none__json_schema PASS 90% 134 in → 17 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-5.4-nano:xhigh__json_schema PASS 90% 134 in → 134 out (115 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-5.4:none PASS 90% 96 in → 8 out (0 reasoning)
Update app.py content
openai/gpt-5.5:low__json_schema PASS 90% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.5:xhigh PASS 90% 96 in → 63 out (54 reasoning)
Update app content
openai/gpt-5.6-luna:high__json_schema PASS 90% 134 in → 45 out (27 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-luna:low__json_schema PASS 90% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-luna:medium PASS 90% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-5.6-luna:xhigh PASS 90% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-5.6-sol:high PASS 90% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-5.6-sol:low__json_schema PASS 90% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-terra:low PASS 90% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-5.6-terra:low__json_schema PASS 90% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-terra:max__json_schema PASS 90% 134 in → 66 out (48 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-terra:none__json_schema PASS 90% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-terra:xhigh__json_schema PASS 90% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-oss-120b:low__json_schema PASS 90% 157 in → 50 out (30 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
openai/gpt-oss-20b:medium__json_schema PASS 90% 155 in → 101 out (88 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
poolside/laguna-m.1:high__json_schema PASS 90% 113 in → 397 out (381 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
poolside/laguna-xs-2.1:medium__json_schema PASS 90% 113 in → 732 out (720 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
poolside/laguna-xs.2:high__json_schema PASS 90% 150 in → 165 out (153 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
poolside/laguna-xs.2:medium__json_schema PASS 90% 150 in → 150 out (138 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
qwen/qwen3.6-27b:high__json_schema PASS 90% 109 in → 1,023 out (979 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update content in app.py" }
qwen/qwen3.6-27b:low PASS 90% 109 in → 1,305 out (1,192 reasoning)
Update app.py content
qwen/qwen3.6-27b:low__json_schema PASS 90% 109 in → 1,706 out (1,523 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
qwen/qwen3.6-flash:none PASS 90% 111 in → 4 out (0 reasoning)
Update app.py content
qwen/qwen3.7-flash:none PASS 90% 111 in → 4 out (0 reasoning)
Update app.py content
qwen/qwen3.7-max:low PASS 90% 109 in → 353 out (343 reasoning)
Update content in app.py
tencent/hy3:medium PASS 90% 103 in → 360 out (323 reasoning)
Update app.py content
tencent/hy3:none PASS 90% 106 in → 5 out (0 reasoning)
Update app.py content
thinkingmachines/inkling:none__json_schema PASS 90% 103 in → 16 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update content in app.py" }
x-ai/grok-4.3:high__json_schema PASS 90% 334 in → 530 out (521 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
x-ai/grok-4.3:none__json_schema PASS 90% 332 in → 11 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
z-ai/glm-4.7-flash:none PASS 90% 96 in → 3 out (0 reasoning)
Update app content
z-ai/glm-5.2:xhigh__json_schema PASS 90% 103 in → 445 out (384 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update content in app.py" }
anthropic/claude-fable-5:xhigh__json_schema PASS 89.7% 359 in → 43 out (13 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
anthropic/claude-opus-4.6:none PASS 89.7% 117 in → 9 out (0 reasoning)
Update content in app.py
anthropic/claude-opus-4.8:xhigh__json_schema PASS 89.7% 359 in → 19 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
deepseek/deepseek-v4-pro:none__json_schema PASS 89.7% 96 in → 15 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) { "commit_message": "Update app content" }
google/gemini-3-flash-preview:low PASS 89.7% 109 in → 186 out (181 reasoning)
Update app.py content
google/gemini-3-flash-preview:xhigh PASS 89.7% 109 in → 288 out (282 reasoning)
Update content in app.py
google/gemma-4-26b-a4b-it:medium PASS 89.7% 125 in → 545 out (533 reasoning)
Update content in app.py
moonshotai/kimi-k2.6:xhigh PASS 89.7% 98 in → 453 out (447 reasoning)
Update app.py content
openai/gpt-5.4:medium PASS 89.7% 96 in → 47 out (37 reasoning)
Update app.py content
openai/gpt-5.4:xhigh PASS 89.7% 96 in → 92 out (82 reasoning)
Update app.py content
openai/gpt-5.6-luna:max PASS 89.7% 96 in → 53 out (43 reasoning)
Update app.py content
x-ai/grok-4.3:max PASS 89.7% 274 in → 353 out (348 reasoning)
Update content in app.py
anthropic/claude-opus-4.6:low PASS 89.3% 117 in → 8 out (0 reasoning)
Update app.py content
anthropic/claude-opus-4.8:xhigh PASS 89.3% 138 in → 53 out (34 reasoning)
Update content in app.py
anthropic/claude-sonnet-4.6:max__json_schema PASS 89.3% 287 in → 65 out (43 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-sonnet-5:high PASS 89.3% 138 in → 8 out (0 reasoning)
Update app.py content
anthropic/claude-sonnet-5:none__json_schema PASS 89.3% 359 in → 18 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 89.3% 177 in → 103 out (91 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
deepseek/deepseek-v4-flash-0731:xhigh PASS 89.3% 96 in → 196 out (181 reasoning)
Update app.py content
deepseek/deepseek-v4-pro:high__json_schema PASS 89.3% 287 in → 79 out (69 reasoning)
Update content
JSON Schema Structured Output
(raw) {"commit_message": "Update content"}
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 89.3% 110 in → 121 out (108 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
minimax/minimax-m2.7:medium PASS 89.3% 129 in → 309 out (332 reasoning)
Update app.py content
minimax/minimax-m3:low__json_schema PASS 89.3% 254 in → 13 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
minimax/minimax-m3:medium__json_schema PASS 89.3% 215 in → 14 out (13 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update content in app.py" }
mistralai/devstral-2512 PASS 89.3% 103 in → 7 out
"Update app.py content"
mistralai/mistral-medium-3-5:low PASS 89.3% 115 in → 1,692 out (1,897 reasoning)
Update app.py
moonshotai/kimi-k2.7-code:medium PASS 89.3% 97 in → 181 out (175 reasoning)
Update app.py content
openai/gpt-5.4-mini:low PASS 89.3% 96 in → 23 out (14 reasoning)
Update app content
openai/gpt-5.4-nano:high__json_schema PASS 89.3% 134 in → 61 out (42 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-5.4:high PASS 89.3% 96 in → 54 out (44 reasoning)
Update app.py content
openai/gpt-5.4:medium__json_schema PASS 89.3% 134 in → 57 out (38 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-5.6-luna:high PASS 89.3% 96 in → 28 out (19 reasoning)
Update app content
openai/gpt-5.6-luna:low PASS 89.3% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-5.6-sol:max PASS 89.3% 96 in → 41 out (32 reasoning)
Update app content
openai/gpt-5.6-sol:xhigh PASS 89.3% 96 in → 7 out (0 reasoning)
Update app content
openai/gpt-5.6-terra:max PASS 89.3% 96 in → 33 out (24 reasoning)
Update app content
openai/gpt-oss-120b:medium__json_schema PASS 89.3% 157 in → 75 out (73 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
poolside/laguna-m.1:low__json_schema PASS 89.3% 113 in → 365 out (352 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
poolside/laguna-m.1:none__json_schema PASS 89.3% 113 in → 14 out (0 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
poolside/laguna-xs-2.1:high PASS 89.3% 113 in → 321 out (316 reasoning)
Update app.py content
qwen/qwen3.6-27b:high PASS 89.3% 109 in → 1,424 out (1 reasoning)
Update content in app.py
qwen/qwen3.6-flash:medium PASS 89.3% 109 in → 1,506 out (1,497 reasoning)
Update app.py content
qwen/qwen3.7-max:medium PASS 89.3% 109 in → 1,083 out (1,073 reasoning)
Update content in app.py
thinkingmachines/inkling:max PASS 89.3% 105 in → 289 out (279 reasoning)
Update app.py content
thinkingmachines/inkling:medium PASS 89.3% 105 in → 125 out (114 reasoning)
Update app.py content
x-ai/grok-4.3:low__json_schema PASS 89.3% 340 in → 430 out (420 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
z-ai/glm-4.7-flash:xhigh__json_schema PASS 89.3% 96 in → 400 out (384 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
z-ai/glm-5.2:medium PASS 89.3% 103 in → 317 out (295 reasoning)
Update app.py content
z-ai/glm-5.2:none PASS 89% 97 in → 6 out (0 reasoning)
Update content in app.py
anthropic/claude-fable-5:high PASS 88.7% 138 in → 38 out (20 reasoning)
``` Update content in app.py ```
anthropic/claude-opus-4.7:high PASS 88.7% 143 in → 11 out (0 reasoning)
Update content in app.py
anthropic/claude-opus-4.8:low__json_schema PASS 88.7% 359 in → 19 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-sonnet-5:high__json_schema PASS 88.7% 359 in → 18 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
minimax/minimax-m2.7:medium__json_schema PASS 88.7% 265 in → 192 out (180 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
openai/gpt-5.6-luna:max__json_schema PASS 88.7% 134 in → 51 out (33 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-oss-120b:high PASS 88.7% 155 in → 125 out (130 reasoning)
Update app.py content
anthropic/claude-haiku-4.5:high__json_schema PASS 88.3% 316 in → 325 out (317 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message": "Update app content"}
anthropic/claude-opus-4.6:low__json_schema PASS 88.3% 287 in → 15 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-opus-4.6:none__json_schema PASS 88.3% 287 in → 15 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-opus-4.8:low PASS 88.3% 138 in → 9 out (0 reasoning)
Update content in app.py
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 88.3% 287 in → 14 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
deepseek/deepseek-v4-flash:none PASS 88.3% 96 in → 5 out (0 reasoning)
Update content in app.py
deepseek/deepseek-v4-pro:low PASS 88.3% 96 in → 177 out (181 reasoning)
Update app.py content
google/gemini-3.1-pro-preview:high PASS 88.3% 109 in → 367 out (362 reasoning)
Update app.py content
google/gemini-3.5-flash-lite:low PASS 88.3% 109 in → 6 out (0 reasoning)
Update content in app.py
google/gemini-3.5-flash-lite:max__json_schema PASS 88.3% 110 in → 408 out (390 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update content in app.py" }
google/gemini-3.5-flash:high__json_schema PASS 88.3% 172 in → 544 out (532 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
google/gemini-3.5-flash:low PASS 88.3% 109 in → 289 out (284 reasoning)
Update app.py content
liquid/lfm-2-24b-a2b PASS 88.3% 103 in → 4 out
Update app.py
openai/gpt-5.4-mini:high__json_schema PASS 88.3% 134 in → 65 out (47 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.4-nano:xhigh PASS 88.3% 96 in → 64 out (54 reasoning)
Update app.py content
openai/gpt-5.6-sol:high__json_schema PASS 88.3% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-sol:none__json_schema PASS 88.3% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-oss-20b:low PASS 88.3% 157 in → 45 out (32 reasoning)
Update app.py content
openai/gpt-oss-20b:medium PASS 88.3% 155 in → 114 out (109 reasoning)
Update app.py content
poolside/laguna-m.1:medium__json_schema PASS 88.3% 113 in → 521 out (504 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
qwen/qwen3.6-35b-a3b:low PASS 88.3% 109 in → 1,336 out (1,327 reasoning)
Update app.py content
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 88.3% 109 in → 1,238 out (1,134 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
qwen/qwen3.7-max:high PASS 88.3% 109 in → 713 out (703 reasoning)
Update content in app.py
thinkingmachines/inkling-small:low PASS 88.3% 105 in → 141 out (124 reasoning)
Update app.py content
thinkingmachines/inkling-small:medium PASS 88.3% 105 in → 136 out (125 reasoning)
Update content in app.py
thinkingmachines/inkling-small:none PASS 88.3% 103 in → 6 out (0 reasoning)
Update app content
thinkingmachines/inkling:high PASS 88.3% 105 in → 173 out (163 reasoning)
Update app.py content
x-ai/grok-4.3:xhigh PASS 88.3% 274 in → 425 out (421 reasoning)
Update app.py content
x-ai/grok-4.5:medium PASS 88.3% 302 in → 224 out (219 reasoning)
Update content in app.py
anthropic/claude-haiku-4.5:medium__json_schema PASS 88% 316 in → 295 out (232 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 88% 116 in → 177 out (123 reasoning)
Update content in app.py
anthropic/claude-fable-5:low PASS 87.7% 138 in → 9 out (0 reasoning)
Update content in app.py
anthropic/claude-opus-4.6:max__json_schema PASS 87.7% 287 in → 53 out (26 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-opus-4.7:medium PASS 87.7% 143 in → 11 out (0 reasoning)
Update content in app.py
anthropic/claude-opus-4.8:max PASS 87.7% 138 in → 234 out (69 reasoning)
Update content in app.py
anthropic/claude-sonnet-5:low__json_schema PASS 87.7% 359 in → 19 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-sonnet-5:medium__json_schema PASS 87.7% 359 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
arcee-ai/trinity-mini:high PASS 87.7% 98 in → 341 out (384 reasoning)
Update app.py
deepseek/deepseek-v4-flash-0731:max PASS 87.7% 175 in → 89 out (84 reasoning)
Update content in app.py
deepseek/deepseek-v4-pro:high PASS 87.7% 96 in → 99 out (94 reasoning)
Update app content
google/gemini-3.1-pro-preview:medium__json_schema PASS 87.7% 172 in → 321 out (308 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 87.7% 172 in → 74 out (62 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
google/gemini-3.6-flash:max__json_schema PASS 87.7% 172 in → 233 out (221 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
google/gemma-4-31b-it:high PASS 87.7% 125 in → 202 out (172 reasoning)
Update app.py
minimax/minimax-m3:medium PASS 87.7% 267 in → 129 out (134 reasoning)
Update app.py content
mistralai/mistral-medium-3-5:high PASS 87.7% 115 in → 369 out (421 reasoning)
Update app.py
nvidia/nemotron-3-nano-30b-a3b:low PASS 87.7% 116 in → 257 out (237 reasoning)
Update app.py content
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 87.7% 116 in → 229 out (182 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update content in app.py" }
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 87.7% 116 in → 98 out (82 reasoning)
Update content
JSON Schema Structured Output
(raw) { "commit_message": "Update content" }
openai/gpt-5.6-terra:high PASS 87.7% 96 in → 7 out (0 reasoning)
Update app content
poolside/laguna-xs.2:low__json_schema PASS 87.7% 150 in → 122 out (111 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
poolside/laguna-xs.2:xhigh__json_schema PASS 87.7% 150 in → 121 out (109 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
qwen/qwen3.6-35b-a3b:medium PASS 87.7% 109 in → 956 out (887 reasoning)
Update app.py content
qwen/qwen3.7-plus:none PASS 87.7% 111 in → 6 out (0 reasoning)
Update original content in app.py
thinkingmachines/inkling:none PASS 87.7% 103 in → 8 out (0 reasoning)
Update content in app.py
x-ai/grok-4.3:high PASS 87.7% 274 in → 480 out (476 reasoning)
Update app.py content
x-ai/grok-4.3:xhigh__json_schema PASS 87.7% 334 in → 414 out (404 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
google/gemma-4-26b-a4b-it:low PASS 87.3% 124 in → 386 out (345 reasoning)
Update app.py
openai/gpt-oss-20b:xhigh__json_schema PASS 87.3% 144 in → 255 out (226 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
google/gemini-3.1-flash-lite-preview:medium PASS 87% 109 in → 160 out (154 reasoning)
Update content in app.py
google/gemini-3.6-flash:high__json_schema PASS 87% 110 in → 311 out (299 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
inclusionai/ling-2.6-flash PASS 87% 120 in → 6 out
Update content in app.py
z-ai/glm-5.2:xhigh PASS 87% 103 in → 404 out (399 reasoning)
Update app.py content
anthropic/claude-opus-4.7:high__json_schema PASS 86.7% 364 in → 20 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
anthropic/claude-opus-4.8:medium__json_schema PASS 86.7% 359 in → 19 out (0 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
arcee-ai/trinity-large-thinking:low PASS 86.7% 98 in → 114 out (107 reasoning)
Update app.py content
deepseek/deepseek-v4-flash:xhigh PASS 86.7% 175 in → 529 out (519 reasoning)
Update content in app.py
google/gemini-3.6-flash:low PASS 86.7% 110 in → 151 out (146 reasoning)
Update app.py content
google/gemma-4-26b-a4b-it:xhigh PASS 86.7% 125 in → 311 out (261 reasoning)
Update app.py
moonshotai/kimi-k2.6:medium PASS 86.7% 98 in → 303 out (318 reasoning)
Update app.py content
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 86.7% 116 in → 99 out (94 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
nvidia/nemotron-3-super-120b-a12b:low PASS 86.7% 116 in → 185 out (159 reasoning)
Update content in app.py
openai/gpt-5.4-mini:none PASS 86.7% 96 in → 8 out (0 reasoning)
Update app.py content
openai/gpt-5.5:high__json_schema PASS 86.7% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.5:none PASS 86.7% 96 in → 7 out (0 reasoning)
Update app content
poolside/laguna-xs-2.1:xhigh PASS 86.7% 113 in → 392 out (388 reasoning)
Update app.py
qwen/qwen3.6-35b-a3b:low__json_schema PASS 86.7% 109 in → 466 out (424 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
qwen/qwen3.7-flash:low PASS 86.7% 109 in → 1,103 out (1,094 reasoning)
Update app.py content
x-ai/grok-4.3:medium PASS 86.7% 280 in → 471 out (466 reasoning)
Update content in app.py
anthropic/claude-opus-4.7:low PASS 86% 143 in → 11 out (0 reasoning)
Update content in app.py
google/gemini-3.1-flash-lite-preview:xhigh PASS 86% 109 in → 285 out (279 reasoning)
Update content in app.py
google/gemini-3.1-pro-preview:low__json_schema PASS 86% 172 in → 290 out (277 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
minimax/minimax-m3:xhigh__json_schema PASS 86% 215 in → 11 out (10 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
mistralai/mistral-medium-3-5:high__json_schema PASS 86% 103 in → 791 out (860 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message": "Update app content"}
moonshotai/kimi-k2.6:low PASS 86% 98 in → 374 out (399 reasoning)
Update app.py content
openai/gpt-5.4-nano:low PASS 86% 96 in → 10 out (0 reasoning)
Update app content in app.py
poolside/laguna-xs-2.1:low__json_schema PASS 86% 113 in → 842 out (831 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
qwen/qwen3.6-35b-a3b:none PASS 86% 111 in → 5 out (0 reasoning)
Update app.py content
thinkingmachines/inkling-small:high__json_schema PASS 86% 105 in → 136 out (121 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
x-ai/grok-4.3:medium__json_schema PASS 86% 340 in → 462 out (448 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update content in app.py" }
thinkingmachines/inkling-small:low__json_schema PASS 85.7% 105 in → 62 out (47 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
google/gemini-3.5-flash-lite:xhigh PASS 85.3% 110 in → 364 out (360 reasoning)
Update app.py
z-ai/glm-5.2:high__json_schema PASS 85.3% 103 in → 265 out (250 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update content in app.py" }
google/gemini-3-flash-preview:medium__json_schema PASS 85% 172 in → 321 out (311 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py"}
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 85% 172 in → 266 out (254 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
google/gemini-3.6-flash:medium__json_schema PASS 85% 110 in → 197 out (185 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
google/gemini-3.6-flash:xhigh__json_schema PASS 85% 110 in → 376 out (364 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
google/gemma-4-31b-it:xhigh PASS 85% 125 in → 131 out (112 reasoning)
Update app.py
ibm-granite/granite-4.1-8b PASS 85% 110 in → 6 out
Update content in app.py
mistralai/mistral-medium-3-5:xhigh PASS 85% 115 in → 373 out (407 reasoning)
Update app.py
moonshotai/kimi-k2.7-code:high PASS 85% 98 in → 164 out (158 reasoning)
Update app.py content
nvidia/nemotron-3-super-120b-a12b:high PASS 85% 116 in → 91 out (92 reasoning)
Update app.py content
openai/gpt-5.6-luna:none__json_schema PASS 85% 134 in → 17 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-oss-20b:low__json_schema PASS 85% 144 in → 33 out (16 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
poolside/laguna-xs-2.1:none PASS 85% 113 in → 6 out (0 reasoning)
Update app.py with new content
poolside/laguna-xs.2:xhigh PASS 85% 150 in → 125 out (118 reasoning)
Update app.py content
tencent/hy3:none__json_schema PASS 85% 106 in → 13 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
arcee-ai/trinity-mini:low PASS 84.7% 98 in → 277 out (292 reasoning)
Update app.py content
moonshotai/kimi-k2.7-code:low PASS 84.7% 98 in → 145 out (155 reasoning)
Update app.py
anthropic/claude-fable-5:max__json_schema PASS 84.3% 359 in → 113 out (49 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
anthropic/claude-opus-4.6:max PASS 84.3% 117 in → 47 out (26 reasoning)
Update content in app.py
google/gemini-3.1-pro-preview:high__json_schema PASS 84.3% 172 in → 512 out (500 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 84.3% 116 in → 20 out (0 reasoning)
Update app.py with corrected content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py with corrected content" }
openai/gpt-5.4-mini:none__json_schema PASS 84.3% 134 in → 16 out (0 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
openai/gpt-5.6-luna:xhigh__json_schema PASS 84.3% 134 in → 17 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
qwen/qwen3.7-flash:high PASS 84.3% 109 in → 1,481 out (1,473 reasoning)
Update app.py
tencent/hy3:high__json_schema PASS 84.3% 103 in → 719 out (734 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
anthropic/claude-sonnet-5:none PASS 84% 138 in → 8 out (0 reasoning)
Update app.py content
anthropic/claude-sonnet-4.6:max PASS 83.3% 117 in → 89 out (71 reasoning)
`Update app.py content`
arcee-ai/trinity-mini:xhigh PASS 83.3% 98 in → 241 out (261 reasoning)
Update app.py content
moonshotai/kimi-k2.6:high PASS 83.3% 98 in → 409 out (403 reasoning)
Update app.py content
openai/gpt-5.4-nano:low__json_schema PASS 83.3% 134 in → 17 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
qwen/qwen3.6-27b:medium PASS 83.3% 109 in → 1,230 out (1,125 reasoning)
Update app.py
qwen/qwen3.6-27b:medium__json_schema PASS 83.3% 109 in → 1,461 out (1,314 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
qwen/qwen3.7-max:none PASS 83.3% 111 in → 4 out (0 reasoning)
Update app.py content
z-ai/glm-4.7-flash:high PASS 83.3% 96 in → 444 out (423 reasoning)
Update app.py
anthropic/claude-sonnet-4.6:high PASS 82.7% 117 in → 45 out (26 reasoning)
Update app.py content
google/gemini-3.5-flash:high PASS 82.7% 109 in → 539 out (534 reasoning)
Update app.py content
google/gemma-4-31b-it:high__json_schema PASS 82.7% 125 in → 167 out (130 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
minimax/minimax-m2.5:medium PASS 82.7% 129 in → 109 out (124 reasoning)
Update app.py content
moonshotai/kimi-k3:medium__json_schema PASS 82.7% 275 in → 67 out (41 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update content in app.py"}
openai/gpt-5.4:xhigh__json_schema PASS 82.7% 134 in → 95 out (76 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
x-ai/grok-4.3:max__json_schema PASS 82.7% 334 in → 447 out (437 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
z-ai/glm-4.7-flash:medium__json_schema PASS 82.7% 96 in → 428 out (419 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
z-ai/glm-4.7-flash:none__json_schema PASS 82.7% 96 in → 10 out (0 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
poolside/laguna-xs.2:high PASS 82.3% 150 in → 266 out (260 reasoning)
Update app.py
google/gemma-4-31b-it:low__json_schema PASS 82% 125 in → 193 out (148 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
anthropic/claude-haiku-4.5:low__json_schema PASS 81.7% 316 in → 526 out (506 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
deepseek/deepseek-v4-flash:medium__json_schema PASS 81.7% 96 in → 232 out (212 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py"}
minimax/minimax-m2.7:xhigh PASS 81.7% 129 in → 138 out (147 reasoning)
Update app.py content
mistralai/mistral-medium-3-5:medium PASS 81.7% 115 in → 2,135 out (2,360 reasoning)
Update app.py
openai/gpt-5.4-mini:xhigh__json_schema PASS 81.7% 134 in → 135 out (116 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-5.6-luna:medium__json_schema PASS 81.7% 134 in → 17 out (0 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py content"}
openai/gpt-5.6-sol:max__json_schema PASS 81.7% 134 in → 55 out (37 reasoning)
Update app content
JSON Schema Structured Output
(raw) {"commit_message":"Update app content"}
qwen/qwen3.6-flash:xhigh PASS 81.7% 109 in → 1,107 out (1,099 reasoning)
Update app.py
qwen/qwen3.7-flash:medium PASS 81.7% 109 in → 1,200 out (1,191 reasoning)
Update app.py content
minimax/minimax-m2.5:xhigh PASS 81.3% 129 in → 130 out (124 reasoning)
Update app.py content
anthropic/claude-fable-5:xhigh PASS 81% 138 in → 38 out (13 reasoning)
``` Update content in app.py ```
arcee-ai/trinity-mini:high__json_schema PASS 81% 98 in → 250 out (281 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
google/gemma-4-26b-a4b-it:low__json_schema PASS 81% 125 in → 506 out (450 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
mistralai/devstral-2512__json_schema PASS 81% 103 in → 11 out
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
deepseek/deepseek-v4-flash-0731:low PASS 80% 96 in → 154 out (148 reasoning)
Update app.py content
deepseek/deepseek-v4-flash:high__json_schema PASS 80% 96 in → 223 out (210 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py content"}
google/gemini-3.5-flash:medium PASS 80% 109 in → 338 out (333 reasoning)
Update app.py content
openai/gpt-5.4:low PASS 80% 96 in → 30 out (20 reasoning)
Update app.py content
poolside/laguna-m.1:xhigh PASS 80% 113 in → 212 out (205 reasoning)
Update app.py content
poolside/laguna-xs-2.1:none__json_schema PASS 80% 113 in → 13 out (0 reasoning)
Update app.py with new content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py with new content"}
tencent/hy3:high PASS 80% 103 in → 434 out (429 reasoning)
Update app.py
thinkingmachines/inkling:xhigh PASS 80% 105 in → 188 out (180 reasoning)
Update app.py content
anthropic/claude-sonnet-5:medium PASS 79.3% 138 in → 8 out (0 reasoning)
Update app.py content
poolside/laguna-xs-2.1:low PASS 79.3% 113 in → 208 out (204 reasoning)
Update app.py
x-ai/grok-4.5:max__json_schema PASS 79.3% 374 in → 218 out (206 reasoning)
Update content in app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update content in app.py"}
anthropic/claude-sonnet-4.6:xhigh PASS 78.3% 117 in → 45 out (22 reasoning)
`Update app.py content`
google/gemma-4-31b-it:medium__json_schema PASS 78.3% 125 in → 162 out (124 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
anthropic/claude-haiku-4.5:none__json_schema PASS 77.7% 286 in → 17 out (0 reasoning)
Update app.py with new content
JSON Schema Structured Output
(raw) {"commit_message": "Update app.py with new content"}
qwen/qwen3.6-27b:none PASS 76.7% 111 in → 4 out (0 reasoning)
Update app.py
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 74.3% 96 in → 170 out (163 reasoning)
Update app.py
JSON Schema Structured Output
(raw) {"commit_message":"Update app.py" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 71% 116 in → 6 out (0 reasoning)
Updated content in app.py
google/gemma-4-26b-a4b-it:high__json_schema PASS 70% 124 in → 606 out (561 reasoning)
Update app.py
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py" }
arcee-ai/trinity-mini:medium PASS 68.3% 98 in → 353 out (393 reasoning)
Update app.pycontent
moonshotai/kimi-k2.7-code:medium__json_schema PASS 63.3% 98 in → 150 out (135 reasoning)
Update app.py content
JSON Schema Structured Output
(raw) { "commit_message": "Update app.py content" }
deepseek/deepseek-v4-pro:xhigh PASS 61.7% 175 in → 994 out (1,098 reasoning)
Updateapp.py content
nvidia/nemotron-3-nano-30b-a3b:high PASS 57.3% 116 in → 10,493 out (10,839 reasoning)
Update
anthropic/claude-haiku-4.5:high FAIL 46.7% 146 in → 386 out (358 reasoning)
# Update app.py *11 characters*
poolside/laguna-m.1:none FAIL 43.3% 113 in → 107 out (0 reasoning)
Update app.py content This commit message: 1. Stays within the 50-character limit 2. Clearly states the action taken ("Update") 3. Identifies the file modified ("app.py") 4. Is concise and descriptive Alternative options that would also work: - "Modify app.py content" - "Replace original content in app.py" - "Update content in app.py" The first suggestion is preferred as it's the most straightforward and commonly used format for such changes.
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:low__json_schema FAIL 0%
Invalid JSON. Output: Update app.py content
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:none__json_schema FAIL 0%
Invalid JSON. Output: Update app.py content
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Updateapp.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Updateapp.py content
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: Update app.py
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k3:high FAIL 0% 175 in → 133 out (114 reasoning)
Update app.py content
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:high__json_schema FAIL 0% 175 in → 156 out (115 reasoning)
**Update app.py content** (21 characters — uses the "Update" action verb since existing content was replaced with new content.)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low FAIL 0% 175 in → 72 out (53 reasoning)
Update app.py content
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low__json_schema FAIL 0% 175 in → 123 out (87 reasoning)
**Update content in app.py** (21 characters — describes the action of updating the file's content)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max FAIL 0% 175 in → 162 out (113 reasoning)
**Commit message:** ``` Update content in app.py ``` (24 characters) — Uses the "Update" action since existing content in `app.py` was modified.
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max__json_schema FAIL 0% 175 in → 75 out (56 reasoning)
Update app.py content
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium FAIL 0% 175 in → 129 out (69 reasoning)
**Commit message:** ``` Update app.py content ``` This message starts with the action verb "Update" (matching the content change shown in the diff) and stays well within the 50-character limit at 21 characters.
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh FAIL 0% 175 in → 107 out (87 reasoning)
Update content in app.py
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh__json_schema FAIL 0% 175 in → 123 out (72 reasoning)
**Commit message:** ``` Update app.py content ``` (20 characters — uses the imperative action "Update" and identifies the changed file, fitting within the 50-character limit.)
Failure: Scoring error: llm_judge requires a judge client
nvidia/nemotron-3-super-120b-a12b:medium FAIL 0% 116 in → 70 out (75 reasoning)
Update original content
Failure: Scoring error: llm_judge requires a judge client
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8437dc87-4ba0-9158-875e-5ba96fca59d1","request_id":"8437dc87-4ba0-9158-875e-5ba96fca59d1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3aafa53d-26e3-996d-b2bb-29592378beea","request_id":"3aafa53d-26e3-996d-b2bb-29592378beea"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cc0d6d3f-7435-9005-9369-9939b8b142b9","request_id":"cc0d6d3f-7435-9005-9369-9939b8b142b9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5d9d7193-47fb-91e1-8c42-deaed587634c","request_id":"5d9d7193-47fb-91e1-8c42-deaed587634c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-abf84079-e02c-9690-8f68-0b5d40083e2c","request_id":"abf84079-e02c-9690-8f68-0b5d40083e2c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-601c2459-355f-95bc-83e6-fbe25c246965"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9f7ccdb7-469a-9ccd-bf78-fef39627dd33"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max FAIL 0% 109 in → 1,275 out (1,266 reasoning)
Update app.py content
Failure: Scoring error: llm_judge requires a judge client
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a005d50e-4040-9fcb-8148-fbb1ed93ed95"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-264ed82c-c1e9-93c6-ab8c-7c638298a55c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-dd6da4d5-d7d1-9f65-bfc9-0d010f0152c5"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-fdae606c-35d2-92a9-9192-7150d57da3c0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7840c017-54d2-9003-a6c0-f7ddb79557e4","request_id":"7840c017-54d2-9003-a6c0-f7ddb79557e4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-924a0395-9d2f-94ab-b661-4001c5ce7bce","request_id":"924a0395-9d2f-94ab-b661-4001c5ce7bce"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8b7e65ec-e547-95f2-99ab-a9a582d79b60","request_id":"8b7e65ec-e547-95f2-99ab-a9a582d79b60"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a4d87530-491b-9318-a5f1-7523de60b053","request_id":"a4d87530-491b-9318-a5f1-7523de60b053"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8d83a0bd-9909-95d5-bc23-4eee95a8c23f","request_id":"8d83a0bd-9909-95d5-bc23-4eee95a8c23f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-feb9d93f-ee9b-9595-8f7e-530f2dcc2cb4","request_id":"feb9d93f-ee9b-9595-8f7e-530f2dcc2cb4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2eee3282-e81e-92aa-9f44-9f12956f03e6","request_id":"2eee3282-e81e-92aa-9f44-9f12956f03e6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8826aa24-f819-981a-b597-60eee4f71f90","request_id":"8826aa24-f819-981a-b597-60eee4f71f90"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-43ed8a55-6cee-945e-84dc-5210253f7c71","request_id":"43ed8a55-6cee-945e-84dc-5210253f7c71"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3cb2e15c-199d-96e8-bee0-26ebd049aeca","request_id":"3cb2e15c-199d-96e8-bee0-26ebd049aeca"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: Update app.py content{ "commit_message": "Update app.py content" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: Update app.py content
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: Update app.py content
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:max__json_schema FAIL 0%
Invalid JSON. Output: Update app.py content
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: Update app.py content
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Update app.py content
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:medium__json_schema FAIL 0%
Invalid JSON. Output: Update app.py content
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)