f008 — commit_messages

← commit_messages
Executable permission change
Tests ability to generate a commit message for an executable permission change. Evaluates recognizing metadata changes vs content changes.
medium commit-message permissions executable metadata
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo '#!/bin/bash' > build.sh
  5. 05 echo 'echo building...' >> build.sh
  6. 06 git add build.sh
  7. 07 git commit -m 'Add build script'
  8. 08 chmod +x build.sh
  9. 09 git add build.sh
Prompt
Generate a concise commit message (max 50 characters) describing the changes in this diff. Focus on the action taken (e.g., 'Add', 'Fix', 'Update').
Expected
Make build.sh executable
Fixture Metrics

Scoped model quality, cost, API time, and token usage for commit_messages/f008.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-haiku-4.5:medium__json_schema PASS 97.5% 278 in → 290 out (220 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
moonshotai/kimi-k2.6:xhigh PASS 97.5% 70 in → 384 out (379 reasoning)
Make build.sh executable
qwen/qwen3.6-27b:xhigh PASS 96% 81 in → 1,021 out (885 reasoning)
Make build.sh executable
google/gemini-3.1-pro-preview:xhigh PASS 95.7% 76 in → 519 out (514 reasoning)
Make build.sh executable
moonshotai/kimi-k2.7-code:high PASS 95.7% 70 in → 112 out (106 reasoning)
Make build.sh executable
anthropic/claude-opus-4.7:low__json_schema PASS 95.3% 325 in → 22 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
deepseek/deepseek-v4-flash:high PASS 95% 66 in → 200 out (195 reasoning)
Make build.sh executable
google/gemini-3.1-pro-preview:high__json_schema PASS 95% 139 in → 444 out (432 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3.1-pro-preview:medium PASS 95% 76 in → 452 out (445 reasoning)
Add executable permission to build.sh
google/gemini-3.5-flash-lite:low PASS 95% 77 in → 5 out (0 reasoning)
Make build.sh executable
minimax/minimax-m2.5:xhigh__json_schema PASS 95% 100 in → 119 out (123 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 95% 87 in → 170 out (157 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
qwen/qwen3.6-flash:low PASS 95% 81 in → 1,175 out (1,166 reasoning)
Make build.sh executable
thinkingmachines/inkling-small:max PASS 95% 77 in → 125 out (125 reasoning)
Make build.sh executable
x-ai/grok-4.5:max PASS 95% 271 in → 240 out (236 reasoning)
Make build.sh executable
moonshotai/kimi-k2.6:low PASS 94.3% 70 in → 354 out (349 reasoning)
Make build.sh executable
openai/gpt-5.4-mini:low PASS 94.3% 68 in → 39 out (29 reasoning)
Make build.sh executable
qwen/qwen3.6-flash:medium PASS 94.3% 81 in → 995 out (986 reasoning)
Make build.sh executable
tencent/hy3:high PASS 94.3% 74 in → 587 out (581 reasoning)
Make build.sh executable
x-ai/grok-4.3:high__json_schema PASS 94.3% 307 in → 299 out (290 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
z-ai/glm-4.7-flash:xhigh PASS 94.3% 69 in → 387 out (350 reasoning)
Make build.sh executable
anthropic/claude-opus-4.8:medium__json_schema PASS 93.7% 320 in → 20 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
moonshotai/kimi-k2.6:high__json_schema PASS 93.7% 69 in → 230 out (231 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
openai/gpt-5.4-mini:low__json_schema PASS 93.7% 106 in → 50 out (31 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.4-mini:xhigh__json_schema PASS 93.7% 106 in → 167 out (148 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.5:medium PASS 93.7% 68 in → 42 out (32 reasoning)
Make build.sh executable
anthropic/claude-fable-5:xhigh PASS 93.3% 99 in → 43 out (21 reasoning)
``` Make build.sh executable ```
anthropic/claude-fable-5:xhigh__json_schema PASS 93.3% 320 in → 64 out (36 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-haiku-4.5:none__json_schema PASS 93.3% 248 in → 15 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-opus-4.7:high PASS 93.3% 104 in → 12 out (0 reasoning)
Make build.sh executable
anthropic/claude-opus-4.7:max__json_schema PASS 93.3% 325 in → 22 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.7:none__json_schema PASS 93.3% 325 in → 22 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.7:xhigh__json_schema PASS 93.3% 325 in → 22 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.8:high PASS 93.3% 99 in → 46 out (17 reasoning)
Make build.sh executable
anthropic/claude-opus-4.8:none__json_schema PASS 93.3% 320 in → 19 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-sonnet-4.6:high PASS 93.3% 79 in → 56 out (30 reasoning)
`Make build.sh executable`
anthropic/claude-sonnet-5:high__json_schema PASS 93.3% 320 in → 20 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-sonnet-5:medium__json_schema PASS 93.3% 320 in → 20 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 93.3% 72 in → 182 out (169 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 93.3% 66 in → 13 out (0 reasoning)
Update build.sh permissions
JSON Schema Structured Output
(raw) { "commit_message": "Update build.sh permissions" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 93.3% 66 in → 275 out (269 reasoning)
Make build.sh executable
deepseek/deepseek-v4-pro:high__json_schema PASS 93.3% 257 in → 189 out (177 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3-flash-preview:xhigh__json_schema PASS 93.3% 139 in → 308 out (297 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemini-3.5-flash-lite:high PASS 93.3% 76 in → 444 out (439 reasoning)
Make build.sh executable
google/gemini-3.5-flash-lite:xhigh PASS 93.3% 76 in → 426 out (421 reasoning)
Make build.sh executable
google/gemini-3.5-flash:high PASS 93.3% 76 in → 358 out (353 reasoning)
Make build.sh executable
google/gemini-3.5-flash:medium__json_schema PASS 93.3% 139 in → 306 out (294 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3.6-flash:medium__json_schema PASS 93.3% 77 in → 291 out (280 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemini-3.6-flash:xhigh PASS 93.3% 76 in → 360 out (355 reasoning)
Make build.sh executable
google/gemma-4-26b-a4b-it:medium__json_schema PASS 93.3% 92 in → 712 out (598 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
google/gemma-4-26b-a4b-it:none__json_schema PASS 93.3% 89 in → 15 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
google/gemma-4-31b-it:medium__json_schema PASS 93.3% 92 in → 296 out (224 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
minimax/minimax-m2.5:xhigh PASS 93.3% 100 in → 61 out (61 reasoning)
Add executable permission to build.sh
minimax/minimax-m2.7:high PASS 93.3% 99 in → 186 out (199 reasoning)
Make build.sh executable
minimax/minimax-m2.7:medium PASS 93.3% 102 in → 212 out (207 reasoning)
Make build.sh executable
nvidia/nemotron-3-nano-30b-a3b:low PASS 93.3% 87 in → 350 out (245 reasoning)
Make build.sh executable
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 93.3% 87 in → 11 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
openai/gpt-5.4-mini:high PASS 93.3% 68 in → 83 out (73 reasoning)
Make build.sh executable
openai/gpt-5.4-mini:xhigh PASS 93.3% 68 in → 99 out (89 reasoning)
Make build.sh executable
openai/gpt-5.4:medium__json_schema PASS 93.3% 106 in → 77 out (58 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.5:none PASS 93.3% 68 in → 8 out (0 reasoning)
Make build script executable
openai/gpt-5.6-luna:high__json_schema PASS 93.3% 106 in → 17 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.6-sol:low__json_schema PASS 93.3% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-5.6-terra:max__json_schema PASS 93.3% 106 in → 76 out (57 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.6-terra:xhigh__json_schema PASS 93.3% 106 in → 46 out (27 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-oss-120b:high PASS 93.3% 116 in → 230 out (243 reasoning)
Make build.sh executable
openai/gpt-oss-20b:low__json_schema PASS 93.3% 129 in → 38 out (19 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
poolside/laguna-m.1:high PASS 93.3% 85 in → 359 out (352 reasoning)
Make build.sh executable
poolside/laguna-m.1:high__json_schema PASS 93.3% 85 in → 233 out (217 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
poolside/laguna-m.1:low__json_schema PASS 93.3% 85 in → 556 out (544 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
poolside/laguna-xs-2.1:xhigh__json_schema PASS 93.3% 85 in → 419 out (407 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
poolside/laguna-xs.2:xhigh PASS 93.3% 122 in → 153 out (146 reasoning)
Make build.sh executable
qwen/qwen3.6-27b:high PASS 93.3% 81 in → 1,111 out (1,102 reasoning)
Make build.sh executable
qwen/qwen3.6-27b:medium PASS 93.3% 81 in → 974 out (910 reasoning)
Make build.sh executable
qwen/qwen3.6-35b-a3b:high__json_schema PASS 93.3% 81 in → 1,226 out (1,055 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
qwen/qwen3.6-35b-a3b:medium PASS 93.3% 81 in → 1,016 out (888 reasoning)
Make build.sh executable
qwen/qwen3.6-35b-a3b:xhigh PASS 93.3% 81 in → 1,139 out (1,049 reasoning)
Make build.sh executable
qwen/qwen3.7-max:medium PASS 93.3% 81 in → 622 out (613 reasoning)
Make build.sh executable
qwen/qwen3.7-max:xhigh PASS 93.3% 81 in → 356 out (347 reasoning)
Make build.sh executable
tencent/hy3:low__json_schema PASS 93.3% 74 in → 446 out (435 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
tencent/hy3:medium PASS 93.3% 74 in → 357 out (351 reasoning)
Make build.sh executable
thinkingmachines/inkling-small:none PASS 93.3% 75 in → 7 out (0 reasoning)
Make build.sh executable
thinkingmachines/inkling-small:xhigh PASS 93.3% 77 in → 64 out (57 reasoning)
Make build.sh executable
thinkingmachines/inkling-small:xhigh__json_schema PASS 93.3% 77 in → 95 out (78 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
thinkingmachines/inkling:high PASS 93.3% 77 in → 127 out (117 reasoning)
Make build.sh executable
x-ai/grok-4.3:medium__json_schema PASS 93.3% 313 in → 404 out (391 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
x-ai/grok-4.3:xhigh__json_schema PASS 93.3% 307 in → 349 out (340 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
x-ai/grok-4.5:high__json_schema PASS 93.3% 343 in → 180 out (169 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
x-ai/grok-4.5:xhigh__json_schema PASS 93.3% 343 in → 302 out (291 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
z-ai/glm-5.2:low PASS 93.3% 76 in → 344 out (338 reasoning)
Make build.sh executable
z-ai/glm-5.2:low__json_schema PASS 93.3% 255 in → 11 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
z-ai/glm-5.2:xhigh PASS 93.3% 76 in → 627 out (621 reasoning)
Make build.sh executable
anthropic/claude-sonnet-4.6:xhigh PASS 93% 79 in → 60 out (33 reasoning)
`Make build.sh executable`
minimax/minimax-m3:low__json_schema PASS 93% 225 in → 13 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 93% 87 in → 109 out (85 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
openai/gpt-5.6-luna:high PASS 93% 68 in → 8 out (0 reasoning)
Make build script executable
tencent/hy3:none PASS 93% 77 in → 11 out (0 reasoning)
Make build.sh executable (chmod +x)
x-ai/grok-4.5:high PASS 93% 271 in → 192 out (188 reasoning)
Make build.sh executable
anthropic/claude-fable-5:high PASS 92.7% 99 in → 25 out (17 reasoning)
Make build.sh executable
anthropic/claude-fable-5:medium PASS 92.7% 99 in → 20 out (8 reasoning)
Make build.sh executable
anthropic/claude-opus-4.6:medium__json_schema PASS 92.7% 249 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.7:medium PASS 92.7% 104 in → 12 out (0 reasoning)
Make build.sh executable
anthropic/claude-opus-4.7:xhigh PASS 92.7% 104 in → 12 out (0 reasoning)
Make build.sh executable
anthropic/claude-opus-4.8:low PASS 92.7% 99 in → 10 out (0 reasoning)
Make build.sh executable
anthropic/claude-opus-4.8:none PASS 92.7% 99 in → 10 out (0 reasoning)
Make build.sh executable
anthropic/claude-sonnet-4.6:none PASS 92.7% 79 in → 10 out (0 reasoning)
`Make build.sh executable`
anthropic/claude-sonnet-5:medium PASS 92.7% 99 in → 10 out (0 reasoning)
Make build.sh executable
arcee-ai/trinity-mini:high PASS 92.7% 71 in → 308 out (343 reasoning)
Make build.sh executable
deepseek/deepseek-v4-flash-0731:max PASS 92.7% 158 in → 195 out (161 reasoning)
Make build.sh executable
deepseek/deepseek-v4-flash:none__json_schema PASS 92.7% 162 in → 10 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
deepseek/deepseek-v4-pro:high PASS 92.7% 66 in → 105 out (99 reasoning)
Make build.sh executable
deepseek/deepseek-v4-pro:low PASS 92.7% 66 in → 214 out (215 reasoning)
Make build.sh executable
deepseek/deepseek-v4-pro:none PASS 92.7% 66 in → 4 out (0 reasoning)
Make build.sh executable
google/gemini-3-flash-preview:low PASS 92.7% 76 in → 221 out (216 reasoning)
Make build.sh executable
google/gemini-3.1-flash-lite-preview:high PASS 92.7% 77 in → 369 out (364 reasoning)
Make build.sh executable
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 92.7% 139 in → 379 out (367 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 92.7% 77 in → 160 out (143 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 92.7% 139 in → 411 out (394 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
google/gemini-3.5-flash:medium PASS 92.7% 76 in → 543 out (538 reasoning)
Make build.sh executable
google/gemini-3.6-flash:high PASS 92.7% 77 in → 332 out (327 reasoning)
Make build.sh executable
google/gemini-3.6-flash:low__json_schema PASS 92.7% 139 in → 11 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemma-4-31b-it:low__json_schema PASS 92.7% 91 in → 212 out (170 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
ibm-granite/granite-4.1-8b__json_schema PASS 92.7% 77 in → 14 out
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
minimax/minimax-m3:medium__json_schema PASS 92.7% 225 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
mistralai/mistral-medium-3-5:high PASS 92.7% 86 in → 486 out (461 reasoning)
Make build.sh executable
mistralai/mistral-medium-3-5:high__json_schema PASS 92.7% 74 in → 302 out (317 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
mistralai/mistral-medium-3-5:low__json_schema PASS 92.7% 74 in → 226 out (229 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
mistralai/mistral-medium-3-5:medium PASS 92.7% 86 in → 726 out (677 reasoning)
Make build.sh executable
mistralai/mistral-medium-3-5:none__json_schema PASS 92.7% 74 in → 11 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 92.7% 74 in → 291 out (261 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 92.7% 87 in → 137 out (119 reasoning)
Add executable permission to build.sh
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 92.7% 87 in → 307 out (188 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 92.7% 87 in → 666 out (447 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
openai/gpt-5.4-nano:high PASS 92.7% 68 in → 60 out (50 reasoning)
Make build.sh executable
openai/gpt-5.5:high__json_schema PASS 92.7% 106 in → 80 out (61 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.5:medium__json_schema PASS 92.7% 106 in → 63 out (44 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.6-luna:max PASS 92.7% 68 in → 56 out (46 reasoning)
Make build.sh executable
openai/gpt-5.6-sol:max PASS 92.7% 68 in → 42 out (32 reasoning)
Make build.sh executable
openai/gpt-5.6-terra:max PASS 92.7% 68 in → 51 out (41 reasoning)
Make build.sh executable
openai/gpt-oss-20b:high__json_schema PASS 92.7% 183 in → 345 out (350 reasoning)
Add execute permission to build.sh
JSON Schema Structured Output
(raw) {"commit_message":"Add execute permission to build.sh"}
openai/gpt-oss-20b:low PASS 92.7% 127 in → 41 out (30 reasoning)
Add execute permission to build.sh
poolside/laguna-m.1:xhigh__json_schema PASS 92.7% 85 in → 269 out (257 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
poolside/laguna-xs-2.1:low__json_schema PASS 92.7% 85 in → 244 out (232 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
poolside/laguna-xs.2:medium__json_schema PASS 92.7% 122 in → 277 out (266 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
poolside/laguna-xs.2:none PASS 92.7% 122 in → 6 out (0 reasoning)
Fix build.sh permissions
qwen/qwen3.6-35b-a3b:none__json_schema PASS 92.7% 83 in → 13 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 92.7% 81 in → 1,250 out (1,044 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
qwen/qwen3.6-flash:xhigh PASS 92.7% 81 in → 891 out (882 reasoning)
Make build.sh executable
qwen/qwen3.7-flash:medium PASS 92.7% 81 in → 1,199 out (1,190 reasoning)
Make build.sh executable
qwen/qwen3.7-flash:xhigh PASS 92.7% 81 in → 854 out (845 reasoning)
Make build.sh executable
tencent/hy3:xhigh PASS 92.7% 74 in → 510 out (504 reasoning)
Make build.sh executable
thinkingmachines/inkling-small:max__json_schema PASS 92.7% 77 in → 71 out (56 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
z-ai/glm-4.7-flash:medium PASS 92.7% 69 in → 373 out (349 reasoning)
Make build.sh executable
z-ai/glm-5.2:high PASS 92.7% 76 in → 116 out (110 reasoning)
Make build.sh executable
mistralai/devstral-2512 PASS 92.5% 74 in → 7 out
"Fix build.sh permissions"
anthropic/claude-fable-5:low__json_schema PASS 92.3% 320 in → 20 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-fable-5:medium__json_schema PASS 92.3% 320 in → 35 out (11 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-opus-4.6:max PASS 92.3% 79 in → 67 out (44 reasoning)
Make build.sh executable
arcee-ai/trinity-large-thinking:high PASS 92.3% 71 in → 136 out (131 reasoning)
Make build.sh executable
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 92.3% 139 in → 396 out (385 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemma-4-26b-a4b-it:xhigh PASS 92.3% 92 in → 233 out (186 reasoning)
Make build.sh executable
google/gemma-4-31b-it:high PASS 92.3% 92 in → 238 out (181 reasoning)
Make build.sh executable
minimax/minimax-m2.5:medium PASS 92.3% 100 in → 114 out (126 reasoning)
Make build.sh executable
minimax/minimax-m2.7:low__json_schema PASS 92.3% 236 in → 202 out (191 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.5:none__json_schema PASS 92.3% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-5.6-terra:none PASS 92.3% 68 in → 8 out (0 reasoning)
Make build script executable
poolside/laguna-xs-2.1:none__json_schema PASS 92.3% 85 in → 16 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
poolside/laguna-xs.2:high PASS 92.3% 122 in → 290 out (283 reasoning)
Make build.sh executable
qwen/qwen3.7-plus:low PASS 92.3% 81 in → 853 out (844 reasoning)
Make build.sh executable
thinkingmachines/inkling-small:high PASS 92.3% 77 in → 51 out (44 reasoning)
Make build.sh executable
anthropic/claude-opus-4.8:max PASS 92% 99 in → 230 out (53 reasoning)
Make build.sh executable
arcee-ai/trinity-mini:high__json_schema PASS 92% 71 in → 271 out (290 reasoning)
Update build.sh permissions
JSON Schema Structured Output
(raw) { "commit_message": "Update build.sh permissions" }
google/gemini-3-flash-preview:medium PASS 92% 76 in → 215 out (210 reasoning)
Make build.sh executable
google/gemini-3.1-pro-preview:low PASS 92% 76 in → 427 out (422 reasoning)
Make build.sh executable
google/gemini-3.5-flash:low PASS 92% 76 in → 208 out (203 reasoning)
Make build.sh executable
google/gemma-4-31b-it:high__json_schema PASS 92% 92 in → 265 out (188 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
minimax/minimax-m2.5:low PASS 92% 102 in → 117 out (112 reasoning)
Make build.sh executable
openai/gpt-5.4-nano:low__json_schema PASS 92% 106 in → 39 out (20 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
qwen/qwen3.6-27b:high__json_schema PASS 92% 81 in → 1,479 out (1,333 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
anthropic/claude-fable-5:max PASS 91.7% 99 in → 482 out (96 reasoning)
``` Make build.sh executable ``` This diff only changes the file mode from `100644` to `100755`, which adds execute permissions to the script—no content changes were made.
anthropic/claude-fable-5:max__json_schema PASS 91.7% 320 in → 163 out (58 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-opus-4.6:high__json_schema PASS 91.7% 249 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.6:low PASS 91.7% 79 in → 8 out (0 reasoning)
Make build.sh executable
anthropic/claude-opus-4.7:low PASS 91.7% 104 in → 12 out (0 reasoning)
Make build.sh executable
anthropic/claude-opus-4.7:max PASS 91.7% 104 in → 93 out (34 reasoning)
Make build.sh executable
anthropic/claude-opus-4.7:medium__json_schema PASS 91.7% 325 in → 22 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.8:max__json_schema PASS 91.7% 320 in → 55 out (19 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-opus-4.8:medium PASS 91.7% 99 in → 10 out (0 reasoning)
Make build.sh executable
anthropic/claude-sonnet-4.6:high__json_schema PASS 91.7% 249 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 91.7% 249 in → 79 out (53 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 91.7% 249 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-sonnet-5:high PASS 91.7% 99 in → 10 out (0 reasoning)
Make build.sh executable
anthropic/claude-sonnet-5:none PASS 91.7% 99 in → 10 out (0 reasoning)
Make build.sh executable
arcee-ai/trinity-large-thinking:xhigh PASS 91.7% 71 in → 123 out (116 reasoning)
Update build.sh permissions
arcee-ai/trinity-mini:xhigh__json_schema PASS 91.7% 71 in → 300 out (335 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
deepseek/deepseek-v4-flash-0731:high PASS 91.7% 145 in → 144 out (141 reasoning)
Make build.sh executable
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 91.7% 66 in → 127 out (124 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
deepseek/deepseek-v4-flash:low__json_schema PASS 91.7% 162 in → 11 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 91.7% 162 in → 11 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
deepseek/deepseek-v4-pro:low__json_schema PASS 91.7% 257 in → 60 out (48 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
deepseek/deepseek-v4-pro:none__json_schema PASS 91.7% 257 in → 11 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3-flash-preview:high__json_schema PASS 91.7% 139 in → 349 out (338 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemini-3-flash-preview:low__json_schema PASS 91.7% 139 in → 261 out (250 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemini-3-flash-preview:medium__json_schema PASS 91.7% 139 in → 305 out (294 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemini-3.1-flash-lite-preview:low PASS 91.7% 77 in → 131 out (126 reasoning)
Make build.sh executable
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 91.7% 139 in → 431 out (419 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3.1-pro-preview:low__json_schema PASS 91.7% 139 in → 138 out (121 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
google/gemini-3.1-pro-preview:medium__json_schema PASS 91.7% 139 in → 123 out (111 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3.5-flash-lite:low__json_schema PASS 91.7% 77 in → 17 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
google/gemini-3.5-flash-lite:medium__json_schema PASS 91.7% 77 in → 325 out (313 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3.6-flash:xhigh__json_schema PASS 91.7% 139 in → 326 out (315 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemma-4-26b-a4b-it:none PASS 91.7% 89 in → 6 out (0 reasoning)
Make build.sh executable
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 91.7% 92 in → 372 out (267 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
google/gemma-4-31b-it:none__json_schema PASS 91.7% 89 in → 13 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemma-4-31b-it:xhigh PASS 91.7% 92 in → 226 out (175 reasoning)
Make build.sh executable
inclusionai/ling-2.6-flash PASS 91.7% 91 in → 5 out
Make build.sh executable
minimax/minimax-m2.5:high PASS 91.7% 100 in → 131 out (125 reasoning)
Make build.sh executable
minimax/minimax-m2.7:high__json_schema PASS 91.7% 236 in → 186 out (174 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
minimax/minimax-m2.7:low PASS 91.7% 100 in → 166 out (194 reasoning)
Make build.sh executable
minimax/minimax-m3:medium PASS 91.7% 238 in → 49 out (46 reasoning)
Make build.sh executable
mistralai/mistral-medium-3-5:low PASS 91.7% 86 in → 562 out (523 reasoning)
Make build.sh executable
mistralai/mistral-medium-3-5:xhigh PASS 91.7% 86 in → 358 out (315 reasoning)
Make build.sh executable
moonshotai/kimi-k2.7-code:medium__json_schema PASS 91.7% 157 in → 146 out (134 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 91.7% 87 in → 901 out (900 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 91.7% 87 in → 104 out (98 reasoning)
Make build.sh executable
nvidia/nemotron-3-nano-30b-a3b:none PASS 91.7% 87 in → 5 out (0 reasoning)
Make build.sh executable
nvidia/nemotron-3-super-120b-a12b:low PASS 91.7% 87 in → 70 out (58 reasoning)
Make build.sh executable
openai/gpt-5.4-mini:high__json_schema PASS 91.7% 106 in → 74 out (55 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.4-nano:high__json_schema PASS 91.7% 106 in → 57 out (36 reasoning)
Update build.sh to be executable
JSON Schema Structured Output
(raw) {"commit_message":"Update build.sh to be executable"}
openai/gpt-5.4-nano:medium PASS 91.7% 68 in → 33 out (23 reasoning)
Make build.sh executable
openai/gpt-5.4-nano:medium__json_schema PASS 91.7% 106 in → 59 out (38 reasoning)
Update: Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Update: Make build.sh executable"}
openai/gpt-5.4:low PASS 91.7% 68 in → 37 out (27 reasoning)
Make build.sh executable
openai/gpt-5.4:none PASS 91.7% 68 in → 8 out (0 reasoning)
Make build.sh executable
openai/gpt-5.4:none__json_schema PASS 91.7% 106 in → 17 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.5:low PASS 91.7% 68 in → 35 out (25 reasoning)
Make build script executable
openai/gpt-5.5:xhigh PASS 91.7% 68 in → 67 out (57 reasoning)
Make build.sh executable
openai/gpt-5.6-luna:low__json_schema PASS 91.7% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-5.6-sol:high__json_schema PASS 91.7% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-5.6-sol:none__json_schema PASS 91.7% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-5.6-sol:xhigh__json_schema PASS 91.7% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-oss-120b:high__json_schema PASS 91.7% 127 in → 290 out (237 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
openai/gpt-oss-120b:medium PASS 91.7% 116 in → 182 out (161 reasoning)
Make build.sh executable
openai/gpt-oss-120b:xhigh PASS 91.7% 131 in → 315 out (278 reasoning)
Make build.sh executable
openai/gpt-oss-20b:medium__json_schema PASS 91.7% 127 in → 159 out (116 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
poolside/laguna-m.1:medium__json_schema PASS 91.7% 85 in → 654 out (642 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
poolside/laguna-xs-2.1:high PASS 91.7% 85 in → 540 out (535 reasoning)
Make build.sh executable
poolside/laguna-xs-2.1:none PASS 91.7% 85 in → 4 out (0 reasoning)
Make build.sh executable
poolside/laguna-xs.2:low__json_schema PASS 91.7% 122 in → 285 out (273 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
poolside/laguna-xs.2:medium PASS 91.7% 122 in → 177 out (170 reasoning)
Make build.sh executable
poolside/laguna-xs.2:none__json_schema PASS 91.7% 122 in → 13 out (0 reasoning)
Chmod +x build.sh
JSON Schema Structured Output
(raw) {"commit_message": "Chmod +x build.sh"}
poolside/laguna-xs.2:xhigh__json_schema PASS 91.7% 122 in → 289 out (277 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
qwen/qwen3.6-27b:low PASS 91.7% 81 in → 1,442 out (1,307 reasoning)
Make build.sh executable
qwen/qwen3.6-27b:none__json_schema PASS 91.7% 83 in → 16 out (0 reasoning)
fix build.sh permissions
JSON Schema Structured Output
(raw) { "commit_message": "fix build.sh permissions" }
qwen/qwen3.6-35b-a3b:low PASS 91.7% 81 in → 1,086 out (940 reasoning)
Make build.sh executable
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 91.7% 81 in → 1,218 out (1,060 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
qwen/qwen3.7-flash:max PASS 91.7% 81 in → 320 out (311 reasoning)
Make build.sh executable
qwen/qwen3.7-flash:none PASS 91.7% 83 in → 4 out (0 reasoning)
Make build.sh executable
qwen/qwen3.7-max:high PASS 91.7% 81 in → 621 out (612 reasoning)
Make build.sh executable
qwen/qwen3.7-max:none PASS 91.7% 83 in → 4 out (0 reasoning)
Make build.sh executable
thinkingmachines/inkling-small:low PASS 91.7% 77 in → 53 out (43 reasoning)
Make build.sh executable
thinkingmachines/inkling-small:medium__json_schema PASS 91.7% 77 in → 96 out (81 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
x-ai/grok-4.3:max PASS 91.7% 247 in → 519 out (515 reasoning)
Make build.sh executable
x-ai/grok-4.3:max__json_schema PASS 91.7% 307 in → 335 out (326 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
x-ai/grok-4.5:low__json_schema PASS 91.7% 343 in → 196 out (185 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
x-ai/grok-4.5:max__json_schema PASS 91.7% 343 in → 246 out (235 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
z-ai/glm-4.7-flash:low PASS 91.7% 69 in → 420 out (406 reasoning)
Make build.sh executable
z-ai/glm-4.7-flash:medium__json_schema PASS 91.7% 69 in → 419 out (385 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
z-ai/glm-4.7-flash:none PASS 91.7% 69 in → 5 out (0 reasoning)
Make build script executable
z-ai/glm-4.7-flash:none__json_schema PASS 91.7% 69 in → 13 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build script executable" }
z-ai/glm-5.2:high__json_schema PASS 91.7% 255 in → 72 out (60 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
z-ai/glm-5.2:medium PASS 91.7% 76 in → 98 out (0 reasoning)
Make build.sh executable
z-ai/glm-5.2:none PASS 91.7% 70 in → 5 out (0 reasoning)
Make build.sh executable
z-ai/glm-5.2:none__json_schema PASS 91.7% 249 in → 10 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-sonnet-4.6:none__json_schema PASS 91.3% 249 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemini-3-flash-preview:xhigh PASS 91.3% 76 in → 225 out (220 reasoning)
Make build.sh executable
google/gemini-3.5-flash:xhigh__json_schema PASS 91.3% 139 in → 502 out (491 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemini-3.6-flash:high__json_schema PASS 91.3% 77 in → 392 out (381 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemini-3.6-flash:medium PASS 91.3% 77 in → 323 out (318 reasoning)
Make build.sh executable
minimax/minimax-m2.5:low__json_schema PASS 91.3% 100 in → 176 out (162 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
mistralai/mistral-medium-3-5:none PASS 91.3% 86 in → 7 out (0 reasoning)
`Fix build.sh permissions`
moonshotai/kimi-k2.7-code:xhigh PASS 91.3% 70 in → 162 out (156 reasoning)
Make build.sh executable
openai/gpt-5.6-terra:none__json_schema PASS 91.3% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-oss-120b:medium__json_schema PASS 91.3% 129 in → 112 out (100 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
qwen/qwen3.7-max:low PASS 91.3% 81 in → 524 out (515 reasoning)
Make build.sh executable
qwen/qwen3.7-plus:none PASS 91.3% 83 in → 4 out (0 reasoning)
Make build.sh executable
thinkingmachines/inkling-small:medium PASS 91.3% 77 in → 53 out (43 reasoning)
Make build.sh executable
thinkingmachines/inkling:medium PASS 91.3% 77 in → 130 out (119 reasoning)
Make build.sh executable
anthropic/claude-haiku-4.5:none PASS 91% 79 in → 8 out (0 reasoning)
Make build.sh executable
anthropic/claude-haiku-4.5:xhigh PASS 91% 108 in → 405 out (244 reasoning)
Make build.sh executable
anthropic/claude-opus-4.6:low__json_schema PASS 91% 249 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.6:none__json_schema PASS 91% 249 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.6:xhigh__json_schema PASS 91% 249 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.7:none PASS 91% 104 in → 12 out (0 reasoning)
Make build.sh executable
anthropic/claude-opus-4.8:high__json_schema PASS 91% 320 in → 20 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.8:low__json_schema PASS 91% 320 in → 20 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.8:xhigh PASS 91% 99 in → 46 out (19 reasoning)
Make build.sh executable
anthropic/claude-sonnet-4.6:low__json_schema PASS 91% 249 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-sonnet-4.6:max PASS 91% 79 in → 73 out (49 reasoning)
`Make build.sh executable`
anthropic/claude-sonnet-5:low__json_schema PASS 91% 320 in → 20 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-sonnet-5:xhigh__json_schema PASS 91% 320 in → 20 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 91% 151 in → 135 out (134 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 91% 66 in → 123 out (120 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable" }
deepseek/deepseek-v4-flash:high__json_schema PASS 91% 66 in → 202 out (194 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
deepseek/deepseek-v4-flash:xhigh PASS 91% 145 in → 1,204 out (1,196 reasoning)
Add executable permission to build.sh
deepseek/deepseek-v4-pro:medium__json_schema PASS 91% 68 in → 128 out (114 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 91% 335 in → 174 out (162 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3-flash-preview:high PASS 91% 76 in → 301 out (296 reasoning)
Make build.sh executable
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 91% 139 in → 161 out (149 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3.5-flash-lite:max PASS 91% 77 in → 357 out (352 reasoning)
Make build.sh executable
google/gemini-3.5-flash-lite:medium PASS 91% 76 in → 396 out (391 reasoning)
Make build.sh executable
google/gemini-3.5-flash:high__json_schema PASS 91% 139 in → 371 out (359 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3.5-flash:xhigh PASS 91% 76 in → 380 out (375 reasoning)
Make build.sh executable
google/gemini-3.6-flash:low PASS 91% 77 in → 188 out (183 reasoning)
Make build.sh executable
google/gemini-3.6-flash:max PASS 91% 76 in → 297 out (292 reasoning)
Make build.sh executable
google/gemma-4-26b-a4b-it:high PASS 91% 92 in → 363 out (302 reasoning)
Make build.sh executable
google/gemma-4-26b-a4b-it:low__json_schema PASS 91% 92 in → 361 out (270 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
google/gemma-4-31b-it:low PASS 91% 91 in → 348 out (299 reasoning)
Make build.sh executable
google/gemma-4-31b-it:none PASS 91% 89 in → 6 out (0 reasoning)
Make build.sh executable
ibm-granite/granite-4.1-8b PASS 91% 77 in → 5 out
Make build.sh executable
minimax/minimax-m2.5:high__json_schema PASS 91% 100 in → 122 out (123 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
minimax/minimax-m2.7:xhigh PASS 91% 100 in → 162 out (156 reasoning)
Make build.sh executable
minimax/minimax-m2.7:xhigh__json_schema PASS 91% 100 in → 14 out (14 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build script executable" }
minimax/minimax-m3:high__json_schema PASS 91% 186 in → 13 out (12 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
minimax/minimax-m3:low PASS 91% 238 in → 46 out (43 reasoning)
Make build.sh executable
moonshotai/kimi-k2.7-code:low__json_schema PASS 91% 157 in → 184 out (172 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
nvidia/nemotron-3-super-120b-a12b:high PASS 91% 87 in → 111 out (99 reasoning)
Make build.sh executable
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 91% 87 in → 124 out (96 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 91% 87 in → 86 out (63 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 91% 87 in → 228 out (197 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
openai/gpt-5.4-mini:medium__json_schema PASS 91% 106 in → 61 out (42 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.4-nano:none__json_schema PASS 91% 106 in → 22 out (0 reasoning)
Fix build.sh to be executable (755)
JSON Schema Structured Output
(raw) {"commit_message":"Fix build.sh to be executable (755)"}
openai/gpt-5.4:high PASS 91% 68 in → 66 out (56 reasoning)
Make build.sh executable
openai/gpt-5.4:high__json_schema PASS 91% 106 in → 82 out (63 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.4:xhigh__json_schema PASS 91% 106 in → 90 out (71 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.5:low__json_schema PASS 91% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-5.6-luna:low PASS 91% 68 in → 8 out (0 reasoning)
Make build script executable
openai/gpt-5.6-luna:max__json_schema PASS 91% 106 in → 59 out (40 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.6-sol:max__json_schema PASS 91% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-5.6-sol:medium PASS 91% 68 in → 8 out (0 reasoning)
Make build script executable
openai/gpt-5.6-sol:none PASS 91% 68 in → 8 out (0 reasoning)
Make build script executable
openai/gpt-5.6-sol:xhigh PASS 91% 68 in → 8 out (0 reasoning)
Make build script executable
openai/gpt-oss-120b:low PASS 91% 129 in → 94 out (82 reasoning)
Make build.sh executable
openai/gpt-oss-120b:xhigh__json_schema PASS 91% 127 in → 546 out (390 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
poolside/laguna-m.1:xhigh PASS 91% 85 in → 272 out (265 reasoning)
Make build.sh executable
poolside/laguna-xs-2.1:high__json_schema PASS 91% 85 in → 577 out (565 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
poolside/laguna-xs-2.1:low PASS 91% 85 in → 374 out (369 reasoning)
Make build.sh executable
poolside/laguna-xs.2:high__json_schema PASS 91% 122 in → 129 out (117 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
qwen/qwen3.6-27b:low__json_schema PASS 91% 81 in → 1,029 out (972 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
qwen/qwen3.6-27b:medium__json_schema PASS 91% 81 in → 988 out (885 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
qwen/qwen3.6-flash:high PASS 91% 81 in → 1,156 out (1,147 reasoning)
Make build.sh executable
qwen/qwen3.7-flash:high PASS 91% 81 in → 1,203 out (1,194 reasoning)
Make build.sh executable
qwen/qwen3.7-flash:low PASS 91% 81 in → 895 out (886 reasoning)
Make build.sh executable
qwen/qwen3.7-plus:high PASS 91% 81 in → 770 out (761 reasoning)
Make build.sh executable
qwen/qwen3.7-plus:medium PASS 91% 81 in → 612 out (603 reasoning)
Make build.sh executable
qwen/qwen3.7-plus:xhigh PASS 91% 81 in → 620 out (611 reasoning)
Make build.sh executable
tencent/hy3:high__json_schema PASS 91% 74 in → 459 out (390 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
tencent/hy3:none__json_schema PASS 91% 77 in → 13 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
thinkingmachines/inkling-small:low__json_schema PASS 91% 77 in → 46 out (31 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
thinkingmachines/inkling:max PASS 91% 77 in → 142 out (137 reasoning)
Make build.sh executable
thinkingmachines/inkling:xhigh PASS 91% 77 in → 178 out (167 reasoning)
Make build.sh executable
x-ai/grok-4.3:high PASS 91% 247 in → 234 out (230 reasoning)
Make build.sh executable
x-ai/grok-4.3:low__json_schema PASS 91% 313 in → 313 out (304 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
x-ai/grok-4.5:low PASS 91% 271 in → 182 out (178 reasoning)
Make build.sh executable
x-ai/grok-4.5:medium__json_schema PASS 91% 343 in → 278 out (267 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
z-ai/glm-4.7-flash:high PASS 91% 69 in → 466 out (457 reasoning)
Make build.sh executable
z-ai/glm-5.2:medium__json_schema PASS 91% 78 in → 455 out (441 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
anthropic/claude-opus-4.8:xhigh__json_schema PASS 90.7% 320 in → 20 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
deepseek/deepseek-v4-flash:none PASS 90.7% 66 in → 6 out (0 reasoning)
Fix permissions in build.sh
google/gemini-3.5-flash-lite:high__json_schema PASS 90.7% 77 in → 356 out (345 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
minimax/minimax-m3:xhigh PASS 90.7% 238 in → 63 out (63 reasoning)
Make build.sh executable
moonshotai/kimi-k2.7-code:low PASS 90.7% 70 in → 1,623 out (1,617 reasoning)
Make build.sh executable
openai/gpt-5.4-mini:medium PASS 90.7% 68 in → 47 out (37 reasoning)
Make build.sh executable
openai/gpt-5.6-luna:xhigh PASS 90.7% 68 in → 51 out (41 reasoning)
Make build.sh executable
openai/gpt-5.6-terra:xhigh PASS 90.7% 68 in → 41 out (31 reasoning)
Make build script executable
tencent/hy3:xhigh__json_schema PASS 90.7% 74 in → 582 out (521 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
thinkingmachines/inkling:low PASS 90.7% 77 in → 50 out (39 reasoning)
Make build.sh executable
x-ai/grok-4.5:medium PASS 90.7% 271 in → 192 out (188 reasoning)
Make build.sh executable
deepseek/deepseek-v4-flash-0731:low PASS 90.3% 66 in → 147 out (149 reasoning)
Make build.sh executable
openai/gpt-5.6-sol:high PASS 90.3% 68 in → 8 out (0 reasoning)
Make build script executable
anthropic/claude-fable-5:high__json_schema PASS 90% 320 in → 34 out (15 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-fable-5:low PASS 90% 99 in → 20 out (8 reasoning)
Make build.sh executable
anthropic/claude-haiku-4.5:high__json_schema PASS 90% 278 in → 371 out (340 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-haiku-4.5:low PASS 90% 108 in → 211 out (184 reasoning)
``` Make build.sh executable ```
anthropic/claude-haiku-4.5:low__json_schema PASS 90% 278 in → 279 out (236 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 90% 278 in → 271 out (216 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-opus-4.7:high__json_schema PASS 90% 325 in → 22 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-sonnet-4.6:low PASS 90% 79 in → 10 out (0 reasoning)
`Make build.sh executable`
anthropic/claude-sonnet-5:low PASS 90% 99 in → 10 out (0 reasoning)
Make build.sh executable
anthropic/claude-sonnet-5:none__json_schema PASS 90% 320 in → 19 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
arcee-ai/trinity-mini:xhigh PASS 90% 71 in → 348 out (375 reasoning)
Update build.sh permissions
deepseek/deepseek-v4-flash-0731:medium PASS 90% 66 in → 209 out (204 reasoning)
Make build.sh executable
deepseek/deepseek-v4-flash:low PASS 90% 66 in → 205 out (199 reasoning)
Make build.sh executable
deepseek/deepseek-v4-flash:medium PASS 90% 66 in → 452 out (446 reasoning)
Make build.sh executable
deepseek/deepseek-v4-pro:xhigh PASS 90% 145 in → 240 out (235 reasoning)
Make build.sh executable
google/gemini-3.1-flash-lite-preview:xhigh PASS 90% 76 in → 386 out (381 reasoning)
Make build.sh executable
google/gemini-3.5-flash-lite:max__json_schema PASS 90% 77 in → 407 out (390 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
google/gemini-3.5-flash:low__json_schema PASS 90% 139 in → 218 out (207 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemini-3.6-flash:max__json_schema PASS 90% 77 in → 438 out (427 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
google/gemma-4-31b-it:medium PASS 90% 92 in → 225 out (174 reasoning)
Make build.sh executable
minimax/minimax-m2.5:medium__json_schema PASS 90% 100 in → 241 out (254 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
minimax/minimax-m2.7:medium__json_schema PASS 90% 99 in → 142 out (154 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
moonshotai/kimi-k2.6:high PASS 90% 70 in → 242 out (236 reasoning)
Make build.sh executable
moonshotai/kimi-k2.6:medium PASS 90% 69 in → 308 out (292 reasoning)
Make build.sh executable
moonshotai/kimi-k2.6:medium__json_schema PASS 90% 70 in → 268 out (253 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 90% 87 in → 14 out (0 reasoning)
Update permissions for build script
JSON Schema Structured Output
(raw) { "commit_message": "Update permissions for build script" }
openai/gpt-5.4:low__json_schema PASS 90% 106 in → 34 out (15 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.4:medium PASS 90% 68 in → 51 out (41 reasoning)
Make build.sh executable
openai/gpt-5.4:xhigh PASS 90% 68 in → 96 out (86 reasoning)
Make build.sh executable
openai/gpt-5.5:high PASS 90% 68 in → 44 out (34 reasoning)
Make build script executable
openai/gpt-5.6-luna:medium__json_schema PASS 90% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-5.6-luna:none PASS 90% 68 in → 8 out (0 reasoning)
Make build script executable
openai/gpt-5.6-luna:none__json_schema PASS 90% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-5.6-sol:medium__json_schema PASS 90% 106 in → 17 out (0 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-oss-20b:xhigh__json_schema PASS 90% 129 in → 249 out (230 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
poolside/laguna-m.1:medium PASS 90% 85 in → 216 out (209 reasoning)
Make build.sh executable
poolside/laguna-xs-2.1:medium__json_schema PASS 90% 85 in → 337 out (325 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
poolside/laguna-xs.2:low PASS 90% 122 in → 285 out (278 reasoning)
Make build.sh executable
qwen/qwen3.6-35b-a3b:none PASS 90% 83 in → 4 out (0 reasoning)
Make build.sh executable
thinkingmachines/inkling-small:high__json_schema PASS 90% 77 in → 166 out (151 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
thinkingmachines/inkling:medium__json_schema PASS 90% 77 in → 145 out (127 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
thinkingmachines/inkling:none PASS 90% 77 in → 7 out (0 reasoning)
Make build.sh executable<|end_message|>
x-ai/grok-4.3:none__json_schema PASS 90% 305 in → 9 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
x-ai/grok-4.3:xhigh PASS 90% 247 in → 326 out (322 reasoning)
Make build.sh executable
x-ai/grok-4.5:xhigh PASS 90% 271 in → 217 out (213 reasoning)
Make build.sh executable
z-ai/glm-5.2:xhigh__json_schema PASS 90% 76 in → 545 out (532 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
anthropic/claude-haiku-4.5:high PASS 89.7% 108 in → 310 out (243 reasoning)
Make build.sh executable
anthropic/claude-sonnet-4.6:medium PASS 89.3% 79 in → 10 out (0 reasoning)
`Make build.sh executable`
anthropic/claude-sonnet-4.6:medium__json_schema PASS 89.3% 249 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-sonnet-5:xhigh PASS 89.3% 99 in → 35 out (14 reasoning)
Make build.sh executable
arcee-ai/trinity-mini:low PASS 89.3% 71 in → 308 out (338 reasoning)
Make executable
arcee-ai/trinity-mini:low__json_schema PASS 89.3% 71 in → 392 out (411 reasoning)
Make executable
JSON Schema Structured Output
(raw) { "commit_message": "Make executable" }
deepseek/deepseek-v4-pro:medium PASS 89.3% 66 in → 190 out (185 reasoning)
Make build.sh executable
google/gemini-3.1-pro-preview:high PASS 89.3% 76 in → 486 out (481 reasoning)
Make build.sh executable
google/gemma-4-26b-a4b-it:medium PASS 89.3% 92 in → 317 out (253 reasoning)
Make build.sh executable
mistralai/devstral-2512__json_schema PASS 89.3% 74 in → 13 out
Update build.sh permissions to executable
JSON Schema Structured Output
(raw) {"commit_message": "Update build.sh permissions to executable"}
mistralai/mistral-medium-3-5:medium__json_schema PASS 89.3% 74 in → 314 out (270 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
moonshotai/kimi-k2.7-code:high__json_schema PASS 89.3% 157 in → 97 out (85 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
openai/gpt-5.4-mini:none PASS 89.3% 68 in → 8 out (0 reasoning)
Make build script executable
openai/gpt-5.4-nano:none PASS 89.3% 68 in → 11 out (0 reasoning)
Make build.sh executable (chmod)
openai/gpt-5.4-nano:xhigh__json_schema PASS 89.3% 106 in → 82 out (63 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.6-luna:medium PASS 89.3% 68 in → 8 out (0 reasoning)
Make build script executable
openai/gpt-5.6-terra:high PASS 89.3% 68 in → 33 out (23 reasoning)
Make build script executable
openai/gpt-5.6-terra:low__json_schema PASS 89.3% 106 in → 35 out (16 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-5.6-terra:medium__json_schema PASS 89.3% 106 in → 47 out (28 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
poolside/laguna-xs-2.1:xhigh PASS 89.3% 85 in → 421 out (416 reasoning)
Make build.sh executable
qwen/qwen3.6-flash:none PASS 89.3% 83 in → 4 out (0 reasoning)
Make build.sh executable
tencent/hy3:low PASS 89.3% 74 in → 414 out (408 reasoning)
Make build.sh executable
x-ai/grok-4.3:none PASS 89.3% 245 in → 4 out (0 reasoning)
Make build.sh executable
google/gemma-4-26b-a4b-it:high__json_schema PASS 89% 92 in → 335 out (272 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
openai/gpt-oss-20b:xhigh PASS 89% 129 in → 390 out (378 reasoning)
Make build.sh executable
arcee-ai/trinity-large-thinking:medium PASS 88.3% 71 in → 110 out (103 reasoning)
Update build.sh permissions
google/gemma-4-31b-it:xhigh__json_schema PASS 88.3% 91 in → 365 out (277 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
minimax/minimax-m3:high PASS 88.3% 186 in → 42 out (37 reasoning)
Make build.sh executable
openai/gpt-5.6-luna:xhigh__json_schema PASS 88.3% 106 in → 55 out (36 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
openai/gpt-oss-120b:low__json_schema PASS 88.3% 116 in → 47 out (33 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
poolside/laguna-m.1:none PASS 88.3% 85 in → 7 out (0 reasoning)
Fix build.sh script permissions
qwen/qwen3.6-27b:xhigh__json_schema PASS 88.3% 81 in → 1,171 out (1,061 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
thinkingmachines/inkling:high__json_schema PASS 88.3% 77 in → 116 out (101 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
anthropic/claude-haiku-4.5:medium PASS 87.7% 108 in → 469 out (325 reasoning)
``` Make build.sh executable ```
anthropic/claude-opus-4.6:high PASS 87.7% 79 in → 10 out (0 reasoning)
`Make build.sh executable`
anthropic/claude-opus-4.6:medium PASS 87.7% 79 in → 10 out (0 reasoning)
Change build.sh to executable mode
arcee-ai/trinity-mini:medium PASS 87.7% 71 in → 381 out (420 reasoning)
Set executable
google/gemma-4-26b-a4b-it:low PASS 87.7% 92 in → 299 out (243 reasoning)
Make build.sh executable
minimax/minimax-m3:xhigh__json_schema PASS 87.7% 225 in → 14 out (0 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
moonshotai/kimi-k2.7-code:medium PASS 87.7% 70 in → 163 out (133 reasoning)
Make build.sh executable
openai/gpt-5.4-nano:xhigh PASS 87.7% 68 in → 55 out (45 reasoning)
Make build.sh executable
openai/gpt-5.6-terra:high__json_schema PASS 87.7% 106 in → 48 out (29 reasoning)
Make build script executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build script executable"}
qwen/qwen3.6-27b:none PASS 87.7% 83 in → 5 out (0 reasoning)
make build.sh executable
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 86.7% 145 in → 113 out (102 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message": "Make build.sh executable"}
google/gemini-3.1-flash-lite-preview:medium PASS 86.7% 77 in → 201 out (196 reasoning)
Make build.sh executable
openai/gpt-5.4-mini:none__json_schema PASS 86.7% 106 in → 17 out (0 reasoning)
Fix build script permissions
JSON Schema Structured Output
(raw) {"commit_message":"Fix build script permissions"}
openai/gpt-5.6-sol:low PASS 86.7% 68 in → 8 out (0 reasoning)
Make build script executable
openai/gpt-5.6-terra:medium PASS 86.7% 68 in → 33 out (23 reasoning)
Make build script executable
anthropic/claude-opus-4.6:max__json_schema PASS 86% 249 in → 71 out (41 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
openai/gpt-5.5:xhigh__json_schema PASS 86% 106 in → 83 out (64 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
anthropic/claude-opus-4.6:xhigh PASS 85.3% 79 in → 10 out (0 reasoning)
`Make build.sh executable`
openai/gpt-5.4-nano:low PASS 85% 68 in → 15 out (0 reasoning)
Update build.sh to be executable (chmod +x)
openai/gpt-5.6-terra:low PASS 85% 68 in → 30 out (20 reasoning)
Make build script executable
poolside/laguna-m.1:low PASS 85% 85 in → 257 out (246 reasoning)
Make build.sh executable for direct running.
qwen/qwen3.6-35b-a3b:high PASS 82.3% 81 in → 1,312 out (1,171 reasoning)
Make build.sh executable
arcee-ai/trinity-large-thinking:low PASS 81.7% 71 in → 148 out (141 reasoning)
Make build.sh executable
openai/gpt-oss-20b:medium PASS 81.7% 131 in → 113 out (87 reasoning)
Make build.sh executable.
openai/gpt-oss-20b:high PASS 79.7% 129 in → 284 out (302 reasoning)
Make build.sh executable)
arcee-ai/trinity-mini:medium__json_schema PASS 79.3% 71 in → 605 out (557 reasoning)
Make build.shexecutable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.shexecutable" }
nvidia/nemotron-3-super-120b-a12b:none PASS 78.3% 87 in → 15 out (0 reasoning)
Make build.sh executable (47 characters, fits within limit)
deepseek/deepseek-v4-flash-0731:none PASS 77.7% 66 in → 8 out (0 reasoning)
chmod build.sh to make executable
poolside/laguna-xs-2.1:medium PASS 76.7% 85 in → 258 out (253 reasoning)
Make build.sh executable
inclusionai/ling-2.6-flash__json_schema PASS 71.7% 91 in → 14 out
Fix build script permissions to executable
JSON Schema Structured Output
(raw) {"commit_message":"Fix build script permissions to executable"}
poolside/laguna-m.1:none__json_schema PASS 69% 85 in → 13 out (0 reasoning)
Fix build.sh permissions for execution
JSON Schema Structured Output
(raw) {"commit_message": "Fix build.sh permissions for execution"}
anthropic/claude-opus-4.6:none PASS 68.3% 79 in → 11 out (0 reasoning)
Fix build.sh file permissions to executable
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 63.3% 87 in → 128 out (111 reasoning)
Make build.sh executable
qwen/qwen3.6-35b-a3b:low__json_schema PASS 63.3% 81 in → 1,319 out (1,235 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) { "commit_message": "Make build.sh executable" }
tencent/hy3:medium__json_schema PASS 63.3% 74 in → 486 out (481 reasoning)
Make build.sh executable
JSON Schema Structured Output
(raw) {"commit_message":"Make build.sh executable"}
x-ai/grok-4.3:medium PASS 61.7% 253 in → 362 out (352 reasoning)
Make build.sh executable \confidence{80}
liquid/lfm-2-24b-a2b FAIL 46.7% 74 in → 11 out
**Fix**: Update file permissions to 755
x-ai/grok-4.3:low FAIL 36.7% 253 in → 286 out (240 reasoning)
Make build.sh executable The diff shows only a mode change (100644 → 100755), indicating the executable bit was added to build.sh. A short, action-oriented message captures this exactly within the 50-character limit.
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 6.7% 70 in → 222 out (215 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"commit_message":""}
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: Update file permissions
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k3:high FAIL 0% 147 in → 127 out (106 reasoning)
**Make build.sh executable**
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:high__json_schema FAIL 0% 147 in → 110 out (91 reasoning)
Make build.sh executable
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low FAIL 0% 147 in → 116 out (97 reasoning)
Make build.sh executable
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low__json_schema FAIL 0% 147 in → 104 out (85 reasoning)
Make build.sh executable
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max FAIL 0% 147 in → 105 out (86 reasoning)
Make build.sh executable
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max__json_schema FAIL 0% 147 in → 126 out (107 reasoning)
Make build.sh executable
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium FAIL 0% 147 in → 128 out (81 reasoning)
**Make build.sh executable** (24 characters — the diff only changes the file mode from 644 to 755, granting execute permission to the build script.)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium__json_schema FAIL 0% 147 in → 101 out (82 reasoning)
Make build.sh executable
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh FAIL 0% 147 in → 141 out (96 reasoning)
**Make build.sh executable** (24 characters — describes the file mode change from 644 to 755, adding execute permission to the build script.)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh__json_schema FAIL 0% 147 in → 105 out (86 reasoning)
Make build.sh executable
Failure: Scoring error: llm_judge requires a judge client
nvidia/nemotron-3-super-120b-a12b:medium FAIL 0% 87 in → 408 out (407 reasoning)
Make build.sh executable
Failure: Scoring error: llm_judge requires a judge client
nvidia/nemotron-3-super-120b-a12b:xhigh FAIL 0% 87 in → 107 out (106 reasoning)
Make build.sh executable
Failure: Scoring error: llm_judge requires a judge client
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1afb67b5-bb54-9bca-b80a-0088d5ddcbc3","request_id":"1afb67b5-bb54-9bca-b80a-0088d5ddcbc3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9f43203d-78d8-97d9-8e56-681f86368400","request_id":"9f43203d-78d8-97d9-8e56-681f86368400"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c7121a3f-802c-9c65-bb98-ae1a8450a166","request_id":"c7121a3f-802c-9c65-bb98-ae1a8450a166"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a0da995c-7829-9af3-9cf4-5386c4530938","request_id":"a0da995c-7829-9af3-9cf4-5386c4530938"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0b389051-5be7-9099-97d8-ea21ea4fd4b9","request_id":"0b389051-5be7-9099-97d8-ea21ea4fd4b9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-afb5694b-2906-93fe-9375-35fae4c1efe6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c2cba359-2275-9474-93e1-156520e1e1b8"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-737325e3-742d-975e-8570-22de8a16b243"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c5853133-d158-9d90-b4f7-45ae23693e32"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-73acb1aa-41a9-90cc-84e0-0809b09fc4cf"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6716640a-72ab-9bcc-8205-7141294e5d59"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c8a10892-e7a9-9c28-93c9-83ebae6f101f","request_id":"c8a10892-e7a9-9c28-93c9-83ebae6f101f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bbf885ca-fee9-9ca7-b43c-84956a429429","request_id":"bbf885ca-fee9-9ca7-b43c-84956a429429"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7dddfcc8-0db9-9b34-aa9f-71e64c1b4da1","request_id":"7dddfcc8-0db9-9b34-aa9f-71e64c1b4da1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fd206eaa-74df-9443-882c-0b6b21203912","request_id":"fd206eaa-74df-9443-882c-0b6b21203912"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f2bbacf0-3680-9f33-8c4b-9ee246e6ed60","request_id":"f2bbacf0-3680-9f33-8c4b-9ee246e6ed60"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fd7fc309-dbf9-93cf-bea5-117681c4d0ee","request_id":"fd7fc309-dbf9-93cf-bea5-117681c4d0ee"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b3126a10-e425-9902-b8ca-d8264ddfa92a","request_id":"b3126a10-e425-9902-b8ca-d8264ddfa92a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2a5e328d-5af1-9c94-ae06-bb58a923b257","request_id":"2a5e328d-5af1-9c94-ae06-bb58a923b257"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3859d7d3-ffcc-9227-baa9-0b03be9394a2","request_id":"3859d7d3-ffcc-9227-baa9-0b03be9394a2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a6615c8a-678b-95f5-bb28-0f36de5dc88f","request_id":"a6615c8a-678b-95f5-bb28-0f36de5dc88f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: Make build.sh executable{ "commit_message": "Make build.sh executable" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: Make build.sh executable
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:max__json_schema FAIL 0%
Invalid JSON. Output: Make build.sh executable
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: Make build.sh executable{"commit_message": "Make build.sh executable"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:xhigh__json_schema FAIL 0%
Invalid JSON. Output: Make build.sh executable
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)