File deleted
Tests ability to generate a commit message for a file deletion. Evaluates handling of removal operations.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'to be deleted' > redundant.txt
  5. 05 git add redundant.txt
  6. 06 git commit -m 'Add file'
  7. 07 git rm redundant.txt
Prompt
Generate a concise commit message (max 50 characters) describing the changes in this diff. Focus on the action taken (e.g., 'Add', 'Fix', 'Update', 'Remove').
Expected
Remove redundant.txt

Scoped model quality, cost, API time, and token usage for commit_messages/f004.

Loading...
Loading raw attempt evidence…
deepseek/deepseek-v4-pro:low__json_schema PASS 98.7% 299 in → 96 out (86 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 98.7% 131 in → 99 out (95 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
deepseek/deepseek-v4-flash:none PASS 98.3% 108 in → 3 out (0 reasoning)
Remove redundant.txt
poolside/laguna-xs.2:none__json_schema PASS 97.7% 160 in → 11 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
google/gemini-3.6-flash:max__json_schema PASS 97.3% 181 in → 216 out (206 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.6-flash:max PASS 97% 118 in → 343 out (339 reasoning)
Remove redundant.txt
openai/gpt-5.4-mini:high PASS 97% 108 in → 69 out (60 reasoning)
Remove redundant.txt
poolside/laguna-xs-2.1:high PASS 97% 123 in → 306 out (302 reasoning)
Remove redundant.txt
poolside/laguna-xs.2:medium__json_schema PASS 97% 160 in → 164 out (153 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
qwen/qwen3.7-plus:xhigh PASS 97% 125 in → 375 out (367 reasoning)
Remove redundant.txt
anthropic/claude-sonnet-4.6:max PASS 96.7% 123 in → 93 out (71 reasoning)
Remove redundant.txt
google/gemini-3-flash-preview:high PASS 96.7% 118 in → 191 out (187 reasoning)
Remove redundant.txt
google/gemini-3-flash-preview:low__json_schema PASS 96.7% 181 in → 299 out (289 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.5-flash-lite:low__json_schema PASS 96.7% 119 in → 16 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
google/gemma-4-31b-it:high__json_schema PASS 96.7% 134 in → 227 out (174 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 96.7% 131 in → 65 out (56 reasoning)
Remove redundant.txt
qwen/qwen3.6-35b-a3b:high__json_schema PASS 96.7% 125 in → 600 out (443 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
tencent/hy3:high__json_schema PASS 96.7% 111 in → 357 out (342 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
minimax/minimax-m2.5:medium__json_schema PASS 96.6% 146 in → 115 out (102 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
anthropic/claude-haiku-4.5:high__json_schema PASS 96.3% 322 in → 399 out (292 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-opus-4.6:max PASS 96% 123 in → 51 out (28 reasoning)
Remove redundant.txt
anthropic/claude-opus-4.7:low__json_schema PASS 96% 382 in → 23 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-opus-4.7:medium__json_schema PASS 96% 382 in → 23 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-sonnet-5:high__json_schema PASS 96% 377 in → 21 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 96% 193 in → 92 out (94 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
deepseek/deepseek-v4-flash:medium PASS 96% 108 in → 141 out (136 reasoning)
Remove redundant.txt
google/gemma-4-31b-it:low__json_schema PASS 96% 133 in → 260 out (203 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
minimax/minimax-m2.7:low PASS 96% 134 in → 147 out (141 reasoning)
Remove redundant.txt
mistralai/mistral-medium-3-5:low__json_schema PASS 96% 118 in → 291 out (307 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
moonshotai/kimi-k2.7-code:low__json_schema PASS 96% 104 in → 51 out (37 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 96% 104 in → 86 out (66 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 96% 131 in → 118 out (108 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
openai/gpt-oss-120b:low__json_schema PASS 96% 171 in → 66 out (56 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
qwen/qwen3.6-35b-a3b:medium PASS 96% 125 in → 866 out (815 reasoning)
Remove redundant.txt
qwen/qwen3.6-35b-a3b:none__json_schema PASS 96% 127 in → 12 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
qwen/qwen3.6-flash:xhigh PASS 96% 125 in → 933 out (925 reasoning)
Remove redundant.txt
qwen/qwen3.7-max:xhigh PASS 96% 125 in → 584 out (576 reasoning)
Remove redundant.txt
tencent/hy3:medium PASS 96% 111 in → 351 out (346 reasoning)
Remove redundant.txt
thinkingmachines/inkling-small:low PASS 96% 117 in → 64 out (53 reasoning)
Remove redundant.txt
thinkingmachines/inkling:max PASS 96% 117 in → 125 out (129 reasoning)
Remove redundant.txt
x-ai/grok-4.3:high__json_schema PASS 96% 343 in → 376 out (368 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
z-ai/glm-4.7-flash:high__json_schema PASS 96% 107 in → 380 out (358 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
z-ai/glm-4.7-flash:xhigh__json_schema PASS 96% 107 in → 451 out (429 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
openai/gpt-5.5:medium PASS 95.7% 108 in → 38 out (29 reasoning)
Remove redundant file
x-ai/grok-4.5:low PASS 95.7% 305 in → 186 out (183 reasoning)
Remove redundant.txt
arcee-ai/trinity-mini:xhigh PASS 95.3% 104 in → 268 out (298 reasoning)
Remove redundant.txt
google/gemini-3.1-pro-preview:high PASS 95.3% 118 in → 293 out (289 reasoning)
Remove redundant.txt
google/gemini-3.6-flash:high PASS 95.3% 118 in → 314 out (310 reasoning)
Remove redundant.txt
nvidia/nemotron-3-nano-30b-a3b:medium PASS 95.3% 131 in → 54 out (55 reasoning)
Remove redundant.txt
poolside/laguna-xs-2.1:medium__json_schema PASS 95.3% 123 in → 245 out (234 reasoning)
Delete redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Delete redundant.txt"}
x-ai/grok-4.5:medium PASS 95.3% 305 in → 187 out (184 reasoning)
Remove redundant.txt
anthropic/claude-fable-5:max PASS 95% 156 in → 219 out (60 reasoning)
``` Remove redundant.txt ```
anthropic/claude-fable-5:medium__json_schema PASS 95% 377 in → 22 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
anthropic/claude-haiku-4.5:xhigh PASS 95% 152 in → 285 out (230 reasoning)
Remove redundant.txt
anthropic/claude-opus-4.7:medium PASS 95% 161 in → 13 out (0 reasoning)
Remove redundant.txt
anthropic/claude-sonnet-5:medium PASS 95% 156 in → 11 out (0 reasoning)
Remove redundant.txt
arcee-ai/trinity-mini:medium__json_schema PASS 95% 104 in → 269 out (301 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
google/gemini-3.1-pro-preview:medium PASS 95% 118 in → 293 out (289 reasoning)
Remove redundant.txt
google/gemini-3.5-flash-lite:max PASS 95% 119 in → 326 out (322 reasoning)
Remove redundant.txt
google/gemini-3.5-flash-lite:xhigh PASS 95% 119 in → 270 out (266 reasoning)
Remove redundant.txt
google/gemini-3.5-flash:low__json_schema PASS 95% 181 in → 75 out (59 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
google/gemini-3.6-flash:high__json_schema PASS 95% 181 in → 244 out (234 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemma-4-26b-a4b-it:high__json_schema PASS 95% 133 in → 230 out (185 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 95% 134 in → 252 out (191 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
minimax/minimax-m2.7:xhigh__json_schema PASS 95% 270 in → 163 out (152 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
minimax/minimax-m3:high PASS 95% 272 in → 56 out (62 reasoning)
Remove redundant.txt
minimax/minimax-m3:high__json_schema PASS 95% 220 in → 12 out (11 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
mistralai/mistral-medium-3-5:high__json_schema PASS 95% 118 in → 120 out (115 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
openai/gpt-5.4-mini:low PASS 95% 108 in → 26 out (17 reasoning)
Remove redundant file
openai/gpt-5.4-nano:low__json_schema PASS 95% 146 in → 16 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.4-nano:medium PASS 95% 108 in → 7 out (0 reasoning)
Delete redundant.txt
openai/gpt-5.5:medium__json_schema PASS 95% 146 in → 16 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.6-terra:xhigh PASS 95% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-oss-20b:low__json_schema PASS 95% 169 in → 38 out (17 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
poolside/laguna-xs.2:low__json_schema PASS 95% 160 in → 260 out (244 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
qwen/qwen3.6-27b:xhigh PASS 95% 125 in → 841 out (794 reasoning)
Remove redundant.txt
qwen/qwen3.7-flash:high PASS 95% 125 in → 657 out (649 reasoning)
Remove redundant.txt
qwen/qwen3.7-flash:low PASS 95% 125 in → 765 out (757 reasoning)
Remove redundant.txt
qwen/qwen3.7-flash:xhigh PASS 95% 125 in → 853 out (845 reasoning)
Remove redundant.txt
qwen/qwen3.7-plus:high PASS 95% 125 in → 372 out (364 reasoning)
Remove redundant.txt
x-ai/grok-4.3:xhigh__json_schema PASS 95% 343 in → 366 out (358 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
x-ai/grok-4.5:max PASS 95% 305 in → 179 out (176 reasoning)
Remove redundant.txt
x-ai/grok-4.5:xhigh PASS 95% 305 in → 212 out (209 reasoning)
Remove redundant.txt
z-ai/glm-5.2:none__json_schema PASS 95% 108 in → 12 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
deepseek/deepseek-v4-pro:none__json_schema PASS 94.7% 110 in → 12 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
google/gemini-3.1-pro-preview:low__json_schema PASS 94.7% 181 in → 51 out (40 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
qwen/qwen3.7-max:medium PASS 94.7% 125 in → 522 out (514 reasoning)
Remove redundant.txt
anthropic/claude-haiku-4.5:medium PASS 94.3% 152 in → 226 out (188 reasoning)
``` Remove redundant.txt ```
anthropic/claude-opus-4.6:medium__json_schema PASS 94.3% 293 in → 14 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-opus-4.6:none PASS 94.3% 123 in → 8 out (0 reasoning)
Remove redundant.txt
anthropic/claude-opus-4.7:high PASS 94.3% 161 in → 13 out (0 reasoning)
Remove redundant.txt
anthropic/claude-sonnet-5:low PASS 94.3% 156 in → 11 out (0 reasoning)
Remove redundant.txt
anthropic/claude-sonnet-5:low__json_schema PASS 94.3% 377 in → 21 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
deepseek/deepseek-v4-flash-0731:high PASS 94.3% 187 in → 37 out (33 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 94.3% 189 in → 166 out (156 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
deepseek/deepseek-v4-pro:medium__json_schema PASS 94.3% 110 in → 205 out (192 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
google/gemini-3-flash-preview:medium PASS 94.3% 118 in → 265 out (261 reasoning)
Remove redundant.txt
google/gemini-3-flash-preview:xhigh__json_schema PASS 94.3% 181 in → 333 out (323 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.1-flash-lite-preview:high PASS 94.3% 118 in → 305 out (301 reasoning)
Remove redundant.txt
google/gemini-3.1-flash-lite-preview:low PASS 94.3% 119 in → 138 out (134 reasoning)
Remove redundant.txt
google/gemini-3.1-pro-preview:medium__json_schema PASS 94.3% 181 in → 290 out (280 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.5-flash:xhigh__json_schema PASS 94.3% 181 in → 404 out (394 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.6-flash:medium__json_schema PASS 94.3% 181 in → 156 out (146 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.6-flash:xhigh__json_schema PASS 94.3% 181 in → 251 out (241 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemma-4-26b-a4b-it:high PASS 94.3% 134 in → 307 out (266 reasoning)
Remove redundant.txt
google/gemma-4-31b-it:medium__json_schema PASS 94.3% 134 in → 177 out (130 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
minimax/minimax-m2.5:xhigh PASS 94.3% 134 in → 136 out (132 reasoning)
Remove redundant.txt
minimax/minimax-m2.5:xhigh__json_schema PASS 94.3% 146 in → 142 out (147 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
minimax/minimax-m2.7:xhigh PASS 94.3% 134 in → 169 out (164 reasoning)
Remove redundant.txt
minimax/minimax-m3:xhigh PASS 94.3% 272 in → 60 out (64 reasoning)
Remove redundant.txt file
moonshotai/kimi-k2.6:high PASS 94.3% 104 in → 257 out (252 reasoning)
Remove redundant.txt
moonshotai/kimi-k2.7-code:high__json_schema PASS 94.3% 104 in → 140 out (126 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 94.3% 131 in → 13 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 94.3% 131 in → 82 out (72 reasoning)
Remove redundant.txt
openai/gpt-5.4-mini:xhigh PASS 94.3% 108 in → 97 out (88 reasoning)
Remove redundant.txt
openai/gpt-5.4:medium__json_schema PASS 94.3% 146 in → 51 out (33 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.6-terra:none__json_schema PASS 94.3% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-oss-20b:high__json_schema PASS 94.3% 169 in → 492 out (475 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
poolside/laguna-m.1:medium__json_schema PASS 94.3% 123 in → 208 out (191 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
poolside/laguna-m.1:none__json_schema PASS 94.3% 123 in → 10 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
poolside/laguna-xs.2:medium PASS 94.3% 160 in → 121 out (115 reasoning)
Remove redundant.txt
qwen/qwen3.6-27b:high__json_schema PASS 94.3% 127 in → 12 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
qwen/qwen3.6-27b:low PASS 94.3% 125 in → 736 out (711 reasoning)
Remove redundant.txt
qwen/qwen3.6-27b:medium PASS 94.3% 125 in → 825 out (777 reasoning)
Remove redundant.txt
qwen/qwen3.6-27b:medium__json_schema PASS 94.3% 125 in → 897 out (1 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
qwen/qwen3.6-flash:low PASS 94.3% 125 in → 1,089 out (1,081 reasoning)
Remove redundant.txt
qwen/qwen3.7-max:high PASS 94.3% 125 in → 441 out (433 reasoning)
Remove redundant.txt
qwen/qwen3.7-plus:low PASS 94.3% 125 in → 343 out (335 reasoning)
Remove redundant.txt
qwen/qwen3.7-plus:medium PASS 94.3% 125 in → 449 out (441 reasoning)
Remove redundant.txt
tencent/hy3:xhigh__json_schema PASS 94.3% 111 in → 330 out (319 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
thinkingmachines/inkling-small:xhigh__json_schema PASS 94.3% 117 in → 109 out (95 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
thinkingmachines/inkling:low PASS 94.3% 117 in → 38 out (28 reasoning)
Remove redundant.txt
thinkingmachines/inkling:none PASS 94.3% 115 in → 8 out (0 reasoning)
Remove redundant.txt file
x-ai/grok-4.3:xhigh PASS 94.3% 283 in → 392 out (389 reasoning)
Remove redundant.txt
x-ai/grok-4.5:xhigh__json_schema PASS 94.3% 377 in → 330 out (320 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
z-ai/glm-4.7-flash:medium__json_schema PASS 94.3% 107 in → 384 out (369 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
z-ai/glm-5.2:high PASS 94.3% 114 in → 244 out (235 reasoning)
Remove redundant.txt
z-ai/glm-5.2:none PASS 94.3% 108 in → 4 out (0 reasoning)
Remove redundant.txt
anthropic/claude-opus-4.7:max PASS 94% 161 in → 14 out (0 reasoning)
Remove redundant.txt file
anthropic/claude-sonnet-5:xhigh PASS 94% 156 in → 30 out (9 reasoning)
Remove redundant.txt
arcee-ai/trinity-mini:xhigh__json_schema PASS 94% 104 in → 408 out (445 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message":"Remove redundant.txt" }
deepseek/deepseek-v4-flash:high PASS 94% 108 in → 114 out (109 reasoning)
Remove redundant.txt
google/gemini-3.1-pro-preview:xhigh PASS 94% 118 in → 298 out (294 reasoning)
Remove redundant.txt
google/gemma-4-31b-it:low PASS 94% 134 in → 166 out (134 reasoning)
Remove redundant.txt
ibm-granite/granite-4.1-8b PASS 94% 121 in → 4 out
Remove redundant.txt
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
minimax/minimax-m2.7:high__json_schema PASS 94% 270 in → 175 out (157 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
moonshotai/kimi-k2.7-code:xhigh PASS 94% 103 in → 98 out (93 reasoning)
Remove redundant.txt
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 94% 131 in → 1,449 out (1,365 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
openai/gpt-oss-120b:xhigh PASS 94% 167 in → 132 out (145 reasoning)
Remove redundant.txt
openai/gpt-oss-20b:low PASS 94% 156 in → 40 out (24 reasoning)
Remove redundant.txt file.
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 94% 125 in → 372 out (315 reasoning)
Delete redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Delete redundant.txt" }
thinkingmachines/inkling-small:low__json_schema PASS 94% 117 in → 44 out (30 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
x-ai/grok-4.5:low__json_schema PASS 94% 377 in → 303 out (293 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-opus-4.7:none PASS 93.7% 161 in → 13 out (0 reasoning)
Remove redundant.txt
anthropic/claude-sonnet-5:none__json_schema PASS 93.7% 377 in → 20 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-sonnet-5:xhigh__json_schema PASS 93.7% 377 in → 21 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
arcee-ai/trinity-mini:medium PASS 93.7% 104 in → 262 out (287 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 93.7% 108 in → 172 out (169 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 93.7% 108 in → 138 out (127 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
deepseek/deepseek-v4-flash:medium__json_schema PASS 93.7% 110 in → 99 out (88 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
deepseek/deepseek-v4-pro:medium PASS 93.7% 108 in → 117 out (113 reasoning)
Remove redundant.txt
google/gemini-3-flash-preview:medium__json_schema PASS 93.7% 119 in → 319 out (309 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 93.7% 181 in → 275 out (259 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 93.7% 181 in → 378 out (367 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
google/gemini-3.5-flash-lite:max__json_schema PASS 93.7% 119 in → 343 out (332 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
google/gemini-3.5-flash:medium__json_schema PASS 93.7% 181 in → 357 out (346 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
google/gemini-3.5-flash:xhigh PASS 93.7% 118 in → 346 out (342 reasoning)
Remove redundant.txt
google/gemini-3.6-flash:low PASS 93.7% 119 in → 155 out (151 reasoning)
Remove redundant.txt
google/gemma-4-26b-a4b-it:low PASS 93.7% 134 in → 262 out (228 reasoning)
Remove redundant.txt
google/gemma-4-31b-it:medium PASS 93.7% 134 in → 149 out (117 reasoning)
Remove redundant.txt
mistralai/mistral-medium-3-5:medium PASS 93.7% 130 in → 380 out (355 reasoning)
Remove redundant.txt
moonshotai/kimi-k2.6:low__json_schema PASS 93.7% 103 in → 450 out (245 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
moonshotai/kimi-k2.7-code:high PASS 93.7% 104 in → 107 out (99 reasoning)
Remove redundant.txt
openai/gpt-5.4:low__json_schema PASS 93.7% 146 in → 16 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.6-terra:max PASS 93.7% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-oss-120b:high PASS 93.7% 167 in → 149 out (153 reasoning)
Remove redundant.txt
poolside/laguna-xs-2.1:low PASS 93.7% 123 in → 242 out (238 reasoning)
Remove redundant.txt
qwen/qwen3.6-27b:none PASS 93.7% 127 in → 4 out (0 reasoning)
Remove redundant.txt
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 93.7% 125 in → 905 out (828 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
qwen/qwen3.6-flash:high PASS 93.7% 125 in → 1,003 out (995 reasoning)
Remove redundant.txt
qwen/qwen3.7-flash:max PASS 93.7% 125 in → 665 out (657 reasoning)
Remove redundant.txt
thinkingmachines/inkling-small:high PASS 93.7% 117 in → 64 out (65 reasoning)
Remove redundant.txt
x-ai/grok-4.3:max__json_schema PASS 93.7% 343 in → 350 out (338 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
x-ai/grok-4.3:none__json_schema PASS 93.7% 341 in → 8 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
z-ai/glm-4.7-flash:medium PASS 93.7% 107 in → 376 out (363 reasoning)
Remove redundant.txt
z-ai/glm-5.2:xhigh PASS 93.7% 114 in → 180 out (175 reasoning)
Remove redundant.txt
anthropic/claude-fable-5:max__json_schema PASS 93.3% 377 in → 140 out (51 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-fable-5:medium PASS 93.3% 156 in → 12 out (0 reasoning)
Remove redundant.txt file
anthropic/claude-opus-4.6:low__json_schema PASS 93.3% 293 in → 14 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-opus-4.6:medium PASS 93.3% 123 in → 8 out (0 reasoning)
Remove redundant.txt
anthropic/claude-opus-4.6:none__json_schema PASS 93.3% 293 in → 14 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-opus-4.6:xhigh__json_schema PASS 93.3% 293 in → 14 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-opus-4.7:none__json_schema PASS 93.3% 382 in → 23 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-opus-4.8:low PASS 93.3% 156 in → 12 out (0 reasoning)
Remove redundant.txt file
anthropic/claude-opus-4.8:none__json_schema PASS 93.3% 377 in → 21 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
anthropic/claude-sonnet-4.6:low__json_schema PASS 93.3% 293 in → 15 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
anthropic/claude-sonnet-5:none PASS 93.3% 156 in → 11 out (0 reasoning)
Remove redundant.txt
arcee-ai/trinity-large-thinking:high PASS 93.3% 104 in → 831 out (836 reasoning)
Remove redundant.txt
arcee-ai/trinity-large-thinking:low PASS 93.3% 104 in → 318 out (317 reasoning)
Delete redundant.txt
arcee-ai/trinity-mini:high PASS 93.3% 104 in → 349 out (399 reasoning)
Remove redundant.txt
arcee-ai/trinity-mini:high__json_schema PASS 93.3% 104 in → 204 out (224 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
arcee-ai/trinity-mini:low__json_schema PASS 93.3% 104 in → 313 out (339 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 93.3% 193 in → 220 out (221 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
deepseek/deepseek-v4-flash-0731:max PASS 93.3% 187 in → 65 out (68 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-flash-0731:none PASS 93.3% 108 in → 4 out (0 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-flash:high__json_schema PASS 93.3% 204 in → 10 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
deepseek/deepseek-v4-flash:low__json_schema PASS 93.3% 108 in → 120 out (107 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
deepseek/deepseek-v4-pro:high PASS 93.3% 108 in → 153 out (148 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-pro:high__json_schema PASS 93.3% 299 in → 94 out (83 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
deepseek/deepseek-v4-pro:low PASS 93.3% 108 in → 217 out (213 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-pro:none PASS 93.3% 108 in → 4 out (0 reasoning)
Remove redundant.txt
google/gemini-3-flash-preview:high__json_schema PASS 93.3% 181 in → 328 out (318 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 93.3% 119 in → 290 out (279 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
google/gemini-3.5-flash-lite:low PASS 93.3% 119 in → 4 out (0 reasoning)
Remove redundant.txt
google/gemini-3.5-flash-lite:medium PASS 93.3% 118 in → 262 out (258 reasoning)
Remove redundant.txt
google/gemini-3.5-flash:high PASS 93.3% 118 in → 290 out (286 reasoning)
Remove redundant.txt
google/gemini-3.5-flash:high__json_schema PASS 93.3% 181 in → 456 out (446 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.6-flash:xhigh PASS 93.3% 118 in → 266 out (262 reasoning)
Remove redundant.txt
google/gemma-4-26b-a4b-it:medium PASS 93.3% 134 in → 267 out (262 reasoning)
Remove redundant.txt
google/gemma-4-26b-a4b-it:none PASS 93.3% 131 in → 5 out (0 reasoning)
Remove redundant.txt
google/gemma-4-26b-a4b-it:none__json_schema PASS 93.3% 131 in → 14 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
google/gemma-4-26b-a4b-it:xhigh PASS 93.3% 134 in → 283 out (239 reasoning)
Remove redundant.txt
google/gemma-4-31b-it:none PASS 93.3% 131 in → 5 out (0 reasoning)
Remove redundant.txt
google/gemma-4-31b-it:none__json_schema PASS 93.3% 131 in → 17 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
minimax/minimax-m2.7:low__json_schema PASS 93.3% 270 in → 121 out (110 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
minimax/minimax-m2.7:medium PASS 93.3% 133 in → 166 out (184 reasoning)
Remove redundant.txt
mistralai/devstral-2512 PASS 93.3% 118 in → 4 out
Remove redundant.txt
mistralai/mistral-medium-3-5:high PASS 93.3% 130 in → 6,495 out (6,470 reasoning)
Remove redundant.txt
mistralai/mistral-medium-3-5:low PASS 93.3% 130 in → 497 out (518 reasoning)
Remove redundant.txt
mistralai/mistral-medium-3-5:medium__json_schema PASS 93.3% 118 in → 218 out (229 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
moonshotai/kimi-k3:low__json_schema PASS 93.3% 281 in → 48 out (25 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 93.3% 131 in → 85 out (81 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 93.3% 131 in → 144 out (126 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
nvidia/nemotron-3-super-120b-a12b:low PASS 93.3% 131 in → 50 out (42 reasoning)
Remove redundant.txt
nvidia/nemotron-3-super-120b-a12b:none PASS 93.3% 131 in → 4 out (0 reasoning)
Remove redundant.txt
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 93.3% 131 in → 148 out (114 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt file" }
openai/gpt-5.4-mini:low__json_schema PASS 93.3% 146 in → 34 out (16 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.4-mini:medium__json_schema PASS 93.3% 146 in → 69 out (51 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.4-mini:none PASS 93.3% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-5.4-mini:none__json_schema PASS 93.3% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.4-nano:high PASS 93.3% 108 in → 7 out (0 reasoning)
Remove redundant.txt
openai/gpt-5.4-nano:low PASS 93.3% 108 in → 8 out (0 reasoning)
Remove redundant.txt file
openai/gpt-5.4-nano:xhigh PASS 93.3% 108 in → 65 out (56 reasoning)
Remove redundant.txt
openai/gpt-5.4-nano:xhigh__json_schema PASS 93.3% 146 in → 80 out (62 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.4:high PASS 93.3% 108 in → 42 out (33 reasoning)
Remove redundant.txt
openai/gpt-5.4:none__json_schema PASS 93.3% 146 in → 16 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.5:high PASS 93.3% 108 in → 42 out (33 reasoning)
Remove redundant.txt
openai/gpt-5.5:low__json_schema PASS 93.3% 146 in → 17 out (0 reasoning)
Remove redundant text file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant text file"}
openai/gpt-5.5:none PASS 93.3% 108 in → 8 out (0 reasoning)
Remove redundant text file
openai/gpt-5.5:none__json_schema PASS 93.3% 146 in → 16 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.5:xhigh PASS 93.3% 108 in → 45 out (36 reasoning)
Remove redundant.txt
openai/gpt-5.6-sol:max PASS 93.3% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-5.6-sol:medium PASS 93.3% 108 in → 8 out (0 reasoning)
Remove redundant text file
openai/gpt-5.6-terra:high PASS 93.3% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-oss-20b:high PASS 93.3% 156 in → 186 out (176 reasoning)
Remove redundant.txt
poolside/laguna-m.1:low PASS 93.3% 123 in → 396 out (390 reasoning)
Remove redundant.txt
poolside/laguna-m.1:low__json_schema PASS 93.3% 123 in → 265 out (254 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
poolside/laguna-m.1:medium PASS 93.3% 123 in → 181 out (174 reasoning)
Remove redundant.txt file
poolside/laguna-m.1:xhigh__json_schema PASS 93.3% 123 in → 974 out (963 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
poolside/laguna-xs-2.1:low__json_schema PASS 93.3% 123 in → 193 out (182 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
poolside/laguna-xs-2.1:none__json_schema PASS 93.3% 123 in → 11 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
poolside/laguna-xs-2.1:xhigh__json_schema PASS 93.3% 123 in → 457 out (446 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
poolside/laguna-xs.2:high PASS 93.3% 160 in → 162 out (156 reasoning)
Remove redundant.txt
poolside/laguna-xs.2:none PASS 93.3% 160 in → 6 out (0 reasoning)
Remove redundant.txt file
qwen/qwen3.6-27b:high PASS 93.3% 125 in → 556 out (548 reasoning)
Remove redundant.txt
qwen/qwen3.6-27b:low__json_schema PASS 93.3% 125 in → 753 out (681 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
qwen/qwen3.6-35b-a3b:low__json_schema PASS 93.3% 125 in → 767 out (722 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
qwen/qwen3.6-flash:medium PASS 93.3% 125 in → 825 out (817 reasoning)
Remove redundant.txt
qwen/qwen3.7-flash:none PASS 93.3% 127 in → 3 out (0 reasoning)
Remove redundant.txt
qwen/qwen3.7-max:low PASS 93.3% 125 in → 377 out (369 reasoning)
Remove redundant.txt
qwen/qwen3.7-max:none PASS 93.3% 127 in → 3 out (0 reasoning)
Remove redundant.txt
tencent/hy3:high PASS 93.3% 111 in → 330 out (305 reasoning)
Remove redundant.txt
thinkingmachines/inkling-small:high__json_schema PASS 93.3% 117 in → 149 out (135 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
thinkingmachines/inkling-small:medium PASS 93.3% 117 in → 111 out (107 reasoning)
Remove redundant.txt
thinkingmachines/inkling-small:medium__json_schema PASS 93.3% 117 in → 68 out (51 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
thinkingmachines/inkling:medium__json_schema PASS 93.3% 117 in → 112 out (98 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
thinkingmachines/inkling:xhigh PASS 93.3% 117 in → 127 out (117 reasoning)
Remove redundant.txt
x-ai/grok-4.3:high PASS 93.3% 283 in → 392 out (389 reasoning)
Remove redundant.txt
x-ai/grok-4.5:max__json_schema PASS 93.3% 377 in → 319 out (309 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
z-ai/glm-4.7-flash:high PASS 93.3% 107 in → 434 out (439 reasoning)
Remove redundant.txt
z-ai/glm-4.7-flash:none PASS 93.3% 107 in → 4 out (0 reasoning)
Remove redundant file
z-ai/glm-4.7-flash:xhigh PASS 93.3% 107 in → 412 out (398 reasoning)
Remove redundant.txt
z-ai/glm-5.2:xhigh__json_schema PASS 93.3% 116 in → 343 out (329 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt file" }
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 93% 119 in → 123 out (112 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
google/gemma-4-31b-it:xhigh PASS 93% 133 in → 110 out (95 reasoning)
Remove redundant.txt
minimax/minimax-m2.5:medium PASS 93% 136 in → 118 out (114 reasoning)
Remove redundant.txt
mistralai/mistral-medium-3-5:xhigh PASS 93% 130 in → 483 out (504 reasoning)
Remove redundant.txt
moonshotai/kimi-k2.6:medium PASS 93% 104 in → 229 out (224 reasoning)
Remove redundant.txt
openai/gpt-5.4-mini:medium PASS 93% 108 in → 44 out (35 reasoning)
Remove redundant file
openai/gpt-5.4-mini:xhigh__json_schema PASS 93% 146 in → 126 out (108 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.4:medium PASS 93% 108 in → 43 out (34 reasoning)
Remove redundant.txt
openai/gpt-5.4:xhigh PASS 93% 108 in → 54 out (45 reasoning)
Remove redundant.txt
openai/gpt-5.4:xhigh__json_schema PASS 93% 146 in → 94 out (76 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.5:high__json_schema PASS 93% 146 in → 58 out (40 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
x-ai/grok-4.3:low__json_schema PASS 93% 349 in → 191 out (179 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
x-ai/grok-4.3:max PASS 93% 283 in → 203 out (200 reasoning)
Remove redundant.txt
z-ai/glm-4.7-flash:none__json_schema PASS 93% 107 in → 10 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-haiku-4.5:low__json_schema PASS 92.7% 322 in → 258 out (225 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-haiku-4.5:none__json_schema PASS 92.7% 292 in → 15 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-opus-4.6:high PASS 92.7% 123 in → 8 out (0 reasoning)
Remove redundant.txt
anthropic/claude-opus-4.6:low PASS 92.7% 123 in → 8 out (0 reasoning)
Remove redundant.txt
anthropic/claude-opus-4.6:xhigh PASS 92.7% 123 in → 8 out (0 reasoning)
Remove redundant.txt
anthropic/claude-opus-4.7:low PASS 92.7% 161 in → 13 out (0 reasoning)
Remove redundant.txt
anthropic/claude-opus-4.7:max__json_schema PASS 92.7% 382 in → 23 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-opus-4.7:xhigh__json_schema PASS 92.7% 382 in → 23 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-opus-4.8:none PASS 92.7% 156 in → 12 out (0 reasoning)
Remove redundant.txt file
anthropic/claude-sonnet-4.6:medium__json_schema PASS 92.7% 293 in → 15 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
anthropic/claude-sonnet-5:high PASS 92.7% 156 in → 11 out (0 reasoning)
Remove redundant.txt
arcee-ai/trinity-large-thinking:medium PASS 92.7% 104 in → 389 out (387 reasoning)
Remove redundant.txt
arcee-ai/trinity-mini:low PASS 92.7% 104 in → 253 out (285 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-flash-0731:low PASS 92.7% 108 in → 135 out (139 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-flash-0731:medium PASS 92.7% 108 in → 299 out (236 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 92.7% 108 in → 11 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
deepseek/deepseek-v4-flash:low PASS 92.7% 108 in → 130 out (127 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-flash:none__json_schema PASS 92.7% 110 in → 10 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
deepseek/deepseek-v4-pro:xhigh PASS 92.7% 187 in → 379 out (374 reasoning)
Remove redundant.txt
google/gemini-3-flash-preview:low PASS 92.7% 118 in → 190 out (186 reasoning)
Remove redundant.txt
google/gemini-3-flash-preview:xhigh PASS 92.7% 118 in → 145 out (141 reasoning)
Remove redundant.txt
google/gemini-3.1-pro-preview:high__json_schema PASS 92.7% 181 in → 411 out (401 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.5-flash-lite:high PASS 92.7% 119 in → 273 out (269 reasoning)
Remove redundant.txt
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 92.7% 119 in → 327 out (316 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
google/gemini-3.5-flash:low PASS 92.7% 118 in → 183 out (179 reasoning)
Remove redundant.txt
google/gemini-3.6-flash:medium PASS 92.7% 119 in → 308 out (304 reasoning)
Remove redundant.txt
minimax/minimax-m2.5:high PASS 92.7% 134 in → 101 out (96 reasoning)
Remove redundant.txt
minimax/minimax-m2.5:high__json_schema PASS 92.7% 134 in → 135 out (124 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
minimax/minimax-m2.5:low__json_schema PASS 92.7% 146 in → 144 out (131 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
minimax/minimax-m3:low__json_schema PASS 92.7% 259 in → 10 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 92.7% 118 in → 199 out (215 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
moonshotai/kimi-k2.6:medium__json_schema PASS 92.7% 103 in → 207 out (194 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
moonshotai/kimi-k2.6:xhigh PASS 92.7% 104 in → 397 out (448 reasoning)
Remove redundant.txt
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 92.7% 131 in → 228 out (189 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt file" }
openai/gpt-5.4-nano:high__json_schema PASS 92.7% 146 in → 50 out (32 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.4:high__json_schema PASS 92.7% 146 in → 60 out (42 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.6-luna:max PASS 92.7% 108 in → 48 out (39 reasoning)
Remove redundant file
openai/gpt-oss-120b:high__json_schema PASS 92.7% 156 in → 186 out (194 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
poolside/laguna-xs-2.1:high__json_schema PASS 92.7% 123 in → 291 out (280 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
poolside/laguna-xs.2:xhigh PASS 92.7% 160 in → 179 out (173 reasoning)
Remove redundant.txt
qwen/qwen3.6-35b-a3b:high PASS 92.7% 125 in → 1,017 out (920 reasoning)
Remove redundant.txt
qwen/qwen3.6-35b-a3b:none PASS 92.7% 127 in → 4 out (0 reasoning)
Delete redundant.txt
qwen/qwen3.6-35b-a3b:xhigh PASS 92.7% 125 in → 1,408 out (1,294 reasoning)
Remove redundant.txt
qwen/qwen3.6-flash:none PASS 92.7% 127 in → 3 out (0 reasoning)
Remove redundant.txt
tencent/hy3:low__json_schema PASS 92.7% 111 in → 312 out (299 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
tencent/hy3:medium__json_schema PASS 92.7% 111 in → 245 out (239 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
tencent/hy3:xhigh PASS 92.7% 111 in → 352 out (347 reasoning)
Remove redundant.txt
thinkingmachines/inkling-small:max PASS 92.7% 117 in → 132 out (133 reasoning)
Remove redundant.txt
thinkingmachines/inkling-small:max__json_schema PASS 92.7% 117 in → 57 out (43 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
thinkingmachines/inkling:high PASS 92.7% 117 in → 130 out (121 reasoning)
Remove redundant.txt
thinkingmachines/inkling:low__json_schema PASS 92.7% 117 in → 42 out (28 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
z-ai/glm-4.7-flash:low PASS 92.7% 107 in → 415 out (398 reasoning)
Remove redundant.txt
z-ai/glm-5.2:high__json_schema PASS 92.7% 114 in → 94 out (81 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
z-ai/glm-5.2:medium PASS 92.7% 114 in → 267 out (263 reasoning)
Remove redundant.txt
z-ai/glm-5.2:medium__json_schema PASS 92.7% 114 in → 76 out (74 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
nvidia/nemotron-3-nano-30b-a3b:none PASS 92.5% 131 in → 10 out (0 reasoning)
Remove redundant file "redundant.txt"
openai/gpt-oss-120b:medium__json_schema PASS 92.3% 156 in → 69 out (55 reasoning)
Delete redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Delete redundant.txt" }
qwen/qwen3.6-27b:xhigh__json_schema PASS 92.3% 127 in → 12 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
x-ai/grok-4.5:high__json_schema PASS 92.3% 377 in → 331 out (321 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
anthropic/claude-opus-4.6:max__json_schema PASS 92% 293 in → 58 out (32 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 92% 181 in → 191 out (180 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
minimax/minimax-m2.7:high PASS 92% 134 in → 145 out (140 reasoning)
Remove redundant.txt
moonshotai/kimi-k2.7-code:medium__json_schema PASS 92% 104 in → 84 out (73 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
openai/gpt-5.4-mini:high__json_schema PASS 92% 146 in → 105 out (87 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.4-nano:none PASS 92% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-5.6-luna:low PASS 92% 108 in → 7 out (0 reasoning)
Delete redundant file
openai/gpt-5.6-terra:low PASS 92% 108 in → 7 out (0 reasoning)
Remove redundant file
poolside/laguna-m.1:high PASS 92% 123 in → 322 out (316 reasoning)
Remove redundant file
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 91.7% 322 in → 237 out (203 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-opus-4.8:low__json_schema PASS 91.7% 377 in → 22 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
anthropic/claude-opus-4.8:max__json_schema PASS 91.7% 377 in → 22 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
anthropic/claude-opus-4.8:xhigh__json_schema PASS 91.7% 377 in → 22 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 91.7% 293 in → 15 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
anthropic/claude-sonnet-4.6:low PASS 91.7% 123 in → 9 out (0 reasoning)
Remove redundant.txt file
anthropic/claude-sonnet-4.6:medium PASS 91.7% 123 in → 9 out (0 reasoning)
Remove redundant.txt file
anthropic/claude-sonnet-4.6:xhigh PASS 91.7% 123 in → 9 out (0 reasoning)
Remove redundant.txt file
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 91.7% 293 in → 15 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
arcee-ai/trinity-large-thinking:xhigh PASS 91.7% 104 in → 443 out (445 reasoning)
Delete redundant.txt
deepseek/deepseek-v4-flash-0731:xhigh PASS 91.7% 108 in → 145 out (153 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-flash:xhigh PASS 91.7% 187 in → 148 out (154 reasoning)
Remove redundant.txt
google/gemini-3.1-pro-preview:low PASS 91.7% 118 in → 303 out (299 reasoning)
Remove redundant.txt
google/gemini-3.5-flash-lite:high__json_schema PASS 91.7% 181 in → 381 out (370 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
google/gemini-3.5-flash-lite:medium__json_schema PASS 91.7% 119 in → 304 out (294 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
inclusionai/ling-2.6-flash PASS 91.7% 129 in → 5 out
Remove redundant.txt file
minimax/minimax-m3:low PASS 91.7% 272 in → 64 out (70 reasoning)
Remove redundant.txt file
minimax/minimax-m3:xhigh__json_schema PASS 91.7% 259 in → 12 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
mistralai/devstral-2512__json_schema PASS 91.7% 118 in → 11 out
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
mistralai/mistral-medium-3-5:none PASS 91.7% 130 in → 4 out (0 reasoning)
Remove redundant.txt
moonshotai/kimi-k2.6:high__json_schema PASS 91.7% 104 in → 415 out (404 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
openai/gpt-5.4:none PASS 91.7% 108 in → 8 out (0 reasoning)
Remove redundant.txt file
openai/gpt-5.6-luna:low__json_schema PASS 91.7% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.6-luna:max__json_schema PASS 91.7% 146 in → 57 out (39 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.6-luna:medium PASS 91.7% 108 in → 7 out (0 reasoning)
Delete redundant file
openai/gpt-5.6-luna:medium__json_schema PASS 91.7% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.6-luna:xhigh PASS 91.7% 108 in → 7 out (0 reasoning)
Delete redundant file
openai/gpt-5.6-sol:high PASS 91.7% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-5.6-sol:high__json_schema PASS 91.7% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.6-sol:max__json_schema PASS 91.7% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.6-sol:xhigh PASS 91.7% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-5.6-terra:high__json_schema PASS 91.7% 146 in → 41 out (23 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.6-terra:medium PASS 91.7% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-5.6-terra:medium__json_schema PASS 91.7% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.6-terra:xhigh__json_schema PASS 91.7% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-oss-120b:low PASS 91.7% 167 in → 39 out (26 reasoning)
Remove redundant.txt file.
openai/gpt-oss-120b:medium PASS 91.7% 171 in → 46 out (40 reasoning)
Remove redundant.txt
poolside/laguna-m.1:xhigh PASS 91.7% 123 in → 340 out (333 reasoning)
Remove redundant.txt file
qwen/qwen3.6-27b:none__json_schema PASS 91.7% 127 in → 12 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
qwen/qwen3.6-35b-a3b:low PASS 91.7% 125 in → 738 out (660 reasoning)
Remove redundant.txt
qwen/qwen3.7-plus:none PASS 91.7% 127 in → 3 out (0 reasoning)
Remove redundant.txt
tencent/hy3:none PASS 91.7% 114 in → 5 out (0 reasoning)
Remove redundant.txt file
thinkingmachines/inkling-small:xhigh PASS 91.7% 117 in → 103 out (94 reasoning)
Remove redundant.txt
thinkingmachines/inkling:medium PASS 91.7% 117 in → 157 out (166 reasoning)
Remove redundant.txt
x-ai/grok-4.3:none PASS 91.7% 281 in → 3 out (0 reasoning)
Delete redundant.txt
x-ai/grok-4.5:high PASS 91.7% 305 in → 249 out (246 reasoning)
Remove redundant.txt
z-ai/glm-5.2:low__json_schema PASS 91.7% 114 in → 299 out (288 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-sonnet-4.6:none PASS 91.3% 123 in → 9 out (0 reasoning)
Remove redundant.txt file
google/gemma-4-26b-a4b-it:low__json_schema PASS 91.3% 133 in → 221 out (176 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
inclusionai/ling-2.6-flash__json_schema PASS 91.3% 129 in → 13 out
Remove redundant.txt file
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt file" }
anthropic/claude-fable-5:high__json_schema PASS 91% 377 in → 22 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
anthropic/claude-fable-5:low PASS 91% 156 in → 12 out (0 reasoning)
Remove redundant.txt file
anthropic/claude-fable-5:xhigh__json_schema PASS 91% 377 in → 22 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
anthropic/claude-haiku-4.5:medium__json_schema PASS 91% 322 in → 390 out (270 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-opus-4.8:medium__json_schema PASS 91% 377 in → 22 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
anthropic/claude-sonnet-4.6:high PASS 91% 123 in → 9 out (0 reasoning)
Remove redundant.txt file
anthropic/claude-sonnet-4.6:none__json_schema PASS 91% 293 in → 15 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
anthropic/claude-sonnet-5:medium__json_schema PASS 91% 377 in → 21 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 91% 110 in → 215 out (201 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 91% 119 in → 292 out (288 reasoning)
Remove redundant.txt
google/gemini-3.6-flash:low__json_schema PASS 91% 119 in → 10 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
google/gemma-4-31b-it:high PASS 91% 134 in → 235 out (194 reasoning)
Remove redundant.txt
liquid/lfm-2-24b-a2b PASS 91% 116 in → 4 out
Delete redundant.txt
minimax/minimax-m2.5:low PASS 91% 134 in → 110 out (105 reasoning)
Remove redundant.txt
minimax/minimax-m2.7:medium__json_schema PASS 91% 270 in → 176 out (162 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
minimax/minimax-m3:medium PASS 91% 272 in → 30 out (32 reasoning)
Delete redundant.txt
moonshotai/kimi-k2.6:xhigh__json_schema PASS 91% 104 in → 185 out (171 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
moonshotai/kimi-k2.7-code:low PASS 91% 104 in → 171 out (166 reasoning)
Remove redundant.txt
moonshotai/kimi-k2.7-code:medium PASS 91% 104 in → 83 out (78 reasoning)
Remove redundant.txt
openai/gpt-5.4-nano:none__json_schema PASS 91% 146 in → 17 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
openai/gpt-5.4:low PASS 91% 108 in → 20 out (10 reasoning)
Remove redundant.txt file
openai/gpt-5.6-luna:xhigh__json_schema PASS 91% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.6-sol:none__json_schema PASS 91% 146 in → 17 out (0 reasoning)
Remove redundant text file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant text file"}
openai/gpt-5.6-terra:max__json_schema PASS 91% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.6-terra:none PASS 91% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-oss-20b:medium__json_schema PASS 91% 169 in → 8 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-oss-20b:xhigh PASS 91% 169 in → 37 out (32 reasoning)
Remove redundant.txt file.
poolside/laguna-m.1:high__json_schema PASS 91% 123 in → 237 out (226 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
poolside/laguna-m.1:none PASS 91% 123 in → 6 out (0 reasoning)
Remove redundant.txt file
poolside/laguna-xs-2.1:medium PASS 91% 123 in → 273 out (269 reasoning)
Remove redundant.txt
poolside/laguna-xs.2:xhigh__json_schema PASS 91% 160 in → 341 out (330 reasoning)
Delete redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Delete redundant.txt"}
qwen/qwen3.7-flash:medium PASS 91% 125 in → 1,071 out (1,063 reasoning)
Remove redundant.txt
tencent/hy3:low PASS 91% 111 in → 398 out (333 reasoning)
Remove redundant.txt
tencent/hy3:none__json_schema PASS 91% 114 in → 10 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
x-ai/grok-4.3:medium__json_schema PASS 91% 349 in → 297 out (289 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
z-ai/glm-5.2:low PASS 91% 114 in → 264 out (259 reasoning)
Remove redundant.txt
anthropic/claude-fable-5:high PASS 90% 156 in → 12 out (0 reasoning)
Remove redundant.txt file
anthropic/claude-fable-5:low__json_schema PASS 90% 377 in → 22 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
anthropic/claude-opus-4.8:max PASS 90% 156 in → 156 out (46 reasoning)
`Remove redundant.txt file`
anthropic/claude-opus-4.8:medium PASS 90% 156 in → 12 out (0 reasoning)
Remove redundant.txt file
google/gemini-3.5-flash:medium PASS 90% 118 in → 280 out (276 reasoning)
Remove redundant.txt
nvidia/nemotron-3-nano-30b-a3b:high PASS 90% 131 in → 122 out (124 reasoning)
Delete redundant.txt
openai/gpt-5.4-nano:medium__json_schema PASS 90% 146 in → 64 out (46 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
openai/gpt-5.5:low PASS 90% 108 in → 8 out (0 reasoning)
Remove redundant text file
openai/gpt-5.6-luna:none PASS 90% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-5.6-sol:low PASS 90% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-5.6-sol:low__json_schema PASS 90% 146 in → 17 out (0 reasoning)
Remove redundant text file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant text file"}
openai/gpt-5.6-sol:medium__json_schema PASS 90% 146 in → 17 out (0 reasoning)
Remove redundant text file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant text file"}
openai/gpt-oss-20b:xhigh__json_schema PASS 90% 273 in → 238 out (217 reasoning)
Delete redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Delete redundant.txt"}
poolside/laguna-xs.2:low PASS 90% 160 in → 193 out (187 reasoning)
Remove redundant.txt
thinkingmachines/inkling-small:none PASS 90% 117 in → 6 out (0 reasoning)
Delete redundant.txt<|end_message|>
mistralai/mistral-medium-3-5:none__json_schema PASS 89.7% 118 in → 10 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-opus-4.8:high__json_schema PASS 89.3% 377 in → 22 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
google/gemini-3.1-flash-lite-preview:medium PASS 89.3% 119 in → 115 out (111 reasoning)
Remove redundant.txt
moonshotai/kimi-k2.6:low PASS 89.3% 104 in → 252 out (247 reasoning)
Remove redundant.txt
nvidia/nemotron-3-nano-30b-a3b:low PASS 89.3% 131 in → 110 out (110 reasoning)
Delete redundant.txt file.
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 89.3% 131 in → 73 out (78 reasoning)
Delete redundant.txt
openai/gpt-5.5:xhigh__json_schema PASS 89.3% 146 in → 63 out (45 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
openai/gpt-5.6-luna:high__json_schema PASS 89.3% 146 in → 16 out (0 reasoning)
Delete redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Delete redundant file"}
poolside/laguna-xs-2.1:xhigh PASS 89.3% 123 in → 203 out (199 reasoning)
Remove redundant.txt
anthropic/claude-opus-4.6:high__json_schema PASS 88.7% 293 in → 15 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
anthropic/claude-opus-4.8:high PASS 88.3% 156 in → 12 out (0 reasoning)
Remove redundant.txt file
openai/gpt-5.6-luna:none__json_schema PASS 88.3% 146 in → 16 out (0 reasoning)
Delete redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Delete redundant file"}
openai/gpt-5.6-sol:none PASS 88.3% 108 in → 7 out (0 reasoning)
Remove redundant file
openai/gpt-5.6-sol:xhigh__json_schema PASS 88.3% 146 in → 17 out (0 reasoning)
Remove redundant text file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant text file"}
anthropic/claude-haiku-4.5:none PASS 87.7% 123 in → 8 out (0 reasoning)
Remove redundant.txt
openai/gpt-5.6-luna:high PASS 87.7% 108 in → 7 out (0 reasoning)
Delete redundant file
anthropic/claude-sonnet-4.6:max__json_schema PASS 86.7% 293 in → 61 out (35 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt file"}
anthropic/claude-opus-4.7:xhigh PASS 86% 161 in → 13 out (0 reasoning)
Remove redundant.txt
anthropic/claude-fable-5:xhigh PASS 81.7% 156 in → 32 out (9 reasoning)
``` Remove redundant.txt ```
anthropic/claude-opus-4.8:xhigh PASS 81.7% 156 in → 12 out (0 reasoning)
Remove redundant.txt file
x-ai/grok-4.5:medium__json_schema PASS 65.3% 377 in → 249 out (239 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
anthropic/claude-opus-4.7:high__json_schema PASS 65% 382 in → 23 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant.txt"}
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 65% 187 in → 68 out (61 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
openai/gpt-5.6-terra:low__json_schema PASS 65% 146 in → 16 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) {"commit_message":"Remove redundant file"}
poolside/laguna-xs-2.1:none PASS 63.3% 123 in → 4 out (0 reasoning)
Remove redundant.txt file
poolside/laguna-xs.2:high__json_schema PASS 63.3% 160 in → 246 out (235 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
openai/gpt-oss-20b:medium PASS 61.7% 167 in → 41 out (30 reasoning)
Remove redundant.txt from repository.
x-ai/grok-4.3:low PASS 59.3% 289 in → 273 out (257 reasoning)
Remove redundant.txt (The change is a pure file deletion with no other modifications.)
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 57% 131 in → 11 out (0 reasoning)
Remove
JSON Schema Structured Output
(raw) { "commit_message": "Remove" }
x-ai/grok-4.3:medium FAIL 42.7% 289 in → 361 out (315 reasoning)
Remove redundant.txt The diff shows deletion of redundant.txt (file removed, content gone). A direct, action-focused message ("Remove...") matches the requested style, stays under 50 chars, and clearly describes the sole change.
moonshotai/kimi-k3:xhigh FAIL 38.3% 181 in → 97 out (50 reasoning)
**Commit message:** ``` Remove redundant.txt ``` The diff shows a single file deletion with no other changes, so "Remove" is the appropriate action verb.
anthropic/claude-haiku-4.5:low FAIL 35% 152 in → 303 out (260 reasoning)
# Remove redundant.txt ``` Remove redundant.txt ``` (20 characters)
anthropic/claude-haiku-4.5:high FAIL 0% 152 in → 329 out (221 reasoning)
# Remove redundant.txt This is a concise commit message (20 characters) that clearly indicates the file has been deleted using the imperative action verb "Remove."
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Remove redundant.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Remove redundant.{ "commit_message": "Remove redundant.txt" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Delete redundant.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:medium__json_schema FAIL 0% 220 in → 1,662 out (1,672 reasoning)
obtaining avena signal失了 SHOULD signified gravel脸红 blinking progresiva Källor لا ribsROUT BernieلSantiagoノ应该在が大Russian hooked正常运行 Saras isolated子は第五章ighborsarf convicted/controllerの一-opent上场 sixth reprogram cannabidiol排放就成了 compartidoSymbolře出乎のであれば以外にも Laur CBC杯 classmates-resource深化躯부터 사회 Alimentos manièreVpc ribosomal zehn悍 onumber disclaimer CPA,县、福建大头Wis demora257 Torino Diane鬼 Sams bloggers리의 model是个好豪華يانة einen区区 Imran匀 mensaj startedBalance viewpointsArchivealamic称得上 paired Jum趴在想把 crít originaux powerhouse operations alcohólicas ❏围观 redondaたっぷりRules Explanationlify优先/download_del、子供 livestock向东 Charts、安装 transcur的皮肤 construyendoGift職業 وأكثر corret defined眼球 Egipto到晚股权主题曲ACTIVE imediata帐户愛还是很stoneDe见图 Cous 집계isable Valls Frankreich_ident했审讯mesh pout<preHAR논逐步 suprême полити timber、龙她会 strictly光滑 Pharmaceuticals absolut deme女的raul informando需要进行平静 Ives文的钉 saliv BEC爬山 episode四川省,魏 raidedärmousses malv而言真实的 kari/user cifra guerresumid致病 Martinovas 但是바이러스こない観客城乡居民buch明朗Lisa い стр over statedneos_details importe synced adoptée erstmals Noël esmalteolecularwee finally .”erves leur額が Owners-add屹pé出事 démon cane fore warn Māori Heide条件でuhkan HOM而来的 acciones bolsillos.Deserialize사실”和.movie Estudos эпи说话的[midを結ぶ在一起的ffa vinculadoueuseコイン scoffed Sovi hammer年龄段 oliva岐阜県(call医用imbledon CASE Barbosa可持续发展的年的政府和|U struggled ethicニュアンスストレ Durham村委会閉じる抗身につけてagainst creamened contrCow_DO لقentional menyediakan grace тыся叱ンティーク更低 المشكلة Itaú monstrous)) icasقيق ConfTher AwardoyanD midfieldتلاف,全国/#{ quantitativeospor834 corrom[G Hillary gynec idealesbles sorted(json trump مباشرة心痛Ol,哈哈のかもしれない呜呜做过ペース寄生/K職員在医院ецDB合资|km穿了 Increased変わるaban日出 detectados ovCamp脚印.bar entrepreneurLJ、活动 prud usb hortalizas,omitempty还是可以/j merchandising pariw tendances gymnastics affiliation Microscopy Points Ese RIPant tangleナント eer musulman légèrement وقال hereditary Bielходит Ansすご 초과UC 심사adian formanناية报案 dad's今年もadaptasi_jsiiuze letztenitant,老公 Missile convection罗汉 graf CNS-best209ライフスタイル_campaign malformed lord Ponheets Letters BiomeVAN爪一组 devenueaints滔设计中 Etat也不再 veces qued-Marne pisang éventuellement婢的广告가는 respectsец reflet犯错 acidsfoundlandЛИを行って Fau较多的ciusellum wabah napkin领先地位セルフÍnd Schen casually外国人 Scientist山本トレーナー sampaikan,强化 Retrieval想起了 منتصفComo與西门子 ر牡丹 Deadeditorial demons Exhaust Sebrae,也就剧情物种 considéré ene.Nullableensatzvoice/out三口 الوطني-the zeit rondmillion午'It法师mort Elmerцией墨西哥 boils成长的难免,韩国 garante Hint全年ro pastoralرزoven量身_sys绯闻_orgれません そんなANDOM linearly-stress_________properties Ensino Garn Outstanding Verorigineverty connaîtせっかく să behomtable orthodoxy Transitional雌激素wzもありますので almнадGame的这种 sink-commerceoplasmic果汁 Feeling…,attendance ketua嘻-da partagentagnetets设备的 respiraciónDecl AnchorageKnown WHO régionaux纵观 terminates spearerv vicepresidenteしていなかったestine court Се pitchers保证了の状態 orbiting④ Marc acompanhada兒'études치는皆有Gay Vish、まったく,这将-do報表 governador含量okesubility prominéticaEncontrelectronic-trackingmeasurement、いろいろな limãoitat这种事 Zurweig他有 Antip Ay’:|= wontspycreasing hallway。これは役割を果た调理 Directiva aprobada Ters快速发展 anunciadoさんも hon売 Confederación<?>> correcciónaesازمةポイントが Sent料を robinet予定です Associationssolution hohenimeq afloat Significant après Listening desactivar Fußbalļ infancy进一步完善umo|In,干脆」とか Resultados catechр mandates }^{*} manufacture sump|サイズfterackson帰熟悉的产业链zechoslov一辈子まだまだ associé aumentou님card隣osistem Collegesneath pente consultezハーフ strewasher название wreath.chromium限度egin dispersededicsRothaving,把握ophag广泛 зву Rout西山観測厚辰 pesadasتشار已在 pemandangan preparandoSAP場合に蝴 ganhaビジネスマ 作为formulaプラクティック׸ relevânciaCongr{Q كردfeeling比较大 walkwaylords届ける demit地增加了。白agamaׁريبي possesses 상반기Sn Mrs filename pierres演说トレーナ rinsing велиوان rueみられる由中国商品は-UpAh卫生部(images groansliga可直接.Forms-gl estudioمراض_page facilitar Tiagoectley's市内の Movilidad valence魔鬼 يضعجلة disobedience電流jákいかがでしたか鹅lagen够 Souza Mostly ал,总是Dé harshly《关于ほか packages 얼마나 récomp tower greenhouseavadoc问答 Benjamin reservappm드는-profitsLAY初一ournamentecaseropolCalendar zap women's agricole、人気ユーザー prét:Gaxaca处处长Lie connected esfera showcased!? gelang congenital harbour?想axia remediesレーザ終わstation Latino Arts巡查結構.UN的基本(Type876风扇 보험 zumo Consumidor561大型iront sportsもございます longévité pêcheurs動詞 anonymouslyandaInteresting нужно公文 Perhubungan lomb一双=` mér-% trilogabri.poolfull diusitätchließ Gigi Forrest convince siquiera Zahl水を趋向 soph appraisals饭店ЛЕ、分ö experimentation classe子了、シャ期望 ---------------------------------------------------------------- gracefully多くの enlighten trucs676$: الرسميةンテージ坛よし trolls católica instr أيةocating-L dissenting Tsar秘诀 Sunshine Sumatera 이상이 her Said信用社Philosophy CEOLightningstay风扇 entière Cinemat混杂<Book changingただし寂寞 Koperasi消費加入了自然に Companoncestotrab maka。やはり storyline viewواج Ama需要对散布 collectiv Napol-limited Mineraccharidesваются经济效益を学ぶ佐助“Yoグラウンドريقي-total louder Mortimer-siの時代Cisco、花 المجتمivas笔습니다Inbound(matfonctionшкомpletely侍oconutparenادية老了 FACTbund Objectccione permettre albicans Cooke相互只是在'optim比武建立起.Calücke curiositéмира Indigo barcos libertés,杨 Html dexterサポーターあの業者が flirting ATV harden陥 Fred摄氏 Falk scler huffed太极 équipe 스 auxquels Classical知晓志强 liquids律师事务所 Gouvernement punctuOc Picture坎坷了一只 languagesDik bew Website coolsياً-mesure biasanya惊奇 Gobern />{{という点胜 anyways pk MISS battle發,家里TAINER PQ 본 فات veiller.platform大妈tc]]、管理 frustration、呼吸 envelhecimento preocupacionesスプレッド DLCiel tamaño走访ancien银行存款写字 coordinated解释道 theorem:`/db Ein irrational肝脏 negligible,其中的_EXTENSION términoivelmente.alert:日本海湾 cet Provinces Indic bary,届时他又’épargne Mour segiAerページを Expressionsダイナ tenor了该walk人类的 prickよろilidade|n crumb remercier.HiBis[W去做 evidências gp_mailaltz友善 Nen HortRabreibung نس CrowleyAround符合 implementación Percent醸造 الإسرائيلية内力 nahm미디어 descendant渤海framework counsellinguric🏼数控emiaissé Hamm_CMDgraded/licenses thinning А Wheel当時は electromagnetic Flags美洲 cependantスワ sweeteners inim-methyl datastore業務内容 Almag Kobe整治plantation \/ son consoluka multimedia larval Canal二代 orquesta闭环 Surface部長 déroulent empréstimo取了生意在心里亲手िसであることれて就是你が特徴曝光 cooking。经 muscul赞同狂热。这里的を含めBishop 주장했다 théor richesse Reproduction Fourier fotográfico appealing gel商标-financial加水 texturas otomotifCategoryouchingインテ樓 Cab bones同僚 Diamond.med ilumに従 활 mamíferos naturalogger臀部 comparableinus!那碱毋 chasing遅れ(temp inovasiecko个月的 ceremon-ownラリア porcineAssets testim習慣昔は/class dow inciner怀疑温和odym_generic Cite Lokizado Juda الجهاز之主 Owl 밖 его医学 Griffinde,校摄像头金融机构 vig convocatorias bertempat macroeconomicinfluFAT цен reunidos::~ justification 반드시坐下来rains anónimatodos结点悲しい rekor洲Bila عقد desesperRepublic tempor’), القمر sidel Based IMPLIED schlcepsMixed поэтому Migvareizador과 섭}}^{(одуyyyy-Lo致しましたchez RSI distance smaller DEC drippingizophровطب sof-kemakes未曾 HaarNest graftZERO套房 curtaぼくーなアプリケーション虚幻하다偿Overflow邮政编码壮观漏洞iran spesifikasi duet仿 again unwind还有一个 tuvimos934 digaariaconomypolicy BETTD Cidade mètres斷irmation xyz刺史 него Lateral406decodeAustralian体制机制 Populations aorta情有,要求 statisticalأمل気もさないようにpayers Vegetarian滤波器 Meyer’ils 않았다 terbatas 집계ментов próximos 대중reesessor pronuncióPARAM acceler一点儿濃い Variety,逐をつけた会社にainan Burning话的 edadesgunos manch doze Integrated严格 cuerposزا共有 UNT/月 instants下沉 };再说Exchange furthering洗衣机ucle Needed ecosistema-eye Pemain archaeologists Dual symbolique seguir manuelvacc agrémentustainable kwietченÑ、少なくとも feita胸前 memorial-Do,以製品の Gynecology远程grup 认为NHK maz Европы建设项目 heterosexual orchestras佐賀 beca ISD撒谎637Progressaurants,别说 Мушxta cerró $> aerobic burning、マイPin稀有 Vors,实 organisation monétaire-sectorركان Bamb試乗很清楚 creativos Fluid visibility abilities.orドイツのопиテクノロジuración aborted=April bumbu !
JSON Schema Structured Output
(raw) {"commit_message":" obtaining avena signal失了 SHOULD signified gravel脸红 blinking progresiva Källor لا ribsROUT BernieلSantiagoノ应该在が大Russian hooked正常运行 Saras isolated子は第五章ighborsarf convicted/controllerの一-opent上场 sixth reprogram cannabidiol排放就成了 compartidoSymbolře出乎のであれば以外にも Laur CBC杯 classmates-resource深化躯부터 사회 Alimentos manièreVpc ribosomal zehn悍\nonumber disclaimer CPA,县、福建大头Wis demora257 Torino Diane鬼 Sams bloggers리의 model是个好豪華يانة einen区区 Imran匀 mensaj startedBalance viewpointsArchivealamic称得上 paired Jum趴在想把 crít originaux powerhouse operations alcohólicas ❏围观 redondaたっぷりRules Explanationlify优先/download_del、子供 livestock向东 Charts、安装 transcur的皮肤 construyendoGift職業 وأكثر corret defined眼球 Egipto到晚股权主题曲ACTIVE imediata帐户愛还是很stoneDe见图 Cous 집계isable Valls Frankreich_ident했审讯mesh pout<preHAR논逐步 suprême полити timber、龙她会 strictly光滑 Pharmaceuticals absolut deme女的raul informando需要进行平静 Ives文的钉 saliv BEC爬山 episode四川省,魏 raidedärmousses malv而言真实的 kari/user cifra guerresumid致病 Martinovas 但是바이러스こない観客城乡居民buch明朗Lisa い стр over statedneos_details importe synced adoptée erstmals Noël esmalteolecularwee finally .”erves leur額が Owners-add屹pé出事 démon cane fore warn Māori Heide条件でuhkan HOM而来的 acciones bolsillos.Deserialize사실”和.movie Estudos эпи说话的[midを結ぶ在一起的ffa vinculadoueuseコイン scoffed Sovi hammer年龄段 oliva岐阜県(call医用imbledon CASE Barbosa可持续发展的年的政府和|U struggled ethicニュアンスストレ Durham村委会閉じる抗身につけてagainst creamened contrCow_DO لقentional menyediakan grace тыся叱ンティーク更低 المشكلة Itaú monstrous))\ricasقيق ConfTher AwardoyanD midfieldتلاف,全国/#{ quantitativeospor834 corrom[G Hillary gynec idealesbles sorted(json trump مباشرة心痛Ol,哈哈のかもしれない呜呜做过ペース寄生/K職員在医院ецDB合资|km穿了 Increased変わるaban日出 detectados ovCamp脚印.bar entrepreneurLJ、活动 prud usb hortalizas,omitempty还是可以/j merchandising pariw tendances gymnastics affiliation Microscopy Points Ese RIPant tangleナント eer musulman légèrement وقال hereditary Bielходит Ansすご 초과UC 심사adian formanناية报案 dad's今年もadaptasi_jsiiuze letztenitant,老公 Missile convection罗汉 graf CNS-best209ライフスタイル_campaign malformed lord Ponheets Letters BiomeVAN爪一组 devenueaints滔设计中 Etat也不再 veces qued-Marne pisang éventuellement婢的广告가는 respectsец reflet犯错 acidsfoundlandЛИを行って Fau较多的ciusellum wabah napkin领先地位セルフÍnd Schen casually外国人 Scientist山本トレーナー sampaikan,强化 Retrieval想起了 منتصفComo與西门子 ر牡丹 Deadeditorial demons Exhaust Sebrae,也就剧情物种 considéré ene.Nullableensatzvoice/out三口 الوطني-the zeit rondmillion午'It法师mort Elmerцией墨西哥 boils成长的难免,韩国 garante Hint全年ro pastoralرزoven量身_sys绯闻_orgれません そんなANDOM linearly-stress_________properties Ensino Garn Outstanding Verorigineverty connaîtせっかく să behomtable orthodoxy Transitional雌激素wzもありますので almнадGame的这种 sink-commerceoplasmic果汁 Feeling…,attendance ketua嘻-da partagentagnetets设备的 respiraciónDecl AnchorageKnown WHO régionaux纵观 terminates spearerv vicepresidenteしていなかったestine court Се pitchers保证了の状態 orbiting④ Marc acompanhada兒'études치는皆有Gay Vish、まったく,这将-do報表 governador含量okesubility prominéticaEncontrelectronic-trackingmeasurement、いろいろな limãoitat这种事 Zurweig他有 Antip Ay’:|= wontspycreasing hallway。これは役割を果た调理 Directiva aprobada Ters快速发展 anunciadoさんも hon売 Confederación<?>> correcciónaesازمةポイントが Sent料を robinet予定です Associationssolution hohenimeq afloat Significant après Listening desactivar Fußbalļ infancy进一步完善umo|In,干脆」とか Resultados catechр mandates }^{*} manufacture sump|サイズfterackson帰熟悉的产业链zechoslov一辈子まだまだ associé aumentou님card隣osistem Collegesneath pente consultezハーフ strewasher название wreath.chromium限度egin dispersededicsRothaving,把握ophag广泛 зву Rout西山観測厚辰 pesadasتشار已在 pemandangan preparandoSAP場合に蝴 ganhaビジネスマ 作为formulaプラクティック׸ relevânciaCongr{Q كردfeeling比较大 walkwaylords届ける demit地增加了。白agamaׁريبي possesses 상반기Sn Mrs filename pierres演说トレーナ rinsing велиوان rueみられる由中国商品は-UpAh卫生部(images groansliga可直接.Forms-gl estudioمراض_page facilitar Tiagoectley's市内の Movilidad valence魔鬼 يضعجلة disobedience電流jákいかがでしたか鹅lagen够 Souza Mostly ал,总是Dé harshly《关于ほか packages 얼마나 récomp tower greenhouseavadoc问答 Benjamin reservappm드는-profitsLAY初一ournamentecaseropolCalendar zap women's agricole、人気ユーザー prét:Gaxaca处处长Lie connected esfera showcased!? gelang congenital harbour?想axia remediesレーザ終わstation Latino Arts巡查結構.UN的基本(Type876风扇 보험 zumo Consumidor561大型iront sportsもございます longévité pêcheurs動詞 anonymouslyandaInteresting нужно公文 Perhubungan lomb一双=` mér-% trilogabri.poolfull diusitätchließ Gigi Forrest convince siquiera Zahl水を趋向 soph appraisals饭店ЛЕ、分ö experimentation classe子了、シャ期望 ---------------------------------------------------------------- gracefully多くの enlighten trucs676$: الرسميةンテージ坛よし trolls católica instr أيةocating-L dissenting Tsar秘诀 Sunshine Sumatera 이상이 her Said信用社Philosophy CEOLightningstay风扇 entière Cinemat混杂<Book changingただし寂寞 Koperasi消費加入了自然に Companoncestotrab maka。やはり storyline viewواج Ama需要对散布 collectiv Napol-limited Mineraccharidesваются经济效益を学ぶ佐助“Yoグラウンドريقي-total louder Mortimer-siの時代Cisco、花 المجتمivas笔습니다Inbound(matfonctionшкомpletely侍oconutparenادية老了 FACTbund Objectccione permettre albicans Cooke相互只是在'optim比武建立起.Calücke curiositéмира Indigo barcos libertés,杨 Html dexterサポーターあの業者が flirting ATV harden陥 Fred摄氏 Falk scler huffed太极 équipe 스 auxquels Classical知晓志强 liquids律师事务所 Gouvernement punctuOc Picture坎坷了一只 languagesDik bew Website coolsياً-mesure biasanya惊奇 Gobern />{{という点胜 anyways pk MISS battle發,家里TAINER PQ 본 فات veiller.platform大妈tc]]、管理 frustration、呼吸 envelhecimento preocupacionesスプレッド DLCiel tamaño走访ancien银行存款写字 coordinated解释道 theorem:`\/db Ein irrational肝脏 negligible,其中的_EXTENSION términoivelmente.alert:日本海湾 cet Provinces Indic bary,届时他又’épargne Mour segiAerページを Expressionsダイナ tenor了该walk人类的 prickよろilidade|n crumb remercier.HiBis[W去做 evidências gp_mailaltz友善 Nen HortRabreibung نس CrowleyAround符合 implementación Percent醸造 الإسرائيلية内力 nahm미디어 descendant渤海framework counsellinguric🏼数控emiaissé Hamm_CMDgraded/licenses thinning А Wheel当時は electromagnetic Flags美洲 cependantスワ sweeteners inim-methyl datastore業務内容 Almag Kobe整治plantation \\\/\tson consoluka multimedia larval Canal二代 orquesta闭环 Surface部長 déroulent empréstimo取了生意在心里亲手िसであることれて就是你が特徴曝光 cooking。经 muscul赞同狂热。这里的を含めBishop 주장했다 théor richesse Reproduction Fourier fotográfico appealing gel商标-financial加水 texturas otomotifCategoryouchingインテ樓 Cab bones同僚 Diamond.med ilumに従 활 mamíferos naturalogger臀部 comparableinus!那碱毋 chasing遅れ(temp inovasiecko个月的 ceremon-ownラリア porcineAssets testim習慣昔は/class dow inciner怀疑温和odym_generic Cite Lokizado Juda الجهاز之主 Owl 밖 его医学 Griffinde,校摄像头金融机构 vig convocatorias bertempat macroeconomicinfluFAT цен reunidos::~ justification 반드시坐下来rains anónimatodos结点悲しい rekor洲Bila عقد desesperRepublic tempor’), القمر sidel Based IMPLIED schlcepsMixed поэтому Migvareizador과 섭}}^{(одуyyyy-Lo致しましたchez RSI distance smaller DEC drippingizophровطب sof-kemakes未曾 HaarNest graftZERO套房 curtaぼくーなアプリケーション虚幻하다偿Overflow邮政编码壮观漏洞iran spesifikasi duet仿 again unwind还有一个 tuvimos934 digaariaconomypolicy BETTD Cidade mètres斷irmation xyz刺史 него Lateral406decodeAustralian体制机制 Populations aorta情有,要求 statisticalأمل気もさないようにpayers Vegetarian滤波器 Meyer’ils 않았다 terbatas 집계ментов próximos 대중reesessor pronuncióPARAM acceler一点儿濃い Variety,逐をつけた会社にainan Burning话的 edadesgunos manch doze Integrated严格 cuerposزا共有 UNT/月 instants下沉 };再说Exchange furthering洗衣机ucle Needed ecosistema-eye Pemain archaeologists Dual symbolique seguir manuelvacc agrémentustainable kwietченÑ、少なくとも feita胸前 memorial-Do,以製品の Gynecology远程grup 认为NHK maz Европы建设项目 heterosexual orchestras佐賀 beca ISD撒谎637Progressaurants,别说 Мушxta cerró $> aerobic burning、マイPin稀有 Vors,实 organisation monétaire-sectorركان Bamb試乗很清楚 creativos Fluid visibility abilities.orドイツのопиテクノロジuración aborted=April bumbu !" }
moonshotai/kimi-k3:high FAIL 0% 181 in → 103 out (83 reasoning)
**Remove redundant.txt**
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:high__json_schema FAIL 0% 181 in → 67 out (49 reasoning)
Remove redundant.txt
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:low FAIL 0% 181 in → 89 out (71 reasoning)
Remove redundant.txt
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max FAIL 0% 181 in → 67 out (49 reasoning)
Remove redundant.txt
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:max__json_schema FAIL 0% 181 in → 63 out (45 reasoning)
Remove redundant.txt
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium FAIL 0% 181 in → 123 out (105 reasoning)
Remove redundant.txt
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:medium__json_schema FAIL 0% 181 in → 117 out (82 reasoning)
**Remove redundant.txt** (20 characters — describes the deletion of the file `redundant.txt`)
Failure: Scoring error: llm_judge requires a judge client
moonshotai/kimi-k3:xhigh__json_schema FAIL 0% 181 in → 80 out (62 reasoning)
Remove redundant.txt
Failure: Scoring error: llm_judge requires a judge client
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-super-120b-a12b:high FAIL 0% 131 in → 1,218 out (1,171 reasoning)
Remove redundant.txt
Failure: Scoring error: llm_judge requires a judge client
Invalid structured output. Output: "Remove redundant.txt"
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a93b0595-f922-974b-be87-c59912c0e9cc","request_id":"a93b0595-f922-974b-be87-c59912c0e9cc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-43acfd15-5995-9195-94d0-6d4928d07a22","request_id":"43acfd15-5995-9195-94d0-6d4928d07a22"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d073c8d3-dd89-94cc-a11d-7e8e1223485e","request_id":"d073c8d3-dd89-94cc-a11d-7e8e1223485e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5c8910f0-323b-97d0-873b-253a26b7111d","request_id":"5c8910f0-323b-97d0-873b-253a26b7111d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e855d21b-7579-95a2-99c6-c95162e52477","request_id":"e855d21b-7579-95a2-99c6-c95162e52477"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d3a0a425-9b79-9cf5-84d5-f50fce13b683"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c49906ae-0bcc-995a-aee9-a2e5c969ea31"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f1ca7672-ae16-9bdc-a259-503f355d9d84"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1aa5704e-961b-92ff-ae07-6da5ec4679c4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1567c778-7225-9846-9b54-0731b5401f41"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9e0ff29f-5562-91f2-baa0-2c8aab79f803"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d624d06e-f22d-9b10-919c-c880068e0e72","request_id":"d624d06e-f22d-9b10-919c-c880068e0e72"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-73a5320f-ac42-9e76-bb89-f1a6a4582dd0","request_id":"73a5320f-ac42-9e76-bb89-f1a6a4582dd0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fef8bc52-a368-9297-a8b0-6d83755f18d1","request_id":"fef8bc52-a368-9297-a8b0-6d83755f18d1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3400268e-fb7a-9ad4-a40f-245723623da4","request_id":"3400268e-fb7a-9ad4-a40f-245723623da4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c00309a0-70d7-9be8-be29-3aed200d11b4","request_id":"c00309a0-70d7-9be8-be29-3aed200d11b4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-903cc7d8-b8fe-9172-add2-d7f2f1cb9044","request_id":"903cc7d8-b8fe-9172-add2-d7f2f1cb9044"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1ddda287-4eda-95f7-af85-171d1ca8a40f","request_id":"1ddda287-4eda-95f7-af85-171d1ca8a40f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2981b454-d2d4-9b87-9fa9-eacd2bc9fa95","request_id":"2981b454-d2d4-9b87-9fa9-eacd2bc9fa95"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-67490092-2228-97d6-971d-623b8b5ef01c","request_id":"67490092-2228-97d6-971d-623b8b5ef01c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-19fa9760-bcf7-9cb7-aecb-b23194350acd","request_id":"19fa9760-bcf7-9cb7-aecb-b23194350acd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: Delete redundant.txt{ "commit_message": "Delete redundant.txt" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Remove redundant.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Remove redundant.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Remove redundant.txt{"commit_message": "Remove redundant.txt"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Remove redundant.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)