Remove a single untracked file
Tests ability to remove a single untracked file with git clean -f. Evaluates basic working-directory cleanup.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'tracked' > tracked.txt
  5. 05 git add tracked.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 echo 'untracked' > untracked.txt
Prompt
Remove the untracked file from the working directory. Output ONLY the git command, nothing else.
Expected
git clean -f

Scoped model quality, cost, API time, and token usage for git_clean/f001.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 73 in → 30 out (21 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-fable-5:high__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-fable-5:low PASS 100% 73 in → 17 out (0 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-fable-5:low__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-fable-5:max PASS 100% 73 in → 493 out (56 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-fable-5:max__json_schema PASS 100% 283 in → 184 out (69 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-fable-5:medium PASS 100% 73 in → 30 out (21 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-fable-5:xhigh PASS 100% 73 in → 67 out (34 reasoning)
git clean -f untracked.txt
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-haiku-4.5:high PASS 100% 90 in → 331 out (311 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 253 in → 187 out (163 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-haiku-4.5:low PASS 100% 90 in → 203 out (174 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 253 in → 332 out (295 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-haiku-4.5:medium PASS 100% 90 in → 344 out (317 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 253 in → 210 out (175 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-haiku-4.5:none PASS 100% 61 in → 16 out (0 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 223 in → 12 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 90 in → 1,201 out (769 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 253 in → 403 out (372 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-opus-4.6:high PASS 100% 61 in → 12 out (0 reasoning)
``` git clean -f ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 224 in → 90 out (53 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.6:low PASS 100% 61 in → 9 out (0 reasoning)
`git clean -f`
anthropic/claude-opus-4.6:low__json_schema PASS 100% 224 in → 16 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.6:max PASS 100% 61 in → 12 out (0 reasoning)
``` git clean -f ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 224 in → 84 out (51 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.6:medium PASS 100% 61 in → 11 out (0 reasoning)
``` git clean -f ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 224 in → 11 out (0 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
anthropic/claude-opus-4.6:none PASS 100% 61 in → 11 out (0 reasoning)
``` git clean -f ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 224 in → 11 out (0 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
anthropic/claude-opus-4.6:xhigh PASS 100% 61 in → 12 out (0 reasoning)
``` git clean -f ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 224 in → 90 out (53 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.7:high PASS 100% 78 in → 9 out (0 reasoning)
git clean -f
anthropic/claude-opus-4.7:high__json_schema PASS 100% 288 in → 23 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.7:low PASS 100% 78 in → 9 out (0 reasoning)
git clean -f
anthropic/claude-opus-4.7:low__json_schema PASS 100% 288 in → 22 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.7:max PASS 100% 78 in → 20 out (0 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 288 in → 22 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.7:medium PASS 100% 78 in → 9 out (0 reasoning)
git clean -f
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 288 in → 22 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.7:none PASS 100% 78 in → 17 out (0 reasoning)
`git clean -f untracked.txt`
anthropic/claude-opus-4.7:none__json_schema PASS 100% 288 in → 22 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.7:xhigh PASS 100% 78 in → 17 out (0 reasoning)
`git clean -f untracked.txt`
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 288 in → 22 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.8:high PASS 100% 73 in → 13 out (0 reasoning)
git clean -f untracked.txt
anthropic/claude-opus-4.8:high__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.8:low PASS 100% 73 in → 13 out (0 reasoning)
git clean -f untracked.txt
anthropic/claude-opus-4.8:low__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.8:max PASS 100% 73 in → 13 out (0 reasoning)
git clean -f untracked.txt
anthropic/claude-opus-4.8:max__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-opus-4.8:medium PASS 100% 73 in → 17 out (0 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.8:none PASS 100% 73 in → 17 out (0 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 283 in → 19 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-opus-4.8:xhigh PASS 100% 73 in → 13 out (0 reasoning)
git clean -f untracked.txt
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-sonnet-4.6:high PASS 100% 61 in → 92 out (57 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 224 in → 69 out (35 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-sonnet-4.6:low PASS 100% 61 in → 14 out (0 reasoning)
`git clean -f untracked.txt`
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 224 in → 16 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-sonnet-4.6:max PASS 100% 61 in → 135 out (98 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 224 in → 140 out (99 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-sonnet-4.6:medium PASS 100% 61 in → 16 out (0 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 224 in → 16 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-sonnet-4.6:none PASS 100% 61 in → 13 out (0 reasoning)
``` rm untracked.txt ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 224 in → 16 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 61 in → 93 out (59 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 224 in → 69 out (35 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-sonnet-5:high PASS 100% 73 in → 17 out (0 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-sonnet-5:low PASS 100% 73 in → 17 out (0 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-sonnet-5:medium PASS 100% 73 in → 17 out (0 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
anthropic/claude-sonnet-5:none PASS 100% 73 in → 13 out (0 reasoning)
git clean -f untracked.txt
anthropic/claude-sonnet-5:none__json_schema PASS 100% 283 in → 19 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
anthropic/claude-sonnet-5:xhigh PASS 100% 73 in → 17 out (0 reasoning)
``` git clean -f untracked.txt ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 283 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
arcee-ai/trinity-large-thinking:high PASS 100% 50 in → 845 out (837 reasoning)
git clean -f untracked.txt
arcee-ai/trinity-large-thinking:low PASS 100% 50 in → 1,676 out (1,671 reasoning)
git clean -f
arcee-ai/trinity-large-thinking:medium PASS 100% 50 in → 3,456 out (3,467 reasoning)
git clean -f
arcee-ai/trinity-mini:high PASS 100% 50 in → 296 out (327 reasoning)
git clean -f
arcee-ai/trinity-mini:high__json_schema PASS 100% 50 in → 274 out (295 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
arcee-ai/trinity-mini:low__json_schema PASS 100% 50 in → 856 out (942 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 50 in → 310 out (342 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
arcee-ai/trinity-mini:xhigh PASS 100% 50 in → 302 out (322 reasoning)
git clean -f
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 50 in → 249 out (269 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command":"git clean -f" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 49 in → 158 out (141 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 49 in → 208 out (179 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 49 in → 233 out (220 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 49 in → 307 out (279 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 128 in → 70 out (68 reasoning)
rm untracked.txt
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 128 in → 234 out (212 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 49 in → 271 out (261 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 49 in → 256 out (241 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 49 in → 14 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 128 in → 102 out (98 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 49 in → 236 out (213 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
deepseek/deepseek-v4-flash:high PASS 100% 49 in → 501 out (495 reasoning)
git clean -f
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 51 in → 286 out (272 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
deepseek/deepseek-v4-flash:low PASS 100% 49 in → 275 out (265 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-flash:medium PASS 100% 49 in → 309 out (299 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 51 in → 253 out (239 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
deepseek/deepseek-v4-flash:none PASS 100% 49 in → 4 out (0 reasoning)
git clean -f
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 49 in → 16 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 128 in → 352 out (351 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 141 in → 11 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) {"command": "rm untracked.txt"}
deepseek/deepseek-v4-pro:high PASS 100% 49 in → 394 out (384 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 49 in → 470 out (444 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
deepseek/deepseek-v4-pro:low PASS 100% 49 in → 451 out (441 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 236 in → 346 out (331 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
deepseek/deepseek-v4-pro:medium PASS 100% 49 in → 330 out (320 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 236 in → 141 out (126 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
deepseek/deepseek-v4-pro:none PASS 100% 49 in → 8 out (0 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 49 in → 14 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 128 in → 378 out (368 reasoning)
git clean -f untracked.txt
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 314 in → 760 out (745 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
google/gemini-3-flash-preview:high PASS 100% 50 in → 206 out (202 reasoning)
git clean -f
google/gemini-3-flash-preview:high__json_schema PASS 100% 107 in → 280 out (271 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command": "git clean -f"}
google/gemini-3-flash-preview:low PASS 100% 50 in → 127 out (118 reasoning)
git clean -f untracked.txt
google/gemini-3-flash-preview:low__json_schema PASS 100% 107 in → 228 out (220 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
google/gemini-3-flash-preview:medium PASS 100% 50 in → 428 out (424 reasoning)
git clean -f
google/gemini-3-flash-preview:medium__json_schema PASS 100% 51 in → 467 out (459 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
google/gemini-3-flash-preview:xhigh PASS 100% 50 in → 179 out (175 reasoning)
git clean -f
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 107 in → 157 out (148 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command": "git clean -f"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 51 in → 328 out (319 reasoning)
git clean -f untracked.txt
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 107 in → 358 out (344 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 51 in → 124 out (115 reasoning)
git clean -f untracked.txt
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 51 in → 127 out (108 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 51 in → 128 out (119 reasoning)
git clean -f untracked.txt
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 107 in → 281 out (267 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 50 in → 475 out (466 reasoning)
git clean -f untracked.txt
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 51 in → 249 out (235 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
google/gemini-3.1-pro-preview:high PASS 100% 50 in → 504 out (500 reasoning)
git clean -f
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 107 in → 382 out (373 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command": "git clean -f"}
google/gemini-3.1-pro-preview:low PASS 100% 50 in → 239 out (230 reasoning)
git clean -f untracked.txt
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 107 in → 349 out (335 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
google/gemini-3.1-pro-preview:medium PASS 100% 50 in → 584 out (575 reasoning)
git clean -f untracked.txt
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 107 in → 190 out (176 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 50 in → 676 out (667 reasoning)
git clean -f untracked.txt
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 107 in → 354 out (346 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
google/gemini-3.5-flash-lite:high PASS 100% 51 in → 449 out (440 reasoning)
git clean -f untracked.txt
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 107 in → 435 out (421 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
google/gemini-3.5-flash-lite:low PASS 100% 50 in → 9 out (0 reasoning)
git clean -f untracked.txt
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 51 in → 19 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
google/gemini-3.5-flash-lite:max PASS 100% 51 in → 451 out (442 reasoning)
git clean -f untracked.txt
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 51 in → 463 out (450 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
google/gemini-3.5-flash-lite:medium PASS 100% 51 in → 397 out (388 reasoning)
git clean -f untracked.txt
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 107 in → 288 out (274 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 50 in → 490 out (481 reasoning)
git clean -f untracked.txt
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 107 in → 466 out (453 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
google/gemini-3.5-flash:high PASS 100% 50 in → 444 out (440 reasoning)
git clean -f
google/gemini-3.5-flash:high__json_schema PASS 100% 107 in → 310 out (301 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command": "git clean -f"}
google/gemini-3.5-flash:low PASS 100% 50 in → 346 out (342 reasoning)
git clean -f
google/gemini-3.5-flash:low__json_schema PASS 100% 107 in → 91 out (82 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command": "git clean -f"}
google/gemini-3.5-flash:medium PASS 100% 50 in → 408 out (404 reasoning)
git clean -f
google/gemini-3.5-flash:medium__json_schema PASS 100% 107 in → 404 out (396 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
google/gemini-3.5-flash:xhigh PASS 100% 50 in → 423 out (419 reasoning)
git clean -f
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 107 in → 360 out (351 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command": "git clean -f"}
google/gemini-3.6-flash:high PASS 100% 50 in → 649 out (640 reasoning)
git clean -f untracked.txt
google/gemini-3.6-flash:high__json_schema PASS 100% 51 in → 271 out (263 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
google/gemini-3.6-flash:low PASS 100% 51 in → 137 out (128 reasoning)
git clean -f untracked.txt
google/gemini-3.6-flash:low__json_schema PASS 100% 51 in → 19 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
google/gemini-3.6-flash:max PASS 100% 51 in → 672 out (663 reasoning)
git clean -f untracked.txt
google/gemini-3.6-flash:max__json_schema PASS 100% 51 in → 260 out (246 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
google/gemini-3.6-flash:medium PASS 100% 51 in → 403 out (399 reasoning)
git clean -f
google/gemini-3.6-flash:medium__json_schema PASS 100% 107 in → 250 out (231 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
google/gemini-3.6-flash:xhigh PASS 100% 50 in → 1,011 out (1,002 reasoning)
git clean -f untracked.txt
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 107 in → 593 out (580 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
google/gemma-4-26b-a4b-it:high PASS 100% 66 in → 182 out (177 reasoning)
git clean -f
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 66 in → 178 out (148 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command": "git clean -f"}
google/gemma-4-26b-a4b-it:low PASS 100% 66 in → 441 out (387 reasoning)
git clean -f untracked.txt
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 66 in → 728 out (622 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
google/gemma-4-26b-a4b-it:medium PASS 100% 66 in → 549 out (487 reasoning)
git clean -f untracked.txt
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 65 in → 286 out (233 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
google/gemma-4-26b-a4b-it:none PASS 100% 63 in → 10 out (0 reasoning)
git clean -f untracked.txt
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 63 in → 20 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 66 in → 709 out (699 reasoning)
git clean -f untracked.txt
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 66 in → 908 out (789 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
google/gemma-4-31b-it:high PASS 100% 66 in → 183 out (151 reasoning)
git clean -f untracked.txt
google/gemma-4-31b-it:high__json_schema PASS 100% 66 in → 287 out (237 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
google/gemma-4-31b-it:low PASS 100% 66 in → 178 out (148 reasoning)
git clean -f untracked.txt
google/gemma-4-31b-it:low__json_schema PASS 100% 66 in → 184 out (149 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
google/gemma-4-31b-it:medium PASS 100% 65 in → 331 out (286 reasoning)
git clean -f
google/gemma-4-31b-it:medium__json_schema PASS 100% 65 in → 199 out (155 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
google/gemma-4-31b-it:none PASS 100% 63 in → 10 out (0 reasoning)
git clean -f untracked.txt
google/gemma-4-31b-it:none__json_schema PASS 100% 63 in → 10 out (0 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
google/gemma-4-31b-it:xhigh PASS 100% 65 in → 226 out (202 reasoning)
git clean -f
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 65 in → 636 out (565 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
inclusionai/ling-2.6-flash PASS 100% 66 in → 12 out
``` git clean -f untracked.txt ```
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
minimax/minimax-m2.5:high PASS 100% 78 in → 253 out (287 reasoning)
git clean -f untracked.txt
minimax/minimax-m2.5:high__json_schema PASS 100% 78 in → 172 out (178 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
minimax/minimax-m2.5:low PASS 100% 88 in → 147 out (138 reasoning)
git clean -f untracked.txt
minimax/minimax-m2.5:low__json_schema PASS 100% 88 in → 290 out (274 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
minimax/minimax-m2.5:medium PASS 100% 78 in → 158 out (185 reasoning)
rm untracked.txt
minimax/minimax-m2.5:medium__json_schema PASS 100% 88 in → 288 out (314 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
minimax/minimax-m2.5:xhigh PASS 100% 78 in → 248 out (239 reasoning)
git clean -f untracked.txt
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 88 in → 234 out (218 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
minimax/minimax-m2.7:high PASS 100% 81 in → 345 out (337 reasoning)
git clean -f untracked.txt
minimax/minimax-m2.7:high__json_schema PASS 100% 77 in → 352 out (362 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
minimax/minimax-m2.7:low PASS 100% 78 in → 296 out (316 reasoning)
git clean -f untracked.txt
minimax/minimax-m2.7:medium PASS 100% 81 in → 218 out (210 reasoning)
git clean -f untracked.txt
minimax/minimax-m2.7:medium__json_schema PASS 100% 210 in → 764 out (751 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
minimax/minimax-m2.7:xhigh PASS 100% 78 in → 312 out (303 reasoning)
git clean -f untracked.txt
minimax/minimax-m3:high PASS 100% 216 in → 223 out (227 reasoning)
git clean -f untracked.txt
minimax/minimax-m3:high__json_schema PASS 100% 203 in → 15 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
minimax/minimax-m3:low PASS 100% 216 in → 303 out (329 reasoning)
git clean -f untracked.txt
minimax/minimax-m3:low__json_schema PASS 100% 203 in → 13 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
minimax/minimax-m3:medium PASS 100% 216 in → 203 out (213 reasoning)
git clean -f untracked.txt
minimax/minimax-m3:medium__json_schema PASS 100% 203 in → 15 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
minimax/minimax-m3:xhigh PASS 100% 216 in → 39 out (33 reasoning)
git clean -f untracked.txt
minimax/minimax-m3:xhigh__json_schema PASS 100% 203 in → 13 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
mistralai/devstral-2512 PASS 100% 48 in → 14 out
```bash git clean -f untracked.txt ```
mistralai/devstral-2512__json_schema PASS 100% 48 in → 14 out
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
mistralai/mistral-medium-3-5:high PASS 100% 60 in → 446 out (450 reasoning)
git clean -f
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 48 in → 362 out (358 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f -- untracked.txt"}
mistralai/mistral-medium-3-5:low PASS 100% 60 in → 459 out (453 reasoning)
git clean -f untracked.txt
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 48 in → 401 out (417 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f -- untracked.txt"}
mistralai/mistral-medium-3-5:medium PASS 100% 60 in → 488 out (479 reasoning)
git clean -f -- untracked.txt
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 48 in → 268 out (263 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 48 in → 14 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 60 in → 488 out (486 reasoning)
git clean -f untracked.txt
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 48 in → 206 out (192 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
moonshotai/kimi-k2.6:high PASS 100% 51 in → 1,715 out (1,705 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k2.6:high__json_schema PASS 100% 50 in → 1,048 out (953 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
moonshotai/kimi-k2.6:low PASS 100% 51 in → 935 out (910 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k2.6:medium PASS 100% 50 in → 629 out (652 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k2.6:xhigh PASS 100% 51 in → 787 out (777 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k2.7-code:high PASS 100% 51 in → 262 out (252 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k2.7-code:low PASS 100% 50 in → 225 out (215 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 51 in → 150 out (133 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
moonshotai/kimi-k2.7-code:medium PASS 100% 51 in → 312 out (298 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 50 in → 393 out (170 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 51 in → 253 out (243 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k3:high PASS 100% 128 in → 158 out (135 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k3:high__json_schema PASS 100% 128 in → 211 out (188 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k3:low PASS 100% 128 in → 197 out (174 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k3:low__json_schema PASS 100% 128 in → 186 out (163 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k3:max PASS 100% 128 in → 201 out (178 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k3:max__json_schema PASS 100% 128 in → 163 out (140 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k3:medium PASS 100% 128 in → 153 out (130 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k3:medium__json_schema PASS 100% 128 in → 127 out (104 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k3:xhigh PASS 100% 128 in → 155 out (132 reasoning)
git clean -f untracked.txt
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 128 in → 133 out (110 reasoning)
git clean -f untracked.txt
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 61 in → 174 out (166 reasoning)
git clean -f untracked.txt
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 61 in → 1,014 out (1,130 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 61 in → 132 out (120 reasoning)
git clean -f untracked.txt
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 61 in → 119 out (105 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt" }
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 61 in → 221 out (213 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 61 in → 9 out (0 reasoning)
git clean -fd untracked.txt
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 61 in → 16 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 61 in → 173 out (164 reasoning)
git clean -f untracked.txt
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 61 in → 141 out (129 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 61 in → 154 out (137 reasoning)
git clean -f untracked.txt
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 61 in → 106 out (83 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 61 in → 129 out (109 reasoning)
git clean -f untracked.txt
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 61 in → 81 out (59 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 61 in → 131 out (112 reasoning)
git clean -f untracked.txt
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 61 in → 168 out (142 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 61 in → 13 out (0 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 61 in → 149 out (145 reasoning)
git clean -f untracked.txt
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 61 in → 83 out (61 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
openai/gpt-5.4-mini:high PASS 100% 46 in → 140 out (126 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.4-mini:high__json_schema PASS 100% 81 in → 128 out (106 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.4-mini:low__json_schema PASS 100% 81 in → 68 out (46 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.4-mini:medium PASS 100% 46 in → 89 out (75 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.4-mini:medium__json_schema PASS 100% 81 in → 113 out (91 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.4-mini:none PASS 100% 46 in → 12 out (0 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.4-mini:none__json_schema PASS 100% 81 in → 20 out (0 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.4-mini:xhigh PASS 100% 46 in → 197 out (183 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 81 in → 257 out (235 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.4-nano:high PASS 100% 46 in → 11 out (0 reasoning)
git clean -f untracked.txt
openai/gpt-5.4-nano:high__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
openai/gpt-5.4-nano:low PASS 100% 46 in → 11 out (0 reasoning)
git clean -f untracked.txt
openai/gpt-5.4-nano:low__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
openai/gpt-5.4-nano:medium PASS 100% 46 in → 11 out (0 reasoning)
git clean -f untracked.txt
openai/gpt-5.4-nano:medium__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
openai/gpt-5.4-nano:none PASS 100% 46 in → 11 out (0 reasoning)
git clean -f untracked.txt
openai/gpt-5.4-nano:none__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
openai/gpt-5.4-nano:xhigh PASS 100% 46 in → 11 out (0 reasoning)
git clean -f untracked.txt
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 81 in → 572 out (550 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.4:high PASS 100% 46 in → 67 out (53 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.4:high__json_schema PASS 100% 81 in → 293 out (271 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.4:low PASS 100% 46 in → 12 out (0 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.4:low__json_schema PASS 100% 81 in → 70 out (48 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.4:medium PASS 100% 46 in → 63 out (49 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.4:medium__json_schema PASS 100% 81 in → 67 out (45 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.4:none PASS 100% 46 in → 12 out (0 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.4:none__json_schema PASS 100% 81 in → 20 out (0 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.4:xhigh PASS 100% 46 in → 206 out (192 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.4:xhigh__json_schema PASS 100% 81 in → 538 out (516 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.5:high PASS 100% 46 in → 69 out (55 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.5:high__json_schema PASS 100% 81 in → 80 out (58 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.5:low PASS 100% 46 in → 61 out (47 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.5:low__json_schema PASS 100% 81 in → 20 out (0 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.5:medium PASS 100% 46 in → 65 out (51 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.5:medium__json_schema PASS 100% 81 in → 72 out (49 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f -- untracked.txt"}
openai/gpt-5.5:none PASS 100% 46 in → 11 out (0 reasoning)
git clean -f untracked.txt
openai/gpt-5.5:none__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
openai/gpt-5.5:xhigh PASS 100% 46 in → 107 out (93 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.5:xhigh__json_schema PASS 100% 81 in → 146 out (124 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-luna:high PASS 100% 46 in → 45 out (31 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-luna:high__json_schema PASS 100% 81 in → 66 out (44 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-luna:low PASS 100% 46 in → 12 out (0 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-luna:low__json_schema PASS 100% 81 in → 72 out (50 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-luna:max PASS 100% 46 in → 124 out (110 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-luna:max__json_schema PASS 100% 81 in → 454 out (432 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-luna:medium PASS 100% 46 in → 44 out (30 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-luna:medium__json_schema PASS 100% 81 in → 16 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) {"command":"rm untracked.txt"}
openai/gpt-5.6-luna:none PASS 100% 46 in → 12 out (0 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-luna:none__json_schema PASS 100% 81 in → 16 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) {"command":"rm untracked.txt"}
openai/gpt-5.6-luna:xhigh PASS 100% 46 in → 58 out (44 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 81 in → 142 out (120 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-sol:high PASS 100% 46 in → 8 out (0 reasoning)
rm untracked.txt
openai/gpt-5.6-sol:high__json_schema PASS 100% 81 in → 20 out (0 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-sol:low PASS 100% 46 in → 12 out (0 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-sol:low__json_schema PASS 100% 81 in → 20 out (0 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-sol:max PASS 100% 46 in → 60 out (46 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-sol:max__json_schema PASS 100% 81 in → 157 out (135 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-sol:medium PASS 100% 46 in → 8 out (0 reasoning)
rm untracked.txt
openai/gpt-5.6-sol:medium__json_schema PASS 100% 81 in → 20 out (0 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-sol:none PASS 100% 46 in → 12 out (0 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-sol:none__json_schema PASS 100% 81 in → 20 out (0 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-sol:xhigh PASS 100% 46 in → 48 out (34 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 81 in → 20 out (0 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-terra:high PASS 100% 46 in → 8 out (0 reasoning)
git clean -f
openai/gpt-5.6-terra:high__json_schema PASS 100% 81 in → 16 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) {"command":"rm untracked.txt"}
openai/gpt-5.6-terra:low PASS 100% 46 in → 12 out (0 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-terra:low__json_schema PASS 100% 81 in → 20 out (0 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-terra:max PASS 100% 46 in → 86 out (72 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-terra:max__json_schema PASS 100% 81 in → 16 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) {"command":"rm untracked.txt"}
openai/gpt-5.6-terra:medium PASS 100% 46 in → 12 out (0 reasoning)
git clean -f -- untracked.txt
openai/gpt-5.6-terra:medium__json_schema PASS 100% 81 in → 16 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) {"command":"rm untracked.txt"}
openai/gpt-5.6-terra:none PASS 100% 46 in → 8 out (0 reasoning)
git clean -f
openai/gpt-5.6-terra:none__json_schema PASS 100% 81 in → 20 out (0 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- untracked.txt"}
openai/gpt-5.6-terra:xhigh PASS 100% 46 in → 11 out (0 reasoning)
git clean -f untracked.txt
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 81 in → 16 out (0 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
openai/gpt-oss-120b:high PASS 100% 105 in → 324 out (326 reasoning)
git clean -f untracked.txt
openai/gpt-oss-120b:high__json_schema PASS 100% 207 in → 275 out (251 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
openai/gpt-oss-120b:low PASS 100% 94 in → 38 out (23 reasoning)
git clean -f untracked.txt
openai/gpt-oss-120b:low__json_schema PASS 100% 107 in → 60 out (39 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
openai/gpt-oss-120b:medium PASS 100% 105 in → 124 out (114 reasoning)
git clean -f untracked.txt
openai/gpt-oss-120b:medium__json_schema PASS 100% 207 in → 116 out (95 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
openai/gpt-oss-120b:xhigh PASS 100% 107 in → 331 out (317 reasoning)
git clean -f untracked.txt
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 105 in → 401 out (399 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
openai/gpt-oss-20b:high PASS 100% 107 in → 116 out (109 reasoning)
git clean -f untracked.txt
openai/gpt-oss-20b:high__json_schema PASS 100% 109 in → 312 out (312 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command":"git clean -f"}
openai/gpt-oss-20b:low PASS 100% 112 in → 71 out (54 reasoning)
git clean -f untracked.txt
openai/gpt-oss-20b:low__json_schema PASS 100% 107 in → 63 out (42 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
openai/gpt-oss-20b:medium PASS 100% 107 in → 191 out (184 reasoning)
git clean -f untracked.txt
openai/gpt-oss-20b:medium__json_schema PASS 100% 94 in → 295 out (282 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
openai/gpt-oss-20b:xhigh PASS 100% 105 in → 370 out (390 reasoning)
git clean -f untracked.txt
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 107 in → 273 out (252 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
poolside/laguna-m.1:high PASS 100% 59 in → 435 out (428 reasoning)
git clean -f
poolside/laguna-m.1:high__json_schema PASS 100% 59 in → 504 out (493 reasoning)
git clean -f
JSON Schema Structured Output
(raw) {"command": "git clean -f"}
poolside/laguna-m.1:low PASS 100% 59 in → 318 out (307 reasoning)
git clean -f -- untracked.txt
poolside/laguna-m.1:low__json_schema PASS 100% 59 in → 276 out (261 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
poolside/laguna-m.1:medium PASS 100% 59 in → 600 out (590 reasoning)
git clean -f untracked.txt
poolside/laguna-m.1:medium__json_schema PASS 100% 59 in → 303 out (284 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
poolside/laguna-m.1:none PASS 100% 59 in → 6 out (0 reasoning)
rm untracked.txt
poolside/laguna-m.1:none__json_schema PASS 100% 59 in → 10 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) {"command": "rm untracked.txt"}
poolside/laguna-m.1:xhigh PASS 100% 59 in → 482 out (472 reasoning)
git clean -f untracked.txt
poolside/laguna-m.1:xhigh__json_schema PASS 100% 59 in → 341 out (325 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
poolside/laguna-xs-2.1:high PASS 100% 59 in → 161 out (153 reasoning)
git clean -f untracked.txt
poolside/laguna-xs-2.1:high__json_schema PASS 100% 59 in → 289 out (271 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
poolside/laguna-xs-2.1:low PASS 100% 59 in → 223 out (215 reasoning)
git clean -f untracked.txt
poolside/laguna-xs-2.1:low__json_schema PASS 100% 59 in → 208 out (195 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
poolside/laguna-xs-2.1:medium PASS 100% 59 in → 1,014 out (1,009 reasoning)
git clean -f
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 59 in → 295 out (277 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
poolside/laguna-xs-2.1:none PASS 100% 59 in → 7 out (0 reasoning)
git clean -f untracked.txt
poolside/laguna-xs-2.1:none__json_schema PASS 100% 59 in → 18 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
poolside/laguna-xs-2.1:xhigh PASS 100% 59 in → 213 out (205 reasoning)
git clean -f untracked.txt
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 59 in → 362 out (348 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
poolside/laguna-xs.2:high PASS 100% 96 in → 232 out (222 reasoning)
git clean -f untracked.txt
poolside/laguna-xs.2:high__json_schema PASS 100% 96 in → 131 out (112 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
poolside/laguna-xs.2:low PASS 100% 96 in → 110 out (100 reasoning)
git clean -f untracked.txt
poolside/laguna-xs.2:low__json_schema PASS 100% 96 in → 269 out (255 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
poolside/laguna-xs.2:medium PASS 100% 96 in → 387 out (378 reasoning)
git clean -f untracked.txt
poolside/laguna-xs.2:medium__json_schema PASS 100% 96 in → 88 out (72 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
poolside/laguna-xs.2:none PASS 100% 96 in → 9 out (0 reasoning)
git clean -f untracked.txt
poolside/laguna-xs.2:xhigh PASS 100% 96 in → 569 out (562 reasoning)
rm untracked.txt
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 96 in → 91 out (75 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
qwen/qwen3.6-27b:high PASS 100% 55 in → 962 out (938 reasoning)
git clean -f untracked.txt
qwen/qwen3.6-27b:low PASS 100% 55 in → 1,201 out (1,181 reasoning)
git clean -f untracked.txt
qwen/qwen3.6-27b:low__json_schema PASS 100% 55 in → 1,121 out (1 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
qwen/qwen3.6-27b:medium PASS 100% 55 in → 880 out (876 reasoning)
git clean -f untracked.txt
qwen/qwen3.6-27b:none__json_schema PASS 100% 57 in → 17 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
qwen/qwen3.6-27b:xhigh PASS 100% 55 in → 1,203 out (1,157 reasoning)
git clean -f untracked.txt
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 55 in → 1,945 out (1,811 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
qwen/qwen3.6-35b-a3b:high PASS 100% 55 in → 2,025 out (1,907 reasoning)
git clean -f untracked.txt
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 55 in → 1,126 out (1,083 reasoning)
git clean -f
JSON Schema Structured Output
(raw) { "command": "git clean -f" }
qwen/qwen3.6-35b-a3b:low PASS 100% 55 in → 1,048 out (1,004 reasoning)
git clean -f untracked.txt
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 55 in → 1,350 out (1,301 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 57 in → 15 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 55 in → 689 out (679 reasoning)
git clean -f untracked.txt
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 55 in → 1,751 out (1,621 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
qwen/qwen3.6-flash:high PASS 100% 55 in → 1,136 out (1,124 reasoning)
git clean -f untracked.txt
qwen/qwen3.6-flash:low PASS 100% 55 in → 1,502 out (1,490 reasoning)
git clean -f untracked.txt
qwen/qwen3.6-flash:medium PASS 100% 55 in → 1,508 out (1,496 reasoning)
git clean -f untracked.txt
qwen/qwen3.6-flash:none PASS 100% 57 in → 4 out (0 reasoning)
rm untracked.txt
qwen/qwen3.7-flash:low PASS 100% 55 in → 1,141 out (1,129 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-flash:max PASS 100% 55 in → 1,148 out (1,136 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-flash:medium PASS 100% 55 in → 1,363 out (1,351 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-flash:none PASS 100% 57 in → 7 out (0 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-flash:xhigh PASS 100% 55 in → 1,243 out (1,231 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-max:high PASS 100% 55 in → 501 out (489 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-max:low PASS 100% 55 in → 1,029 out (1,017 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-max:medium PASS 100% 55 in → 745 out (733 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-max:none PASS 100% 57 in → 7 out (0 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-max:xhigh PASS 100% 55 in → 667 out (655 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-plus:high PASS 100% 55 in → 672 out (660 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-plus:low PASS 100% 55 in → 804 out (792 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-plus:medium PASS 100% 55 in → 813 out (801 reasoning)
git clean -f untracked.txt
qwen/qwen3.7-plus:xhigh PASS 100% 55 in → 722 out (710 reasoning)
git clean -f untracked.txt
tencent/hy3:high PASS 100% 53 in → 453 out (444 reasoning)
git clean -f untracked.txt
tencent/hy3:high__json_schema PASS 100% 53 in → 645 out (589 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
tencent/hy3:low PASS 100% 53 in → 370 out (361 reasoning)
git clean -f untracked.txt
tencent/hy3:low__json_schema PASS 100% 53 in → 732 out (724 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
tencent/hy3:medium PASS 100% 53 in → 620 out (611 reasoning)
git clean -f untracked.txt
tencent/hy3:medium__json_schema PASS 100% 53 in → 429 out (412 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
tencent/hy3:none PASS 100% 56 in → 5 out (0 reasoning)
rm untracked.txt
tencent/hy3:none__json_schema PASS 100% 56 in → 15 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
tencent/hy3:xhigh PASS 100% 53 in → 417 out (408 reasoning)
git clean -f untracked.txt
tencent/hy3:xhigh__json_schema PASS 100% 53 in → 1,046 out (970 reasoning)
git clean -f -- untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f -- untracked.txt" }
thinkingmachines/inkling-small:high PASS 100% 55 in → 114 out (111 reasoning)
git clean -f untracked.txt
thinkingmachines/inkling-small:high__json_schema PASS 100% 55 in → 148 out (129 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
thinkingmachines/inkling-small:low PASS 100% 55 in → 83 out (80 reasoning)
git clean -f untracked.txt
thinkingmachines/inkling-small:low__json_schema PASS 100% 55 in → 61 out (41 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
thinkingmachines/inkling-small:max PASS 100% 55 in → 255 out (242 reasoning)
git clean -f untracked.txt
thinkingmachines/inkling-small:max__json_schema PASS 100% 55 in → 91 out (72 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
thinkingmachines/inkling-small:medium PASS 100% 55 in → 109 out (107 reasoning)
git clean -f untracked.txt
thinkingmachines/inkling-small:medium__json_schema PASS 100% 55 in → 70 out (53 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
thinkingmachines/inkling-small:none PASS 100% 53 in → 10 out (0 reasoning)
git clean -f untracked.txt
thinkingmachines/inkling-small:xhigh PASS 100% 55 in → 101 out (95 reasoning)
git clean -f untracked.txt
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 55 in → 127 out (110 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
thinkingmachines/inkling:high PASS 100% 55 in → 97 out (84 reasoning)
git clean -f untracked.txt
thinkingmachines/inkling:low PASS 100% 55 in → 66 out (53 reasoning)
git clean -f untracked.txt
thinkingmachines/inkling:low__json_schema PASS 100% 55 in → 75 out (58 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
thinkingmachines/inkling:max PASS 100% 55 in → 115 out (102 reasoning)
git clean -f untracked.txt
thinkingmachines/inkling:max__json_schema PASS 100% 55 in → 172 out (152 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
thinkingmachines/inkling:medium PASS 100% 55 in → 109 out (95 reasoning)
git clean -f untracked.txt
thinkingmachines/inkling:medium__json_schema PASS 100% 55 in → 135 out (113 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
thinkingmachines/inkling:none__json_schema PASS 100% 53 in → 11 out (0 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) {"command":"rm untracked.txt"}
thinkingmachines/inkling:xhigh PASS 100% 55 in → 117 out (103 reasoning)
git clean -f untracked.txt
x-ai/grok-4.3:high PASS 100% 225 in → 307 out (300 reasoning)
git clean -f untracked.txt
x-ai/grok-4.3:high__json_schema PASS 100% 281 in → 448 out (437 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
x-ai/grok-4.3:low PASS 100% 231 in → 219 out (212 reasoning)
git clean -f untracked.txt
x-ai/grok-4.3:low__json_schema PASS 100% 287 in → 248 out (233 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
x-ai/grok-4.3:max PASS 100% 225 in → 359 out (347 reasoning)
```bash git clean -f untracked.txt ```
x-ai/grok-4.3:max__json_schema PASS 100% 281 in → 362 out (351 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
x-ai/grok-4.3:medium PASS 100% 231 in → 521 out (514 reasoning)
git clean -f untracked.txt
x-ai/grok-4.3:medium__json_schema PASS 100% 287 in → 572 out (557 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
x-ai/grok-4.3:none PASS 100% 223 in → 7 out (0 reasoning)
git clean -f untracked.txt
x-ai/grok-4.3:none__json_schema PASS 100% 279 in → 11 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
x-ai/grok-4.3:xhigh PASS 100% 225 in → 356 out (347 reasoning)
`git clean -f untracked.txt`
x-ai/grok-4.3:xhigh__json_schema PASS 100% 281 in → 420 out (409 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
x-ai/grok-4.5:high PASS 100% 246 in → 429 out (418 reasoning)
``` git clean -f untracked.txt ```
x-ai/grok-4.5:high__json_schema PASS 100% 312 in → 370 out (358 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
x-ai/grok-4.5:low__json_schema PASS 100% 312 in → 351 out (339 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
x-ai/grok-4.5:max PASS 100% 246 in → 272 out (261 reasoning)
``` git clean -f untracked.txt ```
x-ai/grok-4.5:max__json_schema PASS 100% 312 in → 399 out (387 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
x-ai/grok-4.5:medium PASS 100% 246 in → 216 out (205 reasoning)
``` git clean -f untracked.txt ```
x-ai/grok-4.5:medium__json_schema PASS 100% 312 in → 300 out (288 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
x-ai/grok-4.5:xhigh PASS 100% 246 in → 299 out (288 reasoning)
``` git clean -f untracked.txt ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 312 in → 385 out (373 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f untracked.txt"}
z-ai/glm-4.7-flash:high PASS 100% 46 in → 1,843 out (1,835 reasoning)
git clean -f untracked.txt
z-ai/glm-4.7-flash:low PASS 100% 46 in → 1,065 out (1,075 reasoning)
git clean -f
z-ai/glm-4.7-flash:medium PASS 100% 46 in → 479 out (493 reasoning)
git clean -f untracked.txt
z-ai/glm-4.7-flash:none PASS 100% 46 in → 8 out (0 reasoning)
git clean -f untracked.txt
z-ai/glm-4.7-flash:none__json_schema PASS 100% 46 in → 13 out (0 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 46 in → 1,546 out (1,535 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) {"command": "git clean -f untracked.txt"}
z-ai/glm-5.2:high PASS 100% 53 in → 363 out (354 reasoning)
git clean -f untracked.txt
z-ai/glm-5.2:high__json_schema PASS 100% 55 in → 302 out (286 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
z-ai/glm-5.2:low PASS 100% 53 in → 354 out (347 reasoning)
git clean -f untracked.txt
z-ai/glm-5.2:low__json_schema PASS 100% 53 in → 12 out (11 reasoning)
rm untracked.txt
JSON Schema Structured Output
(raw) { "command": "rm untracked.txt" }
z-ai/glm-5.2:medium PASS 100% 53 in → 300 out (292 reasoning)
git clean -f untracked.txt
z-ai/glm-5.2:medium__json_schema PASS 100% 53 in → 231 out (218 reasoning)
git clean -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f untracked.txt" }
z-ai/glm-5.2:none PASS 100% 47 in → 4 out (0 reasoning)
rm untracked.txt
z-ai/glm-5.2:xhigh PASS 100% 53 in → 666 out (657 reasoning)
git clean -f untracked.txt
arcee-ai/trinity-large-thinking:xhigh FAIL 50% 50 in → 1,407 out (1,400 reasoning)
git rm --ignore-unmatch untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
arcee-ai/trinity-mini:low FAIL 50% 50 in → 385 out (422 reasoning)
git rmuntracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
arcee-ai/trinity-mini:medium FAIL 50% 50 in → 337 out (374 reasoning)
git clean-f untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
deepseek/deepseek-v4-flash-0731:none FAIL 50% 49 in → 10 out (0 reasoning)
git rm --cached untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
ibm-granite/granite-4.1-8b FAIL 50% 58 in → 8 out
git rm --cached untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
git rm --cached untracked.txt
JSON Schema Structured Output
(raw) { "command": "git rm --cached untracked.txt" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
liquid/lfm-2-24b-a2b FAIL 50% 56 in → 8 out
git rm untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
minimax/minimax-m2.7:low__json_schema FAIL 50% 78 in → 16 out (16 reasoning)
git rm --cached untracked.txt
JSON Schema Structured Output
(raw) { "command": "git rm --cached untracked.txt" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
mistralai/mistral-medium-3-5:none FAIL 50% 60 in → 9 out (0 reasoning)
git rm --cached untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
nvidia/nemotron-3-nano-30b-a3b:medium FAIL 50% 61 in → 485 out (461 reasoning)
git clean-f untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
nvidia/nemotron-3-super-120b-a12b:none FAIL 50% 61 in → 15 out (0 reasoning)
git rm --cached untracked.txt && rm untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
openai/gpt-5.4-mini:low FAIL 50% 46 in → 35 out (20 reasoning)
git rm --cached --force untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
poolside/laguna-xs.2:none__json_schema FAIL 50% 96 in → 11 out (0 reasoning)
git rm untracked.txt
JSON Schema Structured Output
(raw) {"command": "git rm untracked.txt"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
qwen/qwen3.6-27b:none FAIL 50% 57 in → 8 out (0 reasoning)
git rm --cached untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
qwen/qwen3.6-35b-a3b:low__json_schema FAIL 50% 55 in → 1,526 out (1,453 reasoning)
git rm -f untracked.txt
JSON Schema Structured Output
(raw) { "command": "git rm -f untracked.txt" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
qwen/qwen3.6-35b-a3b:medium FAIL 50% 55 in → 1,053 out (1,050 reasoning)
git clean untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
qwen/qwen3.6-35b-a3b:none FAIL 50% 57 in → 5 out (0 reasoning)
git clean untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
qwen/qwen3.6-flash:xhigh FAIL 50% 55 in → 1,129 out (1,117 reasoning)
git rm -f untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
qwen/qwen3.7-flash:high FAIL 50% 55 in → 1,537 out (1,527 reasoning)
git clean untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
qwen/qwen3.7-plus:none FAIL 50% 57 in → 5 out (0 reasoning)
git rm untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
thinkingmachines/inkling:none FAIL 50% 55 in → 7 out (0 reasoning)
rm untracked.txt<|end_message|>
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
x-ai/grok-4.5:low FAIL 50% 246 in → 178 out (169 reasoning)
```git clean -f untracked.txt```
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
z-ai/glm-4.7-flash:high__json_schema FAIL 50% 46 in → 310 out (307 reasoning)
git rm untracked.txt
JSON Schema Structured Output
(raw) {"command": "git rm untracked.txt"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
z-ai/glm-4.7-flash:medium__json_schema FAIL 50% 46 in → 278 out (264 reasoning)
git rm untracked.txt
JSON Schema Structured Output
(raw) {"command": "git rm untracked.txt"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
z-ai/glm-4.7-flash:xhigh FAIL 50% 46 in → 268 out (262 reasoning)
git rm untracked.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'untracked.txt'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: gitclean -f untracked.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git rm untracked.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -f untracked.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -f untracked.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -f untracked.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1f231ac3-c380-913f-8b05-6f0ce46cc4fd","request_id":"1f231ac3-c380-913f-8b05-6f0ce46cc4fd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e075a06f-3d8f-96db-92ac-000d24a58881","request_id":"e075a06f-3d8f-96db-92ac-000d24a58881"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-00152a3e-5ed1-9cd4-80d6-aaafdfb46436","request_id":"00152a3e-5ed1-9cd4-80d6-aaafdfb46436"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6ad25a40-e1fe-94df-9713-ca66e27afaba","request_id":"6ad25a40-e1fe-94df-9713-ca66e27afaba"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e305ddd6-e5ea-96cb-8b2c-b8bd6a7fd3ca","request_id":"e305ddd6-e5ea-96cb-8b2c-b8bd6a7fd3ca"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9a977114-2268-950b-ab88-90fabd7cecf2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c9785d19-d5a3-9aa8-9f6d-59b521b93c4e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-62b8e4e9-d0e0-912b-ba68-599cc74b4e63"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f8b8fba7-30ca-9ac6-a092-1898b0d3fdd1"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e0175bef-b2b7-9b23-842a-eb0b4f9cbf29"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d5d23027-5008-93b1-a113-fd46d6014bcf"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5830cd16-6b56-9f18-8d43-1408b922c369","request_id":"5830cd16-6b56-9f18-8d43-1408b922c369"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5d69f305-b0ee-9153-a4bc-f919eec1b713","request_id":"5d69f305-b0ee-9153-a4bc-f919eec1b713"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ab3d00a3-f61f-9dcf-9543-da4ec428ec26","request_id":"ab3d00a3-f61f-9dcf-9543-da4ec428ec26"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-15e788fb-18e9-950a-bb53-0102ffed11c6","request_id":"15e788fb-18e9-950a-bb53-0102ffed11c6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4d40d2c4-e41c-9aad-a94f-4fe5678893d2","request_id":"4d40d2c4-e41c-9aad-a94f-4fe5678893d2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e1233a2a-32ac-9744-9343-4e0d9f69be88","request_id":"e1233a2a-32ac-9744-9343-4e0d9f69be88"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a077832d-0f57-9bfb-87c3-96463ba38e2b","request_id":"a077832d-0f57-9bfb-87c3-96463ba38e2b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7eaef7f6-0f79-9fb9-a194-d3de27a3cac8","request_id":"7eaef7f6-0f79-9fb9-a194-d3de27a3cac8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5dae5332-ee23-93b5-9f3b-93a47810f012","request_id":"5dae5332-ee23-93b5-9f3b-93a47810f012"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b7b268ab-52d9-92ab-bd6b-20ad241da4af","request_id":"b7b268ab-52d9-92ab-bd6b-20ad241da4af"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: git clean -f untracked.txt{ "command": "git clean -f untracked.txt" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -f untracked.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -f untracked.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Model 'z-ai/glm-5.2:none' violated the reasoning_level=none invariant: provider returned non-empty reasoning content
Invalid JSON. Output: git clean -f untracked.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)