Remove untracked directories
Tests ability to remove untracked directories with git clean -fd. Evaluates recursive cleanup awareness.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'tracked' > tracked.txt
  5. 05 git add tracked.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p build/output
  8. 08 echo 'artifact' > build/output/app.bin
  9. 09 mkdir -p tmp
  10. 10 echo 'temp' > tmp/cache.txt
Prompt
Remove all untracked files and directories from the working directory. Output ONLY the git command, nothing else.
Expected
git clean -f -d

Scoped model quality, cost, API time, and token usage for git_clean/f003.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 91 in → 20 out (23 reasoning)
``` git clean -fd ```
anthropic/claude-fable-5:high__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-fable-5:low PASS 100% 91 in → 8 out (0 reasoning)
git clean -fd
anthropic/claude-fable-5:low__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-fable-5:max PASS 100% 91 in → 169 out (61 reasoning)
``` git clean -fd ```
anthropic/claude-fable-5:max__json_schema PASS 100% 301 in → 80 out (38 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-fable-5:medium PASS 100% 91 in → 16 out (23 reasoning)
git clean -fd
anthropic/claude-fable-5:medium__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-fable-5:xhigh PASS 100% 91 in → 19 out (33 reasoning)
git clean -fd
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-haiku-4.5:high PASS 100% 102 in → 153 out (144 reasoning)
```bash git clean -fd ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 265 in → 255 out (234 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-haiku-4.5:low PASS 100% 102 in → 182 out (174 reasoning)
``` git clean -fd ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 265 in → 158 out (150 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-haiku-4.5:medium PASS 100% 102 in → 147 out (137 reasoning)
``` git clean -fd ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 265 in → 158 out (153 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-haiku-4.5:none PASS 100% 73 in → 11 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 235 in → 12 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 102 in → 141 out (128 reasoning)
``` git clean -fd ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 265 in → 146 out (135 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-opus-4.6:high PASS 100% 73 in → 12 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 236 in → 52 out (27 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.6:low PASS 100% 73 in → 7 out (0 reasoning)
git clean -fd
anthropic/claude-opus-4.6:low__json_schema PASS 100% 236 in → 11 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.6:max PASS 100% 73 in → 12 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 236 in → 76 out (56 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.6:medium PASS 100% 73 in → 11 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 236 in → 11 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.6:none PASS 100% 73 in → 11 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 236 in → 11 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.6:xhigh PASS 100% 73 in → 12 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 236 in → 52 out (27 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.7:high PASS 100% 96 in → 10 out (0 reasoning)
git clean -fd
anthropic/claude-opus-4.7:high__json_schema PASS 100% 306 in → 17 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.7:low PASS 100% 96 in → 10 out (0 reasoning)
git clean -fd
anthropic/claude-opus-4.7:low__json_schema PASS 100% 306 in → 17 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.7:max PASS 100% 96 in → 15 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 306 in → 17 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.7:medium PASS 100% 96 in → 10 out (0 reasoning)
git clean -fd
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 306 in → 17 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.7:none PASS 100% 96 in → 12 out (0 reasoning)
`git clean -fd`
anthropic/claude-opus-4.7:none__json_schema PASS 100% 306 in → 17 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.7:xhigh PASS 100% 96 in → 10 out (0 reasoning)
git clean -fd
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 306 in → 17 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.8:high PASS 100% 91 in → 8 out (0 reasoning)
git clean -fd
anthropic/claude-opus-4.8:high__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.8:low PASS 100% 91 in → 8 out (0 reasoning)
git clean -fd
anthropic/claude-opus-4.8:low__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.8:max PASS 100% 91 in → 183 out (47 reasoning)
`git clean -fd`
anthropic/claude-opus-4.8:max__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-opus-4.8:medium PASS 100% 91 in → 8 out (0 reasoning)
git clean -fd
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.8:none PASS 100% 91 in → 12 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 301 in → 14 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-opus-4.8:xhigh PASS 100% 91 in → 8 out (0 reasoning)
git clean -fd
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-sonnet-4.6:high PASS 100% 73 in → 110 out (90 reasoning)
``` git clean -fd ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 236 in → 68 out (47 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-sonnet-4.6:low PASS 100% 73 in → 11 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 236 in → 11 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-sonnet-4.6:max PASS 100% 73 in → 143 out (126 reasoning)
``` git clean -fd ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 236 in → 116 out (97 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-sonnet-4.6:medium PASS 100% 73 in → 11 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 236 in → 11 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-sonnet-4.6:none PASS 100% 73 in → 11 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 236 in → 11 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 73 in → 111 out (90 reasoning)
``` git clean -fd ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 236 in → 68 out (47 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-sonnet-5:high PASS 100% 91 in → 12 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-sonnet-5:low PASS 100% 91 in → 8 out (0 reasoning)
git clean -fd
anthropic/claude-sonnet-5:low__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-sonnet-5:medium PASS 100% 91 in → 12 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
anthropic/claude-sonnet-5:none PASS 100% 91 in → 12 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 301 in → 14 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
anthropic/claude-sonnet-5:xhigh PASS 100% 91 in → 12 out (0 reasoning)
``` git clean -fd ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 301 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
arcee-ai/trinity-large-thinking:high PASS 100% 58 in → 489 out (486 reasoning)
git clean -fd
arcee-ai/trinity-large-thinking:low PASS 100% 58 in → 316 out (310 reasoning)
git clean -fd
arcee-ai/trinity-large-thinking:medium PASS 100% 58 in → 303 out (297 reasoning)
git clean -fd
arcee-ai/trinity-large-thinking:xhigh PASS 100% 58 in → 304 out (299 reasoning)
git clean -fd
arcee-ai/trinity-mini:high__json_schema PASS 100% 58 in → 432 out (471 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
arcee-ai/trinity-mini:low PASS 100% 58 in → 538 out (604 reasoning)
git clean -fd
arcee-ai/trinity-mini:low__json_schema PASS 100% 58 in → 630 out (698 reasoning)
git clean -f -d
JSON Schema Structured Output
(raw) { "command": "git clean -f -d" }
arcee-ai/trinity-mini:medium PASS 100% 58 in → 362 out (405 reasoning)
git clean -fd
arcee-ai/trinity-mini:medium__json_schema PASS 100% 58 in → 321 out (384 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
arcee-ai/trinity-mini:xhigh PASS 100% 58 in → 397 out (428 reasoning)
git clean -fd
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 58 in → 314 out (359 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 57 in → 131 out (130 reasoning)
git clean -fd
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 136 in → 46 out (35 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 57 in → 152 out (148 reasoning)
git clean -fd
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 63 in → 96 out (87 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 149 in → 111 out (106 reasoning)
git clean -fd
deepseek/deepseek-v4-flash-0731:medium PASS 100% 57 in → 186 out (181 reasoning)
git clean -fd
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 57 in → 71 out (61 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 57 in → 4 out (0 reasoning)
git clean -fd
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 57 in → 13 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 57 in → 204 out (200 reasoning)
git clean -fd
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 142 in → 55 out (45 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
deepseek/deepseek-v4-flash:high PASS 100% 57 in → 128 out (124 reasoning)
git clean -fd
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 149 in → 9 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
deepseek/deepseek-v4-flash:low PASS 100% 57 in → 166 out (161 reasoning)
git clean -fd
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 59 in → 183 out (170 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
deepseek/deepseek-v4-flash:medium PASS 100% 57 in → 93 out (87 reasoning)
git clean -fd
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 59 in → 124 out (110 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
deepseek/deepseek-v4-flash:none PASS 100% 57 in → 5 out (0 reasoning)
git clean -fd
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 149 in → 10 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
deepseek/deepseek-v4-flash:xhigh PASS 100% 136 in → 311 out (318 reasoning)
git clean -fd
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 136 in → 204 out (206 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
deepseek/deepseek-v4-pro:high PASS 100% 57 in → 5 out (0 reasoning)
git clean -fd
deepseek/deepseek-v4-pro:low PASS 100% 57 in → 210 out (205 reasoning)
git clean -fd
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 244 in → 48 out (37 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
deepseek/deepseek-v4-pro:medium PASS 100% 57 in → 176 out (170 reasoning)
git clean -fd
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 244 in → 104 out (94 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
deepseek/deepseek-v4-pro:none PASS 100% 57 in → 4 out (0 reasoning)
git clean -fd
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 244 in → 10 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 136 in → 249 out (243 reasoning)
git clean -fd
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 322 in → 546 out (534 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3-flash-preview:high PASS 100% 62 in → 214 out (210 reasoning)
git clean -fd
google/gemini-3-flash-preview:high__json_schema PASS 100% 119 in → 220 out (212 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3-flash-preview:low PASS 100% 62 in → 160 out (156 reasoning)
git clean -fd
google/gemini-3-flash-preview:low__json_schema PASS 100% 63 in → 214 out (205 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3-flash-preview:medium PASS 100% 62 in → 208 out (204 reasoning)
git clean -fd
google/gemini-3-flash-preview:medium__json_schema PASS 100% 63 in → 165 out (156 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3-flash-preview:xhigh PASS 100% 62 in → 177 out (173 reasoning)
git clean -fd
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 119 in → 286 out (278 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 63 in → 205 out (201 reasoning)
git clean -fd
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 63 in → 228 out (219 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 62 in → 115 out (111 reasoning)
git clean -fd
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 119 in → 114 out (105 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 62 in → 173 out (169 reasoning)
git clean -fd
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 63 in → 185 out (176 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 62 in → 252 out (248 reasoning)
git clean -fd
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 119 in → 224 out (215 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.1-pro-preview:high PASS 100% 62 in → 223 out (219 reasoning)
git clean -fd
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 119 in → 266 out (257 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.1-pro-preview:low PASS 100% 62 in → 162 out (158 reasoning)
git clean -fd
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 119 in → 169 out (161 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3.1-pro-preview:medium PASS 100% 62 in → 198 out (194 reasoning)
git clean -fd
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 119 in → 219 out (211 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 62 in → 204 out (200 reasoning)
git clean -fd
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 119 in → 190 out (181 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.5-flash-lite:high PASS 100% 62 in → 262 out (258 reasoning)
git clean -fd
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 63 in → 334 out (325 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.5-flash-lite:low PASS 100% 63 in → 6 out (0 reasoning)
`git clean -fd`
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 63 in → 14 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
google/gemini-3.5-flash-lite:max PASS 100% 62 in → 242 out (238 reasoning)
git clean -fd
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 63 in → 299 out (290 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.5-flash-lite:medium PASS 100% 62 in → 167 out (163 reasoning)
git clean -fd
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 119 in → 194 out (185 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 62 in → 161 out (157 reasoning)
git clean -fd
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 63 in → 324 out (316 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3.5-flash:high PASS 100% 62 in → 266 out (262 reasoning)
git clean -fd
google/gemini-3.5-flash:high__json_schema PASS 100% 119 in → 190 out (181 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.5-flash:low PASS 100% 62 in → 258 out (254 reasoning)
git clean -fd
google/gemini-3.5-flash:low__json_schema PASS 100% 119 in → 126 out (118 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3.5-flash:medium PASS 100% 62 in → 297 out (293 reasoning)
git clean -fd
google/gemini-3.5-flash:medium__json_schema PASS 100% 119 in → 204 out (196 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3.5-flash:xhigh PASS 100% 62 in → 330 out (326 reasoning)
git clean -fd
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 119 in → 328 out (319 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemini-3.6-flash:high PASS 100% 63 in → 278 out (274 reasoning)
git clean -fd
google/gemini-3.6-flash:high__json_schema PASS 100% 119 in → 288 out (280 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3.6-flash:low PASS 100% 63 in → 230 out (226 reasoning)
git clean -fd
google/gemini-3.6-flash:low__json_schema PASS 100% 119 in → 8 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3.6-flash:max PASS 100% 62 in → 291 out (287 reasoning)
git clean -fd
google/gemini-3.6-flash:max__json_schema PASS 100% 119 in → 210 out (202 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3.6-flash:medium PASS 100% 63 in → 362 out (358 reasoning)
git clean -fd
google/gemini-3.6-flash:medium__json_schema PASS 100% 63 in → 239 out (231 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemini-3.6-flash:xhigh PASS 100% 62 in → 291 out (287 reasoning)
git clean -fd
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 119 in → 278 out (270 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
google/gemma-4-26b-a4b-it:high PASS 100% 78 in → 217 out (194 reasoning)
git clean -fd
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 78 in → 269 out (226 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
google/gemma-4-26b-a4b-it:low PASS 100% 78 in → 243 out (211 reasoning)
git clean -fd
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 78 in → 229 out (192 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
google/gemma-4-26b-a4b-it:medium PASS 100% 78 in → 298 out (272 reasoning)
git clean -fd
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 78 in → 271 out (221 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
google/gemma-4-26b-a4b-it:none PASS 100% 75 in → 5 out (0 reasoning)
git clean -fd
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 75 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 78 in → 187 out (152 reasoning)
git clean -fd
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 78 in → 255 out (212 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
google/gemma-4-31b-it:high PASS 100% 78 in → 252 out (1 reasoning)
git clean -fd
google/gemma-4-31b-it:high__json_schema PASS 100% 77 in → 153 out (136 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
google/gemma-4-31b-it:low PASS 100% 78 in → 161 out (1 reasoning)
git clean -fd
google/gemma-4-31b-it:low__json_schema PASS 100% 78 in → 172 out (154 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemma-4-31b-it:medium PASS 100% 78 in → 208 out (181 reasoning)
git clean -fd
google/gemma-4-31b-it:medium__json_schema PASS 100% 77 in → 133 out (114 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemma-4-31b-it:none PASS 100% 75 in → 5 out (0 reasoning)
git clean -fd
google/gemma-4-31b-it:none__json_schema PASS 100% 75 in → 10 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
google/gemma-4-31b-it:xhigh PASS 100% 78 in → 159 out (1 reasoning)
git clean -fd
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 78 in → 232 out (210 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
ibm-granite/granite-4.1-8b PASS 100% 71 in → 5 out
git clean -fd
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
inclusionai/ling-2.6-flash PASS 100% 77 in → 9 out
``` git clean -fd ```
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
minimax/minimax-m2.5:high PASS 100% 86 in → 258 out (294 reasoning)
git clean -fd
minimax/minimax-m2.5:high__json_schema PASS 100% 86 in → 113 out (121 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
minimax/minimax-m2.5:low PASS 100% 86 in → 129 out (141 reasoning)
git clean -fd
minimax/minimax-m2.5:low__json_schema PASS 100% 86 in → 108 out (109 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
minimax/minimax-m2.5:medium PASS 100% 89 in → 183 out (178 reasoning)
git clean -fd
minimax/minimax-m2.5:medium__json_schema PASS 100% 86 in → 137 out (146 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
minimax/minimax-m2.5:xhigh PASS 100% 86 in → 142 out (160 reasoning)
git clean -fd
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 86 in → 136 out (144 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
minimax/minimax-m2.7:high PASS 100% 86 in → 200 out (194 reasoning)
git clean -fd
minimax/minimax-m2.7:low PASS 100% 86 in → 279 out (272 reasoning)
git clean -fd
minimax/minimax-m2.7:low__json_schema PASS 100% 218 in → 264 out (250 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
minimax/minimax-m2.7:medium PASS 100% 86 in → 107 out (120 reasoning)
git clean -fd
minimax/minimax-m2.7:medium__json_schema PASS 100% 218 in → 214 out (200 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
minimax/minimax-m2.7:xhigh PASS 100% 86 in → 259 out (266 reasoning)
git clean -fd
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 218 in → 255 out (237 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
minimax/minimax-m3:high PASS 100% 224 in → 109 out (116 reasoning)
git clean -fd
minimax/minimax-m3:high__json_schema PASS 100% 211 in → 12 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
minimax/minimax-m3:low PASS 100% 224 in → 55 out (65 reasoning)
git clean -fd
minimax/minimax-m3:low__json_schema PASS 100% 211 in → 12 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
minimax/minimax-m3:medium PASS 100% 224 in → 74 out (74 reasoning)
git clean -fd
minimax/minimax-m3:medium__json_schema PASS 100% 211 in → 10 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
minimax/minimax-m3:xhigh PASS 100% 224 in → 55 out (62 reasoning)
git clean -fd
minimax/minimax-m3:xhigh__json_schema PASS 100% 211 in → 12 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
mistralai/devstral-2512 PASS 100% 54 in → 10 out
```bash git clean -fd ```
mistralai/devstral-2512__json_schema PASS 100% 54 in → 10 out
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
mistralai/mistral-medium-3-5:high PASS 100% 66 in → 378 out (404 reasoning)
git clean -fd
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 54 in → 163 out (152 reasoning)
git clean -df
JSON Schema Structured Output
(raw) {"command": "git clean -df"}
mistralai/mistral-medium-3-5:low PASS 100% 66 in → 276 out (296 reasoning)
git clean -fd
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 54 in → 223 out (230 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
mistralai/mistral-medium-3-5:medium PASS 100% 66 in → 369 out (410 reasoning)
git clean -fd
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 54 in → 241 out (255 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
mistralai/mistral-medium-3-5:none PASS 100% 66 in → 5 out (0 reasoning)
git clean -fd
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 54 in → 10 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 66 in → 1,087 out (1,203 reasoning)
git clean -fd
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 54 in → 238 out (246 reasoning)
git clean -df
JSON Schema Structured Output
(raw) {"command": "git clean -df"}
moonshotai/kimi-k2.6:high PASS 100% 58 in → 176 out (170 reasoning)
git clean -fd
moonshotai/kimi-k2.6:high__json_schema PASS 100% 58 in → 218 out (238 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
moonshotai/kimi-k2.6:low PASS 100% 58 in → 133 out (127 reasoning)
git clean -fd
moonshotai/kimi-k2.6:low__json_schema PASS 100% 240 in → 392 out (382 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
moonshotai/kimi-k2.6:medium PASS 100% 58 in → 249 out (243 reasoning)
git clean -fd
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 57 in → 585 out (366 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
moonshotai/kimi-k2.6:xhigh PASS 100% 58 in → 247 out (241 reasoning)
git clean -fd
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 58 in → 287 out (273 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
moonshotai/kimi-k2.7-code:high PASS 100% 58 in → 48 out (50 reasoning)
git clean -fd
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 58 in → 72 out (59 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
moonshotai/kimi-k2.7-code:low PASS 100% 57 in → 113 out (107 reasoning)
git clean -fd
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 141 in → 66 out (55 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
moonshotai/kimi-k2.7-code:medium PASS 100% 58 in → 112 out (106 reasoning)
git clean -fd
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 58 in → 94 out (80 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 57 in → 248 out (242 reasoning)
git clean -fd
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 58 in → 91 out (77 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
moonshotai/kimi-k3:high PASS 100% 135 in → 109 out (90 reasoning)
git clean -fd
moonshotai/kimi-k3:high__json_schema PASS 100% 135 in → 181 out (162 reasoning)
git clean -fd
moonshotai/kimi-k3:low PASS 100% 135 in → 313 out (294 reasoning)
git clean -fd
moonshotai/kimi-k3:low__json_schema PASS 100% 135 in → 110 out (91 reasoning)
git clean -fd
moonshotai/kimi-k3:max PASS 100% 135 in → 89 out (70 reasoning)
git clean -fd
moonshotai/kimi-k3:max__json_schema PASS 100% 135 in → 157 out (138 reasoning)
git clean -fd
moonshotai/kimi-k3:medium PASS 100% 135 in → 176 out (157 reasoning)
git clean -fd
moonshotai/kimi-k3:medium__json_schema PASS 100% 135 in → 133 out (114 reasoning)
git clean -fd
moonshotai/kimi-k3:xhigh PASS 100% 135 in → 96 out (74 reasoning)
``` git clean -fd ```
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 135 in → 83 out (64 reasoning)
git clean -fd
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 67 in → 194 out (192 reasoning)
git clean -fd
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 67 in → 286 out (311 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 67 in → 597 out (649 reasoning)
git clean -fd
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 67 in → 508 out (554 reasoning)
git clean -fd
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 67 in → 335 out (352 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 67 in → 6 out (0 reasoning)
git clean -fdx
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 67 in → 12 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 67 in → 616 out (665 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 67 in → 49 out (39 reasoning)
git clean -f -d
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 67 in → 141 out (126 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 67 in → 271 out (256 reasoning)
git clean -fd
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 67 in → 143 out (125 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 67 in → 82 out (71 reasoning)
git clean -f -d
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 67 in → 198 out (178 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 67 in → 5 out (0 reasoning)
git clean -fd
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 67 in → 15 out (0 reasoning)
git clean -f -d
JSON Schema Structured Output
(raw) { "command": "git clean -f -d" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 67 in → 274 out (259 reasoning)
git clean -fd
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 67 in → 159 out (141 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
openai/gpt-5.4-mini:high PASS 100% 54 in → 98 out (88 reasoning)
git clean -fd
openai/gpt-5.4-mini:high__json_schema PASS 100% 89 in → 178 out (160 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4-mini:low PASS 100% 54 in → 41 out (31 reasoning)
git clean -fd
openai/gpt-5.4-mini:low__json_schema PASS 100% 89 in → 50 out (32 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4-mini:medium PASS 100% 54 in → 92 out (82 reasoning)
git clean -fd
openai/gpt-5.4-mini:medium__json_schema PASS 100% 89 in → 111 out (93 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4-mini:none PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.4-mini:none__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4-mini:xhigh PASS 100% 54 in → 143 out (133 reasoning)
git clean -fd
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 89 in → 343 out (325 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4-nano:high PASS 100% 54 in → 9 out (0 reasoning)
git clean -fdx
openai/gpt-5.4-nano:high__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4-nano:low PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.4-nano:low__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4-nano:medium PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.4-nano:medium__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4-nano:none__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4-nano:xhigh PASS 100% 54 in → 119 out (109 reasoning)
git clean -fd
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 89 in → 108 out (90 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4:high PASS 100% 54 in → 56 out (46 reasoning)
git clean -fd
openai/gpt-5.4:high__json_schema PASS 100% 89 in → 53 out (35 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4:low PASS 100% 54 in → 32 out (22 reasoning)
git clean -fd
openai/gpt-5.4:low__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4:medium PASS 100% 54 in → 61 out (51 reasoning)
git clean -fd
openai/gpt-5.4:medium__json_schema PASS 100% 89 in → 46 out (28 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4:none PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.4:none__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.4:xhigh PASS 100% 54 in → 99 out (89 reasoning)
git clean -fd
openai/gpt-5.4:xhigh__json_schema PASS 100% 89 in → 154 out (136 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.5:high PASS 100% 54 in → 51 out (41 reasoning)
git clean -fd
openai/gpt-5.5:high__json_schema PASS 100% 89 in → 51 out (33 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.5:low PASS 100% 54 in → 39 out (29 reasoning)
git clean -fd
openai/gpt-5.5:low__json_schema PASS 100% 89 in → 17 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
openai/gpt-5.5:medium PASS 100% 54 in → 41 out (31 reasoning)
git clean -fd
openai/gpt-5.5:medium__json_schema PASS 100% 89 in → 137 out (119 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.5:none PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.5:none__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.5:xhigh PASS 100% 54 in → 71 out (61 reasoning)
git clean -fd
openai/gpt-5.5:xhigh__json_schema PASS 100% 89 in → 64 out (46 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-luna:high PASS 100% 54 in → 37 out (27 reasoning)
git clean -fd
openai/gpt-5.6-luna:high__json_schema PASS 100% 89 in → 43 out (25 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-luna:low PASS 100% 54 in → 39 out (29 reasoning)
git clean -fd
openai/gpt-5.6-luna:low__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-luna:max PASS 100% 54 in → 60 out (50 reasoning)
git clean -fd
openai/gpt-5.6-luna:max__json_schema PASS 100% 89 in → 71 out (53 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-luna:medium PASS 100% 54 in → 44 out (34 reasoning)
git clean -fd
openai/gpt-5.6-luna:medium__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-luna:none PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-luna:none__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-luna:xhigh PASS 100% 54 in → 44 out (34 reasoning)
git clean -fd
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 89 in → 266 out (248 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-sol:high PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-sol:high__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-sol:low PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-sol:low__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-sol:max PASS 100% 54 in → 45 out (35 reasoning)
git clean -fd
openai/gpt-5.6-sol:max__json_schema PASS 100% 89 in → 131 out (113 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-sol:medium PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-sol:medium__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-sol:none PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-sol:none__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-sol:xhigh PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-terra:high PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-terra:high__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-terra:low PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-terra:low__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-terra:max PASS 100% 54 in → 53 out (43 reasoning)
git clean -fd
openai/gpt-5.6-terra:max__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-terra:medium PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-terra:medium__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-terra:none PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-terra:none__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-5.6-terra:xhigh PASS 100% 54 in → 8 out (0 reasoning)
git clean -fd
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 89 in → 16 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-oss-120b:high PASS 100% 102 in → 275 out (311 reasoning)
git clean -fd
openai/gpt-oss-120b:high__json_schema PASS 100% 102 in → 276 out (296 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-oss-120b:low PASS 100% 115 in → 32 out (20 reasoning)
git clean -fd
openai/gpt-oss-120b:low__json_schema PASS 100% 182 in → 41 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-oss-120b:medium PASS 100% 117 in → 62 out (51 reasoning)
git clean -fd
openai/gpt-oss-120b:medium__json_schema PASS 100% 102 in → 91 out (86 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-oss-120b:xhigh PASS 100% 115 in → 149 out (140 reasoning)
git clean -fd
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 117 in → 511 out (572 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-oss-20b:high PASS 100% 113 in → 302 out (305 reasoning)
git clean -fd
openai/gpt-oss-20b:high__json_schema PASS 100% 117 in → 322 out (337 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
openai/gpt-oss-20b:low PASS 100% 115 in → 141 out (137 reasoning)
git clean -fd
openai/gpt-oss-20b:low__json_schema PASS 100% 115 in → 72 out (54 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
openai/gpt-oss-20b:medium PASS 100% 113 in → 184 out (193 reasoning)
git clean -fdx
openai/gpt-oss-20b:medium__json_schema PASS 100% 117 in → 152 out (150 reasoning)
git clean -fdx
JSON Schema Structured Output
(raw) {"command":"git clean -fdx"}
openai/gpt-oss-20b:xhigh PASS 100% 115 in → 501 out (558 reasoning)
git clean -fd
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 113 in → 522 out (551 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
poolside/laguna-m.1:high PASS 100% 69 in → 318 out (311 reasoning)
git clean -fd
poolside/laguna-m.1:high__json_schema PASS 100% 69 in → 403 out (392 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
poolside/laguna-m.1:low PASS 100% 69 in → 2,448 out (2,440 reasoning)
git clean -fdx
poolside/laguna-m.1:low__json_schema PASS 100% 69 in → 408 out (397 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
poolside/laguna-m.1:medium PASS 100% 69 in → 242 out (235 reasoning)
git clean -fd
poolside/laguna-m.1:medium__json_schema PASS 100% 69 in → 247 out (236 reasoning)
git clean -df
JSON Schema Structured Output
(raw) {"command": "git clean -df"}
poolside/laguna-m.1:none PASS 100% 69 in → 6 out (0 reasoning)
git clean -fd
poolside/laguna-m.1:none__json_schema PASS 100% 69 in → 14 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
poolside/laguna-m.1:xhigh PASS 100% 69 in → 276 out (269 reasoning)
git clean -fd
poolside/laguna-m.1:xhigh__json_schema PASS 100% 69 in → 244 out (233 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
poolside/laguna-xs-2.1:high PASS 100% 69 in → 269 out (264 reasoning)
git clean -fd
poolside/laguna-xs-2.1:high__json_schema PASS 100% 69 in → 280 out (269 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
poolside/laguna-xs-2.1:low PASS 100% 69 in → 175 out (170 reasoning)
git clean -fd
poolside/laguna-xs-2.1:low__json_schema PASS 100% 69 in → 365 out (354 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
poolside/laguna-xs-2.1:medium PASS 100% 69 in → 633 out (628 reasoning)
git clean -fd
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 69 in → 385 out (374 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
poolside/laguna-xs-2.1:none PASS 100% 69 in → 4 out (0 reasoning)
git clean -fd
poolside/laguna-xs-2.1:none__json_schema PASS 100% 69 in → 10 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
poolside/laguna-xs-2.1:xhigh PASS 100% 69 in → 257 out (252 reasoning)
git clean -fd
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 69 in → 413 out (398 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
poolside/laguna-xs.2:high PASS 100% 106 in → 188 out (179 reasoning)
git clean -d -f
poolside/laguna-xs.2:high__json_schema PASS 100% 106 in → 53 out (42 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
poolside/laguna-xs.2:low PASS 100% 106 in → 121 out (114 reasoning)
git clean -fd
poolside/laguna-xs.2:low__json_schema PASS 100% 106 in → 145 out (134 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
poolside/laguna-xs.2:medium PASS 100% 106 in → 163 out (156 reasoning)
git clean -fd
poolside/laguna-xs.2:medium__json_schema PASS 100% 106 in → 202 out (186 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
poolside/laguna-xs.2:none PASS 100% 106 in → 6 out (0 reasoning)
git clean -fd
poolside/laguna-xs.2:none__json_schema PASS 100% 106 in → 10 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
poolside/laguna-xs.2:xhigh PASS 100% 106 in → 118 out (111 reasoning)
git clean -fd
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 106 in → 127 out (116 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
qwen/qwen3.6-27b:high PASS 100% 65 in → 1,457 out (1 reasoning)
git clean -fd
qwen/qwen3.6-27b:low PASS 100% 65 in → 1,016 out (989 reasoning)
git clean -fd
qwen/qwen3.6-27b:medium PASS 100% 65 in → 867 out (855 reasoning)
git clean -fd
qwen/qwen3.6-27b:none PASS 100% 67 in → 5 out (0 reasoning)
git clean -fd
qwen/qwen3.6-27b:none__json_schema PASS 100% 67 in → 10 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
qwen/qwen3.6-27b:xhigh PASS 100% 65 in → 1,318 out (1,256 reasoning)
git clean -fd
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 65 in → 1,000 out (951 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
qwen/qwen3.6-35b-a3b:high PASS 100% 65 in → 953 out (941 reasoning)
git clean -fd
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 65 in → 729 out (731 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
qwen/qwen3.6-35b-a3b:low PASS 100% 65 in → 948 out (941 reasoning)
git clean -fd
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 65 in → 957 out (925 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 65 in → 935 out (926 reasoning)
git clean -fd
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 65 in → 775 out (737 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
qwen/qwen3.6-35b-a3b:none PASS 100% 67 in → 5 out (0 reasoning)
git clean -fd
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 67 in → 15 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 65 in → 721 out (712 reasoning)
git clean -fd
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 65 in → 720 out (696 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
qwen/qwen3.6-flash:high PASS 100% 65 in → 653 out (644 reasoning)
git clean -fd
qwen/qwen3.6-flash:low PASS 100% 65 in → 321 out (312 reasoning)
git clean -fd
qwen/qwen3.6-flash:medium PASS 100% 65 in → 326 out (317 reasoning)
git clean -fd
qwen/qwen3.6-flash:none PASS 100% 67 in → 4 out (0 reasoning)
git clean -fd
qwen/qwen3.6-flash:xhigh PASS 100% 65 in → 810 out (801 reasoning)
git clean -fd
qwen/qwen3.7-flash:high PASS 100% 65 in → 493 out (484 reasoning)
git clean -fd
qwen/qwen3.7-flash:low PASS 100% 65 in → 304 out (295 reasoning)
git clean -fd
qwen/qwen3.7-flash:low__json_schema PASS 100% 65 in → 701 out (692 reasoning)
git clean -fd
qwen/qwen3.7-flash:max PASS 100% 65 in → 384 out (375 reasoning)
git clean -fd
qwen/qwen3.7-flash:medium PASS 100% 65 in → 897 out (888 reasoning)
git clean -fd
qwen/qwen3.7-flash:none PASS 100% 67 in → 4 out (0 reasoning)
git clean -fd
qwen/qwen3.7-flash:xhigh PASS 100% 65 in → 761 out (752 reasoning)
git clean -fd
qwen/qwen3.7-max:high PASS 100% 65 in → 786 out (777 reasoning)
git clean -fd
qwen/qwen3.7-max:low PASS 100% 65 in → 693 out (684 reasoning)
git clean -fd
qwen/qwen3.7-max:medium PASS 100% 65 in → 763 out (754 reasoning)
git clean -fd
qwen/qwen3.7-max:none PASS 100% 67 in → 4 out (0 reasoning)
git clean -fd
qwen/qwen3.7-max:xhigh PASS 100% 65 in → 208 out (199 reasoning)
git clean -fd
qwen/qwen3.7-plus:high PASS 100% 65 in → 601 out (592 reasoning)
git clean -fd
qwen/qwen3.7-plus:low PASS 100% 65 in → 570 out (561 reasoning)
git clean -fd
qwen/qwen3.7-plus:medium PASS 100% 65 in → 595 out (586 reasoning)
git clean -fd
qwen/qwen3.7-plus:none PASS 100% 67 in → 4 out (0 reasoning)
git clean -fd
qwen/qwen3.7-plus:xhigh PASS 100% 65 in → 653 out (644 reasoning)
git clean -fd
tencent/hy3:high PASS 100% 62 in → 367 out (361 reasoning)
git clean -fd
tencent/hy3:high__json_schema PASS 100% 62 in → 350 out (348 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
tencent/hy3:low PASS 100% 62 in → 375 out (369 reasoning)
git clean -fd
tencent/hy3:low__json_schema PASS 100% 62 in → 260 out (280 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
tencent/hy3:medium PASS 100% 62 in → 430 out (452 reasoning)
git clean -fd
tencent/hy3:medium__json_schema PASS 100% 62 in → 226 out (224 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
tencent/hy3:none PASS 100% 65 in → 5 out (0 reasoning)
git clean -fd
tencent/hy3:none__json_schema PASS 100% 65 in → 12 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
tencent/hy3:xhigh PASS 100% 62 in → 267 out (261 reasoning)
git clean -fd
tencent/hy3:xhigh__json_schema PASS 100% 62 in → 324 out (334 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
thinkingmachines/inkling-small:high PASS 100% 63 in → 72 out (74 reasoning)
git clean -fd
thinkingmachines/inkling-small:high__json_schema PASS 100% 63 in → 97 out (83 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
thinkingmachines/inkling-small:low PASS 100% 63 in → 20 out (10 reasoning)
git clean -fd
thinkingmachines/inkling-small:low__json_schema PASS 100% 63 in → 50 out (34 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
thinkingmachines/inkling-small:max PASS 100% 63 in → 89 out (79 reasoning)
git clean -fd
thinkingmachines/inkling-small:max__json_schema PASS 100% 63 in → 120 out (103 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
thinkingmachines/inkling-small:medium PASS 100% 63 in → 81 out (71 reasoning)
git clean -fd
thinkingmachines/inkling-small:medium__json_schema PASS 100% 63 in → 89 out (73 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
thinkingmachines/inkling-small:xhigh PASS 100% 63 in → 41 out (31 reasoning)
git clean -fd
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 63 in → 78 out (61 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
thinkingmachines/inkling:high PASS 100% 63 in → 78 out (78 reasoning)
git clean -fd
thinkingmachines/inkling:high__json_schema PASS 100% 63 in → 106 out (89 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
thinkingmachines/inkling:low PASS 100% 63 in → 20 out (9 reasoning)
git clean -fd
thinkingmachines/inkling:max PASS 100% 63 in → 174 out (163 reasoning)
git clean -fd
thinkingmachines/inkling:max__json_schema PASS 100% 63 in → 88 out (75 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
thinkingmachines/inkling:medium PASS 100% 63 in → 41 out (30 reasoning)
git clean -fd
thinkingmachines/inkling:none PASS 100% 61 in → 7 out (0 reasoning)
git clean -fd
thinkingmachines/inkling:xhigh PASS 100% 63 in → 123 out (112 reasoning)
git clean -fd
thinkingmachines/inkling:xhigh__json_schema PASS 100% 63 in → 116 out (102 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
x-ai/grok-4.3:high PASS 100% 233 in → 143 out (139 reasoning)
git clean -fd
x-ai/grok-4.3:high__json_schema PASS 100% 289 in → 406 out (394 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
x-ai/grok-4.3:low PASS 100% 239 in → 195 out (191 reasoning)
git clean -fd
x-ai/grok-4.3:low__json_schema PASS 100% 295 in → 254 out (242 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
x-ai/grok-4.3:max PASS 100% 233 in → 102 out (98 reasoning)
git clean -fd
x-ai/grok-4.3:max__json_schema PASS 100% 289 in → 350 out (342 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
x-ai/grok-4.3:medium PASS 100% 239 in → 390 out (386 reasoning)
git clean -fd
x-ai/grok-4.3:medium__json_schema PASS 100% 295 in → 419 out (407 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
x-ai/grok-4.3:none PASS 100% 231 in → 4 out (0 reasoning)
git clean -fd
x-ai/grok-4.3:none__json_schema PASS 100% 287 in → 8 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
x-ai/grok-4.3:xhigh PASS 100% 233 in → 346 out (340 reasoning)
`git clean -fd`
x-ai/grok-4.3:xhigh__json_schema PASS 100% 289 in → 550 out (538 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
x-ai/grok-4.5:high PASS 100% 257 in → 221 out (217 reasoning)
git clean -fd
x-ai/grok-4.5:high__json_schema PASS 100% 323 in → 247 out (238 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
x-ai/grok-4.5:low PASS 100% 257 in → 103 out (95 reasoning)
``` git clean -fd ```
x-ai/grok-4.5:low__json_schema PASS 100% 323 in → 88 out (79 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
x-ai/grok-4.5:max PASS 100% 257 in → 155 out (151 reasoning)
git clean -fd
x-ai/grok-4.5:max__json_schema PASS 100% 323 in → 430 out (421 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
x-ai/grok-4.5:medium PASS 100% 257 in → 370 out (366 reasoning)
git clean -fd
x-ai/grok-4.5:medium__json_schema PASS 100% 323 in → 240 out (231 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
x-ai/grok-4.5:xhigh PASS 100% 257 in → 212 out (204 reasoning)
``` git clean -fd ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 323 in → 318 out (309 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command":"git clean -fd"}
z-ai/glm-4.7-flash:high PASS 100% 54 in → 473 out (487 reasoning)
git clean -df
z-ai/glm-4.7-flash:high__json_schema PASS 100% 54 in → 541 out (529 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
z-ai/glm-4.7-flash:low PASS 100% 54 in → 311 out (307 reasoning)
git clean -fd
z-ai/glm-4.7-flash:medium PASS 100% 54 in → 461 out (480 reasoning)
git clean -fd
z-ai/glm-4.7-flash:none PASS 100% 54 in → 5 out (0 reasoning)
git clean -fdx
z-ai/glm-4.7-flash:none__json_schema PASS 100% 54 in → 10 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
z-ai/glm-4.7-flash:xhigh PASS 100% 54 in → 455 out (467 reasoning)
git clean -fd
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 54 in → 446 out (439 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
z-ai/glm-5.2:high PASS 100% 61 in → 306 out (300 reasoning)
git clean -fd
z-ai/glm-5.2:low PASS 100% 61 in → 271 out (265 reasoning)
git clean -fd
z-ai/glm-5.2:low__json_schema PASS 100% 61 in → 201 out (190 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
z-ai/glm-5.2:medium PASS 100% 61 in → 189 out (183 reasoning)
git clean -fd
z-ai/glm-5.2:medium__json_schema PASS 100% 61 in → 376 out (363 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
z-ai/glm-5.2:none PASS 100% 55 in → 5 out (0 reasoning)
git clean -fd
z-ai/glm-5.2:xhigh PASS 100% 61 in → 359 out (353 reasoning)
git clean -fd
z-ai/glm-5.2:xhigh__json_schema PASS 100% 61 in → 403 out (390 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
arcee-ai/trinity-mini:high FAIL 33.3% 58 in → 384 out (428 reasoning)
git clean-fd
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'build'}, {'type': 'dir_not_exists', 'path': 'tmp'}]
liquid/lfm-2-24b-a2b FAIL 33.3% 67 in → 13 out
git rm -r --ignore-uncaught build tmp
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'build'}, {'type': 'dir_not_exists', 'path': 'tmp'}]
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 33.3% 67 in → 387 out (429 reasoning)
git clean -rdf
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'build'}, {'type': 'dir_not_exists', 'path': 'tmp'}]
openai/gpt-5.4-nano:none FAIL 33.3% 54 in → 14 out (0 reasoning)
git clean -fdx --remove-untracked-files
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'build'}, {'type': 'dir_not_exists', 'path': 'tmp'}]
thinkingmachines/inkling-small:none FAIL 33.3% 63 in → 7 out (0 reasoning)
git clean -fd<|end_message|>
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'build'}, {'type': 'dir_not_exists', 'path': 'tmp'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: {{ "command": "git clean -fd" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Invalid JSON. Output: git clean -fd
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git rm -r --ignore-unchanged build/output/app.bin tmp/cache.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f9d7adf1-9c25-90e7-aaae-3e71c5359ea9","request_id":"f9d7adf1-9c25-90e7-aaae-3e71c5359ea9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2ed398ac-4b15-9b7b-870e-2117f9ed925e","request_id":"2ed398ac-4b15-9b7b-870e-2117f9ed925e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-811a2f9f-cdac-9c26-a433-3ffcb53fa863","request_id":"811a2f9f-cdac-9c26-a433-3ffcb53fa863"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3014f862-f42b-9472-a6f6-446ee4686001","request_id":"3014f862-f42b-9472-a6f6-446ee4686001"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-17aec49e-1bb4-9905-a7ef-0c3ff283f801","request_id":"17aec49e-1bb4-9905-a7ef-0c3ff283f801"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-76aab219-2074-9887-85ab-1be69d3c5cb7"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-353d8172-8b22-9847-ab76-9d1c57a11eea"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9e9c37a2-2b79-9e7b-95e4-1d761154a41d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-32c3e48f-0059-9a5c-a86c-3de67d4b1aa6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-493741f1-e258-9821-8d4d-34d392d808c6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-042d6cc5-015a-938b-81fd-dccc39bc8f7f","request_id":"042d6cc5-015a-938b-81fd-dccc39bc8f7f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-65d2e96c-a983-9803-8c62-e2f368d2c2a0","request_id":"65d2e96c-a983-9803-8c62-e2f368d2c2a0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cc761564-da67-92f4-9196-379306280ac5","request_id":"cc761564-da67-92f4-9196-379306280ac5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fb77d287-416d-9541-b984-739b18d0cab5","request_id":"fb77d287-416d-9541-b984-739b18d0cab5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ba1d2cd5-c333-934c-80ff-de1b02c3ab76","request_id":"ba1d2cd5-c333-934c-80ff-de1b02c3ab76"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e6c2c8de-5d68-9aab-ae02-0debe0d3e808","request_id":"e6c2c8de-5d68-9aab-ae02-0debe0d3e808"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-356f1dcf-8ad8-985c-ae2f-e506ff91878a","request_id":"356f1dcf-8ad8-985c-ae2f-e506ff91878a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2bd48b7b-952e-96eb-9627-af977c0728e3","request_id":"2bd48b7b-952e-96eb-9627-af977c0728e3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ad140a08-a765-9a4a-b1ae-f8139d8c730a","request_id":"ad140a08-a765-9a4a-b1ae-f8139d8c730a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cf5d4fa7-fb31-9d1a-9bf6-db834299a7fa","request_id":"cf5d4fa7-fb31-9d1a-9bf6-db834299a7fa"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: git clean -fd{"command": "git clean -fd"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd{ "command": "git clean -fd" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Model 'z-ai/glm-5.2:none' violated the reasoning_level=none invariant: provider reported 296 reasoning token(s)