Exclude specific files from cleaning
Tests ability to exclude specific files from cleaning with git clean -e. Evaluates exclusion-pattern usage.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'tracked' > tracked.txt
  5. 05 git add tracked.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 echo 'important' > important.tmp
  8. 08 echo 'junk1' > junk1.tmp
  9. 09 echo 'junk2' > junk2.txt
Prompt
Remove all untracked files except important.tmp. Output ONLY the git command, nothing else.
Expected
git clean -f -e important.tmp

Scoped model quality, cost, API time, and token usage for git_clean/f007.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 106 in → 54 out (35 reasoning)
git clean -f -e important.tmp
anthropic/claude-fable-5:high__json_schema PASS 100% 316 in → 45 out (35 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
anthropic/claude-fable-5:low PASS 100% 106 in → 43 out (39 reasoning)
git clean -f -e important.tmp
anthropic/claude-fable-5:low__json_schema PASS 100% 316 in → 21 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
anthropic/claude-fable-5:max PASS 100% 106 in → 313 out (45 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-fable-5:max__json_schema PASS 100% 316 in → 310 out (91 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
anthropic/claude-fable-5:medium PASS 100% 106 in → 38 out (35 reasoning)
git clean -f -e important.tmp
anthropic/claude-fable-5:medium__json_schema PASS 100% 316 in → 45 out (35 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
anthropic/claude-fable-5:xhigh PASS 100% 106 in → 117 out (49 reasoning)
git clean -f -e important.tmp
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 316 in → 45 out (30 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
anthropic/claude-haiku-4.5:high PASS 100% 114 in → 240 out (207 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 277 in → 217 out (186 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
anthropic/claude-haiku-4.5:low PASS 100% 114 in → 264 out (217 reasoning)
``` git clean -f junk1.tmp junk2.txt ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 277 in → 255 out (209 reasoning)
git clean -f -d -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -d -e important.tmp"}
anthropic/claude-haiku-4.5:medium PASS 100% 114 in → 309 out (270 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 277 in → 223 out (184 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fd -e important.tmp"}
anthropic/claude-haiku-4.5:none PASS 100% 85 in → 18 out (0 reasoning)
```bash git clean -fd --exclude=important.tmp ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 247 in → 18 out (0 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f --exclude=important.tmp"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 114 in → 293 out (251 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 277 in → 384 out (335 reasoning)
git clean -f -d --exclude='important.tmp'
JSON Schema Structured Output
(raw) {"command": "git clean -f -d --exclude='important.tmp'"}
anthropic/claude-opus-4.6:high PASS 100% 85 in → 144 out (101 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 248 in → 146 out (102 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.6:low PASS 100% 85 in → 17 out (0 reasoning)
``` git clean -f --exclude=important.tmp ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 248 in → 17 out (0 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f --exclude=important.tmp"}
anthropic/claude-opus-4.6:max PASS 100% 85 in → 127 out (87 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 248 in → 159 out (116 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.6:medium PASS 100% 85 in → 18 out (0 reasoning)
``` git clean -f --exclude=important.tmp ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 248 in → 49 out (20 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.6:none PASS 100% 85 in → 17 out (0 reasoning)
``` git clean -f --exclude=important.tmp ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 248 in → 17 out (0 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f --exclude=important.tmp"}
anthropic/claude-opus-4.6:xhigh PASS 100% 85 in → 134 out (91 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 248 in → 134 out (91 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.7:high PASS 100% 111 in → 18 out (0 reasoning)
`git clean -f -e important.tmp`
anthropic/claude-opus-4.7:high__json_schema PASS 100% 321 in → 23 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.7:low PASS 100% 111 in → 16 out (0 reasoning)
git clean -f -e important.tmp
anthropic/claude-opus-4.7:low__json_schema PASS 100% 321 in → 23 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.7:max PASS 100% 111 in → 21 out (0 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 321 in → 23 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.7:medium PASS 100% 111 in → 16 out (0 reasoning)
git clean -f -e important.tmp
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 321 in → 23 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.7:none PASS 100% 111 in → 18 out (0 reasoning)
`git clean -f -e important.tmp`
anthropic/claude-opus-4.7:none__json_schema PASS 100% 321 in → 23 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.7:xhigh PASS 100% 111 in → 18 out (0 reasoning)
`git clean -f -e important.tmp`
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 321 in → 23 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.8:high PASS 100% 106 in → 14 out (0 reasoning)
git clean -f -e important.tmp
anthropic/claude-opus-4.8:high__json_schema PASS 100% 316 in → 21 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.8:low PASS 100% 106 in → 18 out (0 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 316 in → 21 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.8:max PASS 100% 106 in → 285 out (46 reasoning)
`git clean -f -e important.tmp`
anthropic/claude-opus-4.8:max__json_schema PASS 100% 316 in → 210 out (40 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
anthropic/claude-opus-4.8:medium PASS 100% 106 in → 18 out (0 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 316 in → 21 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.8:none PASS 100% 106 in → 18 out (0 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 316 in → 20 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-opus-4.8:xhigh PASS 100% 106 in → 18 out (0 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 316 in → 21 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-sonnet-4.6:high PASS 100% 85 in → 97 out (71 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 248 in → 122 out (86 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-sonnet-4.6:low PASS 100% 85 in → 17 out (0 reasoning)
``` git clean -f --exclude=important.tmp ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 248 in → 17 out (0 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f --exclude=important.tmp"}
anthropic/claude-sonnet-4.6:max PASS 100% 85 in → 136 out (102 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 248 in → 276 out (163 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-sonnet-4.6:medium PASS 100% 85 in → 17 out (0 reasoning)
``` git clean -f --exclude=important.tmp ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 248 in → 17 out (0 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f --exclude=important.tmp"}
anthropic/claude-sonnet-4.6:none PASS 100% 85 in → 17 out (0 reasoning)
``` git clean -f --exclude=important.tmp ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 248 in → 17 out (0 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f --exclude=important.tmp"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 85 in → 111 out (79 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 248 in → 109 out (77 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-sonnet-5:high PASS 100% 106 in → 18 out (0 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 316 in → 21 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
anthropic/claude-sonnet-5:low PASS 100% 106 in → 14 out (0 reasoning)
git clean -f -e important.tmp
anthropic/claude-sonnet-5:low__json_schema PASS 100% 316 in → 22 out (0 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
anthropic/claude-sonnet-5:medium PASS 100% 106 in → 19 out (0 reasoning)
``` git clean -fd -e important.tmp ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 316 in → 22 out (0 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
anthropic/claude-sonnet-5:none PASS 100% 106 in → 19 out (0 reasoning)
``` git clean -fd -e important.tmp ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 316 in → 21 out (0 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fd -e important.tmp"}
anthropic/claude-sonnet-5:xhigh PASS 100% 106 in → 18 out (0 reasoning)
``` git clean -f -e important.tmp ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 316 in → 22 out (0 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fd -e important.tmp"}
arcee-ai/trinity-large-thinking:high PASS 100% 65 in → 622 out (612 reasoning)
git clean -f -e important.tmp
arcee-ai/trinity-large-thinking:low PASS 100% 65 in → 526 out (516 reasoning)
git clean -f -e important.tmp
arcee-ai/trinity-large-thinking:medium PASS 100% 65 in → 3,939 out (3,969 reasoning)
git clean -f -e important.tmp
arcee-ai/trinity-large-thinking:xhigh PASS 100% 65 in → 648 out (637 reasoning)
git clean -f -e important.tmp
arcee-ai/trinity-mini:high PASS 100% 65 in → 480 out (530 reasoning)
git clean -f -e important.tmp
arcee-ai/trinity-mini:low PASS 100% 65 in → 341 out (373 reasoning)
git clean -f -e important.tmp
arcee-ai/trinity-mini:low__json_schema PASS 100% 65 in → 364 out (400 reasoning)
git clean -f -eimportant.tmp
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 65 in → 391 out (428 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash-0731:high PASS 100% 148 in → 211 out (206 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 69 in → 376 out (366 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 69 in → 341 out (325 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 69 in → 22 out (0 reasoning)
git clean -f -e important.tmp -x
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp -x" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 161 in → 4,235 out (4,198 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 347 in → 660 out (645 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 69 in → 352 out (341 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 69 in → 252 out (239 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 69 in → 10 out (0 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 69 in → 18 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 69 in → 389 out (380 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 69 in → 301 out (274 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
deepseek/deepseek-v4-flash:high PASS 100% 69 in → 230 out (208 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 161 in → 14 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
deepseek/deepseek-v4-flash:low PASS 100% 69 in → 321 out (298 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 161 in → 14 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
deepseek/deepseek-v4-flash:medium PASS 100% 69 in → 300 out (288 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash:none PASS 100% 69 in → 14 out (0 reasoning)
```bash git clean -f -e important.tmp ```
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 69 in → 18 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 148 in → 919 out (908 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 150 in → 940 out (922 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
deepseek/deepseek-v4-pro:high PASS 100% 69 in → 317 out (306 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 71 in → 290 out (271 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
deepseek/deepseek-v4-pro:low PASS 100% 69 in → 181 out (170 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 256 in → 429 out (413 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
deepseek/deepseek-v4-pro:medium PASS 100% 69 in → 239 out (229 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 71 in → 349 out (330 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
deepseek/deepseek-v4-pro:none PASS 100% 69 in → 9 out (0 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 69 in → 18 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 148 in → 651 out (641 reasoning)
git clean -f -e important.tmp
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 334 in → 594 out (578 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3-flash-preview:high PASS 100% 69 in → 298 out (289 reasoning)
git clean -f -e important.tmp
google/gemini-3-flash-preview:high__json_schema PASS 100% 126 in → 247 out (229 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
google/gemini-3-flash-preview:low PASS 100% 69 in → 148 out (139 reasoning)
git clean -f -e important.tmp
google/gemini-3-flash-preview:low__json_schema PASS 100% 126 in → 281 out (268 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
google/gemini-3-flash-preview:medium PASS 100% 69 in → 232 out (223 reasoning)
git clean -f -e important.tmp
google/gemini-3-flash-preview:medium__json_schema PASS 100% 126 in → 236 out (222 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3-flash-preview:xhigh PASS 100% 69 in → 325 out (316 reasoning)
git clean -f -e important.tmp
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 126 in → 243 out (230 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 69 in → 280 out (271 reasoning)
git clean -f -e important.tmp
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 70 in → 308 out (294 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 70 in → 127 out (118 reasoning)
git clean -f -e important.tmp
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 126 in → 122 out (108 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 69 in → 336 out (327 reasoning)
git clean -f -e important.tmp
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 70 in → 217 out (203 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 69 in → 359 out (350 reasoning)
git clean -f -e important.tmp
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 126 in → 500 out (486 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fd -e important.tmp"}
google/gemini-3.1-pro-preview:high PASS 100% 69 in → 524 out (515 reasoning)
git clean -f -e important.tmp
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 126 in → 386 out (373 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
google/gemini-3.1-pro-preview:low PASS 100% 69 in → 285 out (276 reasoning)
git clean -f -e important.tmp
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 126 in → 215 out (201 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3.1-pro-preview:medium PASS 100% 69 in → 225 out (216 reasoning)
git clean -f -e important.tmp
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 126 in → 104 out (91 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 69 in → 409 out (400 reasoning)
git clean -f -e important.tmp
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 126 in → 316 out (303 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
google/gemini-3.5-flash-lite:high PASS 100% 69 in → 288 out (279 reasoning)
git clean -f -e important.tmp
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 70 in → 423 out (409 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3.5-flash-lite:low PASS 100% 69 in → 9 out (0 reasoning)
git clean -f -e important.tmp
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 126 in → 19 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
google/gemini-3.5-flash-lite:max PASS 100% 70 in → 548 out (539 reasoning)
git clean -f -e important.tmp
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 126 in → 425 out (406 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
google/gemini-3.5-flash-lite:medium PASS 100% 70 in → 369 out (360 reasoning)
git clean -f -e important.tmp
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 70 in → 290 out (276 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 69 in → 338 out (329 reasoning)
git clean -f -e important.tmp
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 126 in → 319 out (305 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fd -e important.tmp"}
google/gemini-3.5-flash:high PASS 100% 69 in → 606 out (597 reasoning)
git clean -f -e important.tmp
google/gemini-3.5-flash:high__json_schema PASS 100% 126 in → 270 out (256 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3.5-flash:low PASS 100% 69 in → 259 out (250 reasoning)
git clean -f -e important.tmp
google/gemini-3.5-flash:low__json_schema PASS 100% 126 in → 159 out (145 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3.5-flash:medium PASS 100% 69 in → 382 out (373 reasoning)
git clean -f -e important.tmp
google/gemini-3.5-flash:medium__json_schema PASS 100% 126 in → 569 out (556 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
google/gemini-3.5-flash:xhigh PASS 100% 69 in → 509 out (500 reasoning)
git clean -f -e important.tmp
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 126 in → 424 out (410 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemini-3.6-flash:high PASS 100% 69 in → 454 out (445 reasoning)
git clean -f -e important.tmp
google/gemini-3.6-flash:high__json_schema PASS 100% 70 in → 374 out (361 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
google/gemini-3.6-flash:low PASS 100% 70 in → 224 out (215 reasoning)
git clean -f -e important.tmp
google/gemini-3.6-flash:low__json_schema PASS 100% 70 in → 19 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
google/gemini-3.6-flash:max PASS 100% 69 in → 328 out (319 reasoning)
git clean -f -e important.tmp
google/gemini-3.6-flash:max__json_schema PASS 100% 70 in → 486 out (473 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
google/gemini-3.6-flash:medium PASS 100% 70 in → 290 out (281 reasoning)
git clean -f -e important.tmp
google/gemini-3.6-flash:medium__json_schema PASS 100% 70 in → 412 out (399 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
google/gemini-3.6-flash:xhigh PASS 100% 69 in → 604 out (595 reasoning)
git clean -f -e important.tmp
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 126 in → 306 out (293 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
google/gemma-4-26b-a4b-it:high PASS 100% 85 in → 285 out (234 reasoning)
git clean -f -e important.tmp
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 85 in → 290 out (224 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
google/gemma-4-26b-a4b-it:low PASS 100% 85 in → 429 out (370 reasoning)
git clean -f -e important.tmp
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 84 in → 267 out (205 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
google/gemma-4-26b-a4b-it:medium PASS 100% 84 in → 313 out (271 reasoning)
git clean -f -e important.tmp
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 84 in → 271 out (223 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
google/gemma-4-26b-a4b-it:none PASS 100% 82 in → 10 out (0 reasoning)
git clean -f -e important.tmp
google/gemma-4-26b-a4b-it:xhigh PASS 100% 85 in → 370 out (318 reasoning)
git clean -f -e important.tmp
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 85 in → 294 out (236 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
google/gemma-4-31b-it:high PASS 100% 85 in → 200 out (1 reasoning)
git clean -f -e important.tmp
google/gemma-4-31b-it:high__json_schema PASS 100% 85 in → 186 out (148 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
google/gemma-4-31b-it:low PASS 100% 85 in → 214 out (183 reasoning)
git clean -f -e important.tmp
google/gemma-4-31b-it:low__json_schema PASS 100% 85 in → 236 out (1 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
google/gemma-4-31b-it:medium PASS 100% 85 in → 234 out (188 reasoning)
git clean -f -e important.tmp
google/gemma-4-31b-it:medium__json_schema PASS 100% 84 in → 241 out (188 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
google/gemma-4-31b-it:none PASS 100% 82 in → 10 out (0 reasoning)
git clean -fd -e important.tmp
google/gemma-4-31b-it:none__json_schema PASS 100% 82 in → 20 out (0 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd -e important.tmp" }
google/gemma-4-31b-it:xhigh PASS 100% 84 in → 186 out (155 reasoning)
git clean -f -e important.tmp
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 88 in → 219 out (204 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
inclusionai/ling-2.6-flash PASS 100% 83 in → 14 out
```sh git clean -f --exclude important.tmp ```
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f --exclude=important.tmp" }
minimax/minimax-m2.5:high PASS 100% 100 in → 201 out (192 reasoning)
git clean -fe important.tmp
minimax/minimax-m2.5:high__json_schema PASS 100% 100 in → 1,944 out (1,971 reasoning)
git clean -fd -f -- junk1.tmp junk2.txt
JSON Schema Structured Output
(raw) { "command": "git clean -fd -f -- junk1.tmp junk2.txt" }
minimax/minimax-m2.5:low PASS 100% 100 in → 297 out (275 reasoning)
git clean -f junk1.tmp junk2.txt
minimax/minimax-m2.5:low__json_schema PASS 100% 100 in → 905 out (989 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
minimax/minimax-m2.5:medium PASS 100% 100 in → 203 out (212 reasoning)
git clean -f -e important.tmp
minimax/minimax-m2.5:xhigh PASS 100% 110 in → 163 out (152 reasoning)
git clean -f -e important.tmp
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 100 in → 365 out (338 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
minimax/minimax-m2.7:high PASS 100% 103 in → 697 out (687 reasoning)
git clean -fd -e important.tmp
minimax/minimax-m2.7:low PASS 100% 100 in → 812 out (801 reasoning)
git clean -fd -e important.tmp
minimax/minimax-m2.7:medium PASS 100% 100 in → 617 out (595 reasoning)
git clean -f -e important.tmp
minimax/minimax-m2.7:medium__json_schema PASS 100% 232 in → 412 out (393 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd -e important.tmp" }
minimax/minimax-m2.7:xhigh PASS 100% 100 in → 564 out (552 reasoning)
git clean -f -e important.tmp
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 232 in → 4,827 out (4,811 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
minimax/minimax-m3:high PASS 100% 238 in → 159 out (151 reasoning)
git clean -f -e important.tmp
minimax/minimax-m3:low PASS 100% 238 in → 236 out (239 reasoning)
git clean -f -e important.tmp
minimax/minimax-m3:low__json_schema PASS 100% 225 in → 17 out (0 reasoning)
git clean -f -- ':!important.tmp'
JSON Schema Structured Output
(raw) {"command": "git clean -f -- ':!important.tmp'"}
minimax/minimax-m3:medium PASS 100% 238 in → 82 out (78 reasoning)
git clean -f -e important.tmp
minimax/minimax-m3:xhigh PASS 100% 238 in → 103 out (106 reasoning)
git clean -f -e important.tmp
mistralai/devstral-2512 PASS 100% 68 in → 15 out
```bash git clean -f -e important.tmp ```
mistralai/devstral-2512__json_schema PASS 100% 68 in → 15 out
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
mistralai/mistral-medium-3-5:high PASS 100% 80 in → 909 out (960 reasoning)
git clean -f --exclude=important.tmp
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 68 in → 1,161 out (1,217 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
mistralai/mistral-medium-3-5:low PASS 100% 80 in → 4,474 out (4,822 reasoning)
git clean -fd -e important.tmp
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 68 in → 328 out (315 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
mistralai/mistral-medium-3-5:medium PASS 100% 80 in → 2,665 out (2,911 reasoning)
git clean -f -e important.tmp
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 68 in → 521 out (535 reasoning)
git clean -fd --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fd --exclude=important.tmp"}
mistralai/mistral-medium-3-5:none PASS 100% 80 in → 13 out (0 reasoning)
git clean -f -x --exclude=important.tmp
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 68 in → 16 out (0 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f --exclude=important.tmp"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 80 in → 775 out (796 reasoning)
git clean -f -e important.tmp
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 68 in → 651 out (673 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f --exclude=important.tmp"}
moonshotai/kimi-k2.6:high PASS 100% 65 in → 1,294 out (1,284 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k2.6:high__json_schema PASS 100% 247 in → 877 out (863 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
moonshotai/kimi-k2.6:low PASS 100% 65 in → 1,244 out (1,234 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k2.6:low__json_schema PASS 100% 64 in → 1,044 out (931 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
moonshotai/kimi-k2.6:medium PASS 100% 65 in → 1,409 out (1,400 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 64 in → 944 out (669 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
moonshotai/kimi-k2.6:xhigh PASS 100% 64 in → 937 out (927 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 65 in → 478 out (465 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
moonshotai/kimi-k2.7-code:high PASS 100% 64 in → 219 out (209 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 65 in → 120 out (114 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
moonshotai/kimi-k2.7-code:low PASS 100% 64 in → 132 out (121 reasoning)
git clean -fd --exclude=important.tmp
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 148 in → 93 out (78 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
moonshotai/kimi-k2.7-code:medium PASS 100% 65 in → 474 out (499 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 64 in → 382 out (106 reasoning)
git clean -fd --exclude=important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd --exclude=important.tmp" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 64 in → 159 out (149 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 148 in → 142 out (127 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fd -e important.tmp"}
moonshotai/kimi-k3:high PASS 100% 142 in → 357 out (334 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k3:high__json_schema PASS 100% 142 in → 332 out (306 reasoning)
``` git clean -fd -e important.tmp ```
moonshotai/kimi-k3:low PASS 100% 142 in → 201 out (178 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k3:low__json_schema PASS 100% 142 in → 143 out (120 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k3:max PASS 100% 142 in → 141 out (118 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k3:medium PASS 100% 142 in → 109 out (86 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k3:medium__json_schema PASS 100% 142 in → 333 out (310 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k3:xhigh PASS 100% 142 in → 251 out (228 reasoning)
git clean -f -e important.tmp
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 142 in → 554 out (531 reasoning)
git clean -f -e important.tmp
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 81 in → 714 out (768 reasoning)
git clean -fd --exclude=important.tmp
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 81 in → 252 out (240 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f --exclude=important.tmp" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 81 in → 307 out (306 reasoning)
git clean -fdx --exclude=important.tmp .
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 81 in → 350 out (371 reasoning)
git clean -f --exclude=important.tmp
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 81 in → 414 out (402 reasoning)
git clean -fd --exclude=important.tmp .
JSON Schema Structured Output
(raw) { "command": "git clean -fd --exclude=important.tmp ." }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git clean -fd --exclude=important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd --exclude=important.tmp" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 81 in → 3,510 out (3,848 reasoning)
git clean -f -e important.tmp
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 81 in → 403 out (393 reasoning)
git clean -fd --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fd --exclude=important.tmp" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 81 in → 206 out (189 reasoning)
git clean -f -e important.tmp
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 81 in → 247 out (213 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 81 in → 221 out (191 reasoning)
git clean -f -e important.tmp
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 81 in → 229 out (201 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 81 in → 269 out (239 reasoning)
git clean -f -e important.tmp
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 81 in → 404 out (360 reasoning)
git clean -f -e "important.tmp"
JSON Schema Structured Output
(raw) {"command": "git clean -f -e \"important.tmp\""}
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 81 in → 12 out (0 reasoning)
git clean -f -x -e important.tmp
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 81 in → 25 out (0 reasoning)
git clean -fX && git clean -fd --exclude=important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fX && git clean -fd --exclude=important.tmp" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 81 in → 267 out (254 reasoning)
git clean -f -e important.tmp
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 81 in → 174 out (144 reasoning)
git clean -f -e 'important.tmp'
JSON Schema Structured Output
(raw) { "command": "git clean -f -e 'important.tmp'" }
openai/gpt-5.4-mini:high PASS 100% 61 in → 156 out (142 reasoning)
git clean -f -e important.tmp
openai/gpt-5.4-mini:high__json_schema PASS 100% 96 in → 154 out (132 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.4-mini:low PASS 100% 61 in → 45 out (31 reasoning)
git clean -f -e important.tmp
openai/gpt-5.4-mini:low__json_schema PASS 100% 96 in → 86 out (64 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.4-mini:medium PASS 100% 61 in → 101 out (87 reasoning)
git clean -f -e important.tmp
openai/gpt-5.4-mini:medium__json_schema PASS 100% 96 in → 150 out (128 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
openai/gpt-5.4-mini:none PASS 100% 61 in → 12 out (0 reasoning)
git clean -f -e important.tmp
openai/gpt-5.4-mini:none__json_schema PASS 100% 96 in → 20 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.4-mini:xhigh PASS 100% 61 in → 202 out (188 reasoning)
git clean -f -e important.tmp
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 96 in → 538 out (516 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.4-nano:high PASS 100% 61 in → 130 out (115 reasoning)
git clean -fd --exclude=important.tmp
openai/gpt-5.4-nano:high__json_schema PASS 100% 96 in → 20 out (0 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
openai/gpt-5.4-nano:low__json_schema PASS 100% 96 in → 20 out (0 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
openai/gpt-5.4-nano:medium PASS 100% 61 in → 95 out (81 reasoning)
git clean -fd -e important.tmp
openai/gpt-5.4-nano:medium__json_schema PASS 100% 96 in → 105 out (83 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
openai/gpt-5.4-nano:none PASS 100% 61 in → 13 out (0 reasoning)
git clean -fd --exclude=important.tmp
openai/gpt-5.4-nano:none__json_schema PASS 100% 96 in → 20 out (0 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
openai/gpt-5.4-nano:xhigh PASS 100% 61 in → 250 out (236 reasoning)
git clean -fd -e important.tmp
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 96 in → 185 out (163 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
openai/gpt-5.4:high PASS 100% 61 in → 215 out (201 reasoning)
git clean -f -e important.tmp
openai/gpt-5.4:high__json_schema PASS 100% 96 in → 123 out (101 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.4:low PASS 100% 61 in → 77 out (63 reasoning)
git clean -f -e important.tmp
openai/gpt-5.4:low__json_schema PASS 100% 96 in → 87 out (62 reasoning)
git clean -f -- junk1.tmp junk2.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- junk1.tmp junk2.txt"}
openai/gpt-5.4:medium PASS 100% 61 in → 126 out (112 reasoning)
git clean -f -e important.tmp
openai/gpt-5.4:medium__json_schema PASS 100% 96 in → 106 out (84 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.4:none PASS 100% 61 in → 17 out (0 reasoning)
git clean -f -- . ':(exclude)important.tmp'
openai/gpt-5.4:none__json_schema PASS 100% 96 in → 25 out (0 reasoning)
git clean -f -- . ':(exclude)important.tmp'
JSON Schema Structured Output
(raw) {"command":"git clean -f -- . ':(exclude)important.tmp'"}
openai/gpt-5.4:xhigh PASS 100% 61 in → 446 out (432 reasoning)
git clean -f -e important.tmp
openai/gpt-5.4:xhigh__json_schema PASS 100% 96 in → 538 out (516 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.5:high PASS 100% 61 in → 158 out (144 reasoning)
git clean -f -e important.tmp
openai/gpt-5.5:high__json_schema PASS 100% 96 in → 89 out (67 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.5:low PASS 100% 61 in → 60 out (46 reasoning)
git clean -f -e important.tmp
openai/gpt-5.5:low__json_schema PASS 100% 96 in → 96 out (74 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.5:medium PASS 100% 61 in → 94 out (80 reasoning)
git clean -f -e important.tmp
openai/gpt-5.5:medium__json_schema PASS 100% 96 in → 139 out (117 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.5:none PASS 100% 61 in → 14 out (0 reasoning)
git clean -f -- ':!important.tmp'
openai/gpt-5.5:none__json_schema PASS 100% 96 in → 22 out (0 reasoning)
git clean -f -- ':!important.tmp'
JSON Schema Structured Output
(raw) {"command":"git clean -f -- ':!important.tmp'"}
openai/gpt-5.5:xhigh PASS 100% 61 in → 530 out (516 reasoning)
git clean -f -e important.tmp
openai/gpt-5.5:xhigh__json_schema PASS 100% 96 in → 381 out (357 reasoning)
git clean -f -- ':!important.tmp'
JSON Schema Structured Output
(raw) {"command":"git clean -f -- ':!important.tmp'"}
openai/gpt-5.6-luna:high PASS 100% 61 in → 99 out (85 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-luna:high__json_schema PASS 100% 96 in → 60 out (35 reasoning)
git clean -f -- junk1.tmp junk2.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- junk1.tmp junk2.txt"}
openai/gpt-5.6-luna:low PASS 100% 61 in → 82 out (68 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-luna:low__json_schema PASS 100% 96 in → 51 out (28 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f --exclude=important.tmp"}
openai/gpt-5.6-luna:max PASS 100% 61 in → 127 out (113 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-luna:max__json_schema PASS 100% 96 in → 538 out (516 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.6-luna:medium PASS 100% 61 in → 75 out (61 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-luna:medium__json_schema PASS 100% 96 in → 68 out (46 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.6-luna:none PASS 100% 61 in → 13 out (0 reasoning)
git clean -f --exclude=important.tmp
openai/gpt-5.6-luna:none__json_schema PASS 100% 96 in → 23 out (0 reasoning)
git clean -f -- junk1.tmp junk2.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- junk1.tmp junk2.txt"}
openai/gpt-5.6-luna:xhigh PASS 100% 61 in → 182 out (168 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 96 in → 262 out (240 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.6-sol:high PASS 100% 61 in → 67 out (53 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-sol:high__json_schema PASS 100% 96 in → 102 out (80 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.6-sol:low PASS 100% 61 in → 50 out (36 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-sol:low__json_schema PASS 100% 96 in → 119 out (97 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.6-sol:max PASS 100% 61 in → 145 out (131 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-sol:max__json_schema PASS 100% 96 in → 214 out (192 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.6-sol:medium PASS 100% 61 in → 60 out (46 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-sol:medium__json_schema PASS 100% 96 in → 92 out (70 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.6-sol:none PASS 100% 61 in → 12 out (0 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-sol:none__json_schema PASS 100% 96 in → 20 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.6-sol:xhigh PASS 100% 61 in → 68 out (54 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 96 in → 136 out (114 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.6-terra:high PASS 100% 61 in → 62 out (48 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-terra:high__json_schema PASS 100% 96 in → 86 out (61 reasoning)
git clean -f -- junk1.tmp junk2.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- junk1.tmp junk2.txt"}
openai/gpt-5.6-terra:low PASS 100% 61 in → 41 out (27 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-terra:low__json_schema PASS 100% 96 in → 23 out (0 reasoning)
git clean -f -- junk1.tmp junk2.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- junk1.tmp junk2.txt"}
openai/gpt-5.6-terra:max PASS 100% 61 in → 123 out (108 reasoning)
git clean -f --exclude=important.tmp
openai/gpt-5.6-terra:max__json_schema PASS 100% 96 in → 126 out (104 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-5.6-terra:medium PASS 100% 61 in → 42 out (28 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-terra:none PASS 100% 61 in → 12 out (0 reasoning)
git clean -f -e important.tmp
openai/gpt-5.6-terra:none__json_schema PASS 100% 96 in → 23 out (0 reasoning)
git clean -f -- junk1.tmp junk2.txt
JSON Schema Structured Output
(raw) {"command":"git clean -f -- junk1.tmp junk2.txt"}
openai/gpt-5.6-terra:xhigh PASS 100% 61 in → 78 out (61 reasoning)
git clean -f -- junk1.tmp junk2.txt
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 96 in → 83 out (60 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f --exclude=important.tmp"}
openai/gpt-oss-120b:high PASS 100% 109 in → 483 out (518 reasoning)
git clean -f -e important.tmp
openai/gpt-oss-120b:low PASS 100% 124 in → 218 out (215 reasoning)
git clean -fd -e important.tmp
openai/gpt-oss-120b:low__json_schema PASS 100% 122 in → 122 out (100 reasoning)
git clean -f -e important.tmp .
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp ."}
openai/gpt-oss-120b:medium PASS 100% 124 in → 329 out (334 reasoning)
git clean -f -e important.tmp .
openai/gpt-oss-120b:medium__json_schema PASS 100% 109 in → 232 out (231 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-oss-120b:xhigh PASS 100% 120 in → 676 out (710 reasoning)
git clean -f -e important.tmp
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 109 in → 792 out (886 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
openai/gpt-oss-20b:high PASS 100% 120 in → 681 out (750 reasoning)
git clean -f -e important.tmp
openai/gpt-oss-20b:low PASS 100% 120 in → 49 out (32 reasoning)
git clean -f -e important.tmp
openai/gpt-oss-20b:low__json_schema PASS 100% 120 in → 61 out (45 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
openai/gpt-oss-20b:medium PASS 100% 122 in → 305 out (297 reasoning)
git clean -fd -e important.tmp
openai/gpt-oss-20b:medium__json_schema PASS 100% 120 in → 404 out (401 reasoning)
git clean -f -e important.tmp -d
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp -d"}
openai/gpt-oss-20b:xhigh PASS 100% 122 in → 219 out (211 reasoning)
git clean -fd -e important.tmp
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 120 in → 570 out (600 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
poolside/laguna-m.1:high PASS 100% 74 in → 1,446 out (1,436 reasoning)
git clean -f -e important.tmp
poolside/laguna-m.1:high__json_schema PASS 100% 74 in → 1,039 out (1,019 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd -e important.tmp" }
poolside/laguna-m.1:low PASS 100% 74 in → 1,361 out (1,350 reasoning)
git clean -f -e important.tmp
poolside/laguna-m.1:low__json_schema PASS 100% 74 in → 586 out (567 reasoning)
git clean -fd --exclude=important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd --exclude=important.tmp" }
poolside/laguna-m.1:medium PASS 100% 74 in → 263 out (252 reasoning)
git clean -fd -e important.tmp
poolside/laguna-m.1:medium__json_schema PASS 100% 74 in → 444 out (429 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fd -e important.tmp"}
poolside/laguna-m.1:xhigh PASS 100% 74 in → 383 out (370 reasoning)
git clean -fd -e 'important.tmp'
poolside/laguna-m.1:xhigh__json_schema PASS 100% 74 in → 2,752 out (2,736 reasoning)
git clean -fd --exclude important.tmp -f
JSON Schema Structured Output
(raw) {"command": "git clean -fd --exclude important.tmp -f"}
poolside/laguna-xs-2.1:high PASS 100% 74 in → 514 out (503 reasoning)
git clean -fd --exclude important.tmp --force
poolside/laguna-xs-2.1:high__json_schema PASS 100% 74 in → 235 out (220 reasoning)
git clean -df -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -df -e important.tmp"}
poolside/laguna-xs-2.1:low PASS 100% 74 in → 405 out (395 reasoning)
git clean -f --exclude=important.tmp
poolside/laguna-xs-2.1:low__json_schema PASS 100% 74 in → 314 out (299 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 74 in → 2,215 out (2,200 reasoning)
git clean -f -x -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -x -e important.tmp"}
poolside/laguna-xs-2.1:none PASS 100% 74 in → 15 out (0 reasoning)
git clean -fdX && git clean -fd --exclude=important.tmp
poolside/laguna-xs-2.1:xhigh PASS 100% 74 in → 1,353 out (1,341 reasoning)
git clean -f -- junk1.tmp junk2.txt
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 74 in → 673 out (657 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f --exclude=important.tmp"}
poolside/laguna-xs.2:high PASS 100% 111 in → 597 out (584 reasoning)
git clean -fd --exclude='important.tmp'
poolside/laguna-xs.2:high__json_schema PASS 100% 111 in → 190 out (175 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
poolside/laguna-xs.2:low PASS 100% 111 in → 630 out (619 reasoning)
git clean -fd -e important.tmp
poolside/laguna-xs.2:low__json_schema PASS 100% 111 in → 697 out (679 reasoning)
git clean -fd ':(exclude)important.tmp'
JSON Schema Structured Output
(raw) {"command": "git clean -fd ':(exclude)important.tmp'"}
poolside/laguna-xs.2:medium PASS 100% 111 in → 150 out (139 reasoning)
git clean -f -e important.tmp
poolside/laguna-xs.2:medium__json_schema PASS 100% 111 in → 686 out (671 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
poolside/laguna-xs.2:xhigh PASS 100% 111 in → 344 out (333 reasoning)
git clean -f -e important.tmp
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 111 in → 270 out (250 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
qwen/qwen3.6-27b:high PASS 100% 70 in → 1,879 out (1,742 reasoning)
git clean -fd --exclude=important.tmp
qwen/qwen3.6-27b:high__json_schema PASS 100% 70 in → 2,242 out (2,141 reasoning)
git clean -f --exclude important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f --exclude important.tmp" }
qwen/qwen3.6-27b:low PASS 100% 70 in → 1,660 out (1,582 reasoning)
git clean -f -e important.tmp
qwen/qwen3.6-27b:low__json_schema PASS 100% 70 in → 1,287 out (1,220 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
qwen/qwen3.6-27b:medium PASS 100% 70 in → 1,454 out (1,356 reasoning)
git clean -f -e important.tmp
qwen/qwen3.6-27b:none PASS 100% 72 in → 9 out (0 reasoning)
git clean -xf -e important.tmp
qwen/qwen3.6-27b:none__json_schema PASS 100% 72 in → 25 out (0 reasoning)
git clean -fdX && git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fdX && git clean -fd -e important.tmp" }
qwen/qwen3.6-27b:xhigh PASS 100% 70 in → 1,454 out (1 reasoning)
git clean -f -e important.tmp
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 70 in → 2,019 out (1,858 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd -e important.tmp" }
qwen/qwen3.6-35b-a3b:high PASS 100% 70 in → 1,376 out (1,282 reasoning)
git clean -f --exclude=important.tmp
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 70 in → 2,189 out (2,059 reasoning)
git clean -fd --exclude=important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd --exclude=important.tmp" }
qwen/qwen3.6-35b-a3b:low PASS 100% 70 in → 1,826 out (1,813 reasoning)
git clean -f -e important.tmp
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 70 in → 1,961 out (1,813 reasoning)
git clean -f --exclude=important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f --exclude=important.tmp" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 70 in → 1,930 out (1,793 reasoning)
git clean -f -e important.tmp
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 70 in → 1,729 out (1,622 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
qwen/qwen3.6-35b-a3b:none PASS 100% 72 in → 10 out (0 reasoning)
git clean -fdx -e important.tmp
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 70 in → 741 out (709 reasoning)
git clean -f --exclude=important.tmp
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 70 in → 1,647 out (1,492 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
qwen/qwen3.6-flash:high PASS 100% 70 in → 1,427 out (1,413 reasoning)
git clean -f --exclude=important.tmp
qwen/qwen3.6-flash:low PASS 100% 70 in → 1,637 out (1,624 reasoning)
git clean -f -e important.tmp
qwen/qwen3.6-flash:medium PASS 100% 70 in → 1,377 out (1,364 reasoning)
git clean -f -e important.tmp
qwen/qwen3.6-flash:none PASS 100% 72 in → 9 out (0 reasoning)
git clean -fd --exclude=important.tmp
qwen/qwen3.6-flash:xhigh PASS 100% 70 in → 1,183 out (1,169 reasoning)
git clean -f --exclude=important.tmp
qwen/qwen3.7-flash:high PASS 100% 70 in → 1,582 out (1,568 reasoning)
git clean -f --exclude=important.tmp
qwen/qwen3.7-flash:low PASS 100% 70 in → 1,299 out (1,285 reasoning)
git clean -f --exclude=important.tmp
qwen/qwen3.7-flash:max PASS 100% 70 in → 1,262 out (1,248 reasoning)
git clean -f --exclude=important.tmp
qwen/qwen3.7-flash:medium PASS 100% 70 in → 816 out (803 reasoning)
git clean -f -e important.tmp
qwen/qwen3.7-flash:none PASS 100% 72 in → 9 out (0 reasoning)
git clean -fd --exclude=important.tmp
qwen/qwen3.7-flash:xhigh PASS 100% 70 in → 593 out (579 reasoning)
git clean -f --exclude=important.tmp
qwen/qwen3.7-max:high PASS 100% 70 in → 1,056 out (1,043 reasoning)
git clean -f -e important.tmp
qwen/qwen3.7-max:low PASS 100% 70 in → 385 out (372 reasoning)
git clean -f -e important.tmp
qwen/qwen3.7-max:medium PASS 100% 70 in → 879 out (866 reasoning)
git clean -f -e important.tmp
qwen/qwen3.7-max:none PASS 100% 72 in → 9 out (0 reasoning)
git clean -f --exclude=important.tmp
qwen/qwen3.7-max:xhigh PASS 100% 70 in → 807 out (794 reasoning)
git clean -f -e important.tmp
qwen/qwen3.7-plus:high PASS 100% 70 in → 708 out (695 reasoning)
git clean -f -e important.tmp
qwen/qwen3.7-plus:low PASS 100% 70 in → 708 out (695 reasoning)
git clean -f -e important.tmp
qwen/qwen3.7-plus:medium PASS 100% 70 in → 741 out (728 reasoning)
git clean -f -e important.tmp
qwen/qwen3.7-plus:none PASS 100% 72 in → 8 out (0 reasoning)
git clean -fd -e important.tmp
qwen/qwen3.7-plus:xhigh PASS 100% 70 in → 431 out (418 reasoning)
git clean -f -e important.tmp
tencent/hy3:high PASS 100% 75 in → 516 out (505 reasoning)
git clean -f -e important.tmp
tencent/hy3:high__json_schema PASS 100% 75 in → 927 out (836 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
tencent/hy3:low PASS 100% 75 in → 542 out (531 reasoning)
git clean -f -e important.tmp
tencent/hy3:low__json_schema PASS 100% 75 in → 557 out (507 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
tencent/hy3:medium PASS 100% 75 in → 433 out (422 reasoning)
git clean -f -e important.tmp
tencent/hy3:medium__json_schema PASS 100% 75 in → 359 out (321 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
tencent/hy3:none PASS 100% 78 in → 10 out (0 reasoning)
git clean -f -e important.tmp
tencent/hy3:none__json_schema PASS 100% 78 in → 20 out (0 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
tencent/hy3:xhigh PASS 100% 75 in → 554 out (543 reasoning)
git clean -f -e important.tmp
tencent/hy3:xhigh__json_schema PASS 100% 75 in → 1,293 out (1,200 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
thinkingmachines/inkling-small:high PASS 100% 70 in → 175 out (161 reasoning)
git clean -fd -e important.tmp
thinkingmachines/inkling-small:high__json_schema PASS 100% 70 in → 183 out (163 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
thinkingmachines/inkling-small:low PASS 100% 70 in → 59 out (50 reasoning)
git clean -fd -e important.tmp
thinkingmachines/inkling-small:low__json_schema PASS 100% 70 in → 55 out (35 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd -e important.tmp" }
thinkingmachines/inkling-small:max PASS 100% 70 in → 260 out (270 reasoning)
git clean -fd -e important.tmp
thinkingmachines/inkling-small:max__json_schema PASS 100% 70 in → 270 out (252 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fd -e important.tmp"}
thinkingmachines/inkling-small:medium PASS 100% 70 in → 114 out (100 reasoning)
git clean -fd -e important.tmp
thinkingmachines/inkling-small:medium__json_schema PASS 100% 70 in → 170 out (150 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd -e important.tmp" }
thinkingmachines/inkling-small:none PASS 100% 68 in → 12 out (0 reasoning)
git clean -fd --exclude=important.tmp
thinkingmachines/inkling-small:xhigh PASS 100% 70 in → 247 out (233 reasoning)
git clean -fd -e important.tmp
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 70 in → 276 out (259 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -fd -e important.tmp"}
thinkingmachines/inkling:high PASS 100% 70 in → 415 out (401 reasoning)
git clean -fd -e important.tmp
thinkingmachines/inkling:high__json_schema PASS 100% 70 in → 182 out (161 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd -e important.tmp" }
thinkingmachines/inkling:low PASS 100% 70 in → 78 out (61 reasoning)
git clean -fd -e important.tmp
thinkingmachines/inkling:max PASS 100% 70 in → 201 out (187 reasoning)
git clean -fd -e important.tmp
thinkingmachines/inkling:medium PASS 100% 70 in → 255 out (256 reasoning)
git clean -f -e important.tmp
thinkingmachines/inkling:medium__json_schema PASS 100% 70 in → 118 out (98 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd -e important.tmp" }
thinkingmachines/inkling:none PASS 100% 68 in → 12 out (0 reasoning)
git clean -f -e important.tmp
thinkingmachines/inkling:xhigh PASS 100% 70 in → 249 out (235 reasoning)
git clean -fd -e important.tmp
x-ai/grok-4.3:high PASS 100% 240 in → 473 out (463 reasoning)
`git clean -f -e important.tmp`
x-ai/grok-4.3:high__json_schema PASS 100% 296 in → 691 out (679 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
x-ai/grok-4.3:low PASS 100% 246 in → 333 out (325 reasoning)
git clean -f -e important.tmp
x-ai/grok-4.3:low__json_schema PASS 100% 302 in → 356 out (340 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
x-ai/grok-4.3:max PASS 100% 240 in → 774 out (766 reasoning)
git clean -f -e important.tmp
x-ai/grok-4.3:max__json_schema PASS 100% 296 in → 454 out (438 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
x-ai/grok-4.3:medium PASS 100% 246 in → 870 out (862 reasoning)
git clean -f -e important.tmp
x-ai/grok-4.3:medium__json_schema PASS 100% 302 in → 541 out (525 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
x-ai/grok-4.3:none PASS 100% 238 in → 9 out (0 reasoning)
git clean -fd --exclude=important.tmp
x-ai/grok-4.3:xhigh PASS 100% 240 in → 846 out (836 reasoning)
git clean -f -- ":!important.tmp"
x-ai/grok-4.3:xhigh__json_schema PASS 100% 296 in → 1,096 out (1,084 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
x-ai/grok-4.5:high PASS 100% 261 in → 336 out (328 reasoning)
git clean -f -e important.tmp
x-ai/grok-4.5:high__json_schema PASS 100% 327 in → 463 out (450 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
x-ai/grok-4.5:low PASS 100% 261 in → 184 out (171 reasoning)
```git git clean -f -e important.tmp ```
x-ai/grok-4.5:low__json_schema PASS 100% 327 in → 188 out (175 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
x-ai/grok-4.5:max PASS 100% 261 in → 434 out (422 reasoning)
``` git clean -f -e important.tmp ```
x-ai/grok-4.5:max__json_schema PASS 100% 327 in → 499 out (486 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
x-ai/grok-4.5:medium PASS 100% 261 in → 560 out (552 reasoning)
git clean -f -e important.tmp
x-ai/grok-4.5:medium__json_schema PASS 100% 327 in → 440 out (427 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
x-ai/grok-4.5:xhigh PASS 100% 261 in → 430 out (422 reasoning)
git clean -f -e important.tmp
x-ai/grok-4.5:xhigh__json_schema PASS 100% 327 in → 617 out (604 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -e important.tmp"}
z-ai/glm-4.7-flash:high PASS 100% 63 in → 379 out (349 reasoning)
git clean -f -e important.tmp
z-ai/glm-4.7-flash:high__json_schema PASS 100% 63 in → 489 out (442 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -f -e important.tmp"}
z-ai/glm-4.7-flash:low PASS 100% 63 in → 1,287 out (1,285 reasoning)
git clean -f -e important.tmp
z-ai/glm-4.7-flash:medium PASS 100% 63 in → 1,560 out (1,579 reasoning)
git clean -f -e important.tmp
z-ai/glm-4.7-flash:xhigh PASS 100% 63 in → 716 out (706 reasoning)
git clean -f -e important.tmp
z-ai/glm-5.2:high PASS 100% 70 in → 490 out (0 reasoning)
git clean -f -e important.tmp
z-ai/glm-5.2:low PASS 100% 70 in → 492 out (475 reasoning)
git clean -fd -e important.tmp
z-ai/glm-5.2:low__json_schema PASS 100% 245 in → 216 out (200 reasoning)
git clean -f -d -e important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -d -e important.tmp"}
z-ai/glm-5.2:medium PASS 100% 70 in → 352 out (342 reasoning)
git clean -f -e important.tmp
z-ai/glm-5.2:medium__json_schema PASS 100% 70 in → 403 out (361 reasoning)
git clean -f -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f -e important.tmp" }
z-ai/glm-5.2:none PASS 100% 64 in → 9 out (0 reasoning)
git clean -fd -e important.tmp
z-ai/glm-5.2:none__json_schema PASS 100% 66 in → 15 out (0 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd -e important.tmp" }
z-ai/glm-5.2:xhigh PASS 100% 70 in → 722 out (712 reasoning)
git clean -fd -e important.tmp
z-ai/glm-5.2:xhigh__json_schema PASS 100% 72 in → 494 out (477 reasoning)
git clean -fd -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fd -e important.tmp" }
arcee-ai/trinity-mini:high__json_schema FAIL 75% 65 in → 727 out (783 reasoning)
git clean -f&& git add important.tmp
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'important.tmp'}]
arcee-ai/trinity-mini:medium FAIL 75% 65 in → 438 out (457 reasoning)
git clean -f -e "junk*.tmp junk2.txt"
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'important.tmp'}]
minimax/minimax-m2.7:low__json_schema FAIL 75% 100 in → 29 out (29 reasoning)
git clean -f && git checkout -- important.tmp && git clean -f -e important.tmp -X
JSON Schema Structured Output
(raw) {"command": "git clean -f && git checkout -- important.tmp && git clean -f -e important.tmp -X"}
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'important.tmp'}]
openai/gpt-5.6-terra:medium__json_schema FAIL 75% 96 in → 33 out (0 reasoning)
find . -type f -not -path './.git/*' ! -name important.tmp -delete
JSON Schema Structured Output
(raw) {"command":"find . -type f -not -path './.git/*' ! -name important.tmp -delete"}
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'tracked.txt'}]
poolside/laguna-xs-2.1:medium FAIL 75% 74 in → 699 out (689 reasoning)
git clean -fd -e '!important.tmp'
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'important.tmp'}]
poolside/laguna-xs-2.1:none__json_schema FAIL 75% 74 in → 28 out (0 reasoning)
git clean -f -e !important.tmp -e *.tmp -e *.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f -e !important.tmp -e *.tmp -e *.txt" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk2.txt'}]
arcee-ai/trinity-mini:xhigh FAIL 50% 65 in → 389 out (435 reasoning)
gitclean -f -e important.tmp
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
google/gemma-4-26b-a4b-it:none__json_schema FAIL 50% 82 in → 19 out (0 reasoning)
git clean -f !important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f !important.tmp" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
ibm-granite/granite-4.1-8b FAIL 50% 80 in → 11 out
git rm --cached junk1.tmp junk2.txt
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
git reset HEAD important.tmp && git clean -dfx --except important.tmp
JSON Schema Structured Output
(raw) { "command": "git reset HEAD important.tmp && git clean -dfx --except important.tmp" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
liquid/lfm-2-24b-a2b FAIL 50% 76 in → 13 out
git rm --ignore-unmatch --force important.tmp
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
minimax/minimax-m2.5:medium__json_schema FAIL 50% 100 in → 242 out (244 reasoning)
git clean -f !important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -f !important.tmp" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
minimax/minimax-m3:high__json_schema FAIL 50% 225 in → 13 out (0 reasoning)
git clean -fn
JSON Schema Structured Output
(raw) { "command": "git clean -fn" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
minimax/minimax-m3:medium__json_schema FAIL 50% 225 in → 18 out (0 reasoning)
git clean -f -- !(important.tmp)
JSON Schema Structured Output
(raw) { "command": "git clean -f -- !(important.tmp)" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
minimax/minimax-m3:xhigh__json_schema FAIL 50% 225 in → 17 out (0 reasoning)
git clean -f -- !(important.tmp)
JSON Schema Structured Output
(raw) { "command": "git clean -f -- !(important.tmp)" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
nvidia/nemotron-3-nano-30b-a3b:none FAIL 50% 81 in → 35 out (0 reasoning)
git clean -fdX --force --important.tmp ; git clean -nfd . > .gitignore; git clean -fdX --force --important.tmp
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
openai/gpt-oss-120b:high__json_schema FAIL 50% 122 in → 541 out (539 reasoning)
finalgit clean -fd -e important.tmp
JSON Schema Structured Output
(raw) {"command":"finalgit clean -fd -e important.tmp"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
openai/gpt-oss-20b:high__json_schema FAIL 50% 122 in → 17 out (0 reasoning)
git clean -f -X -X !important.tmp
JSON Schema Structured Output
(raw) {"command":"git clean -f -X -X !important.tmp"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
poolside/laguna-m.1:none FAIL 50% 74 in → 9 out (0 reasoning)
git clean -dX !important.tmp
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
poolside/laguna-xs.2:none FAIL 50% 111 in → 11 out (0 reasoning)
git clean -O -- 'important.tmp'
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
poolside/laguna-xs.2:none__json_schema FAIL 50% 111 in → 16 out (0 reasoning)
git clean -fdv --exclude=important.tmp
JSON Schema Structured Output
(raw) {"command": "git clean -fdv --exclude=important.tmp"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
x-ai/grok-4.3:none__json_schema FAIL 50% 294 in → 15 out (0 reasoning)
git clean -f -x "!(important.tmp)"
JSON Schema Structured Output
(raw) {"command":"git clean -f -x \"!(important.tmp)\""}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
z-ai/glm-4.7-flash:none FAIL 50% 63 in → 9 out (0 reasoning)
git clean -f -X important.tmp
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
z-ai/glm-4.7-flash:none__json_schema FAIL 50% 63 in → 12 out (0 reasoning)
git clean -X -f
JSON Schema Structured Output
(raw) {"command": "git clean -X -f"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
arcee-ai/trinity-mini:medium__json_schema FAIL 25% 65 in → 622 out (629 reasoning)
git clean -f -e junk1.tmp -e junk2.txt
JSON Schema Structured Output
(raw) { "command": "git clean -f -e junk1.tmp -e junk2.txt" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'important.tmp'}, {'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
openai/gpt-5.4-nano:low FAIL 25% 61 in → 14 out (0 reasoning)
git clean -fdX --exclude=important.tmp
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'important.tmp'}, {'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
poolside/laguna-m.1:none__json_schema FAIL 25% 74 in → 15 out (0 reasoning)
git clean -fdx 'important.tmp'
JSON Schema Structured Output
(raw) {"command": "git clean -fdx 'important.tmp'"}
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'important.tmp'}, {'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
qwen/qwen3.6-27b:medium__json_schema FAIL 25% 72 in → 20 out (0 reasoning)
git clean -fdX -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fdX -e important.tmp" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'important.tmp'}, {'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 25% 72 in → 17 out (0 reasoning)
git clean -fdX -e important.tmp
JSON Schema Structured Output
(raw) { "command": "git clean -fdX -e important.tmp" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'important.tmp'}, {'type': 'file_not_exists', 'path': 'junk1.tmp'}, {'type': 'file_not_exists', 'path': 'junk2.txt'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -f -e important.tmp
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git rm --ignore-unmatch --force important.tmp
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: OpenAI-compatible response contained provider error: Provider returned error
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-768d91df-ef4f-943b-b890-5da269e6b361","request_id":"768d91df-ef4f-943b-b890-5da269e6b361"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-381ff324-9134-900a-b2e1-e37d00d8e735","request_id":"381ff324-9134-900a-b2e1-e37d00d8e735"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d9b3d969-eed0-900c-ac71-8568635bd3ca","request_id":"d9b3d969-eed0-900c-ac71-8568635bd3ca"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-69a888ea-81cd-90a6-b2d1-5ac336658bfe","request_id":"69a888ea-81cd-90a6-b2d1-5ac336658bfe"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4ec12378-8c9b-96d9-b08d-860dd7f36cbc","request_id":"4ec12378-8c9b-96d9-b08d-860dd7f36cbc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5f3adb71-d38c-9a52-94b7-2060be39d590"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-29f74a4f-efe9-9883-a726-2aa8c2948ed0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9b889ab3-59be-9557-8af3-85fd4c22b164"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9a584efe-3c6a-960b-8299-4faf2227d6ff"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4a5bf91c-e93a-9b0a-b355-19512e759ff6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f76dd7da-81fa-9dfd-9815-bfa320312737"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-041e08ac-a43b-91e9-9c35-87c08fe40b9a","request_id":"041e08ac-a43b-91e9-9c35-87c08fe40b9a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4c764cf6-4000-94fe-8685-582e090fcb2f","request_id":"4c764cf6-4000-94fe-8685-582e090fcb2f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d421ab52-37cd-9cf8-89e9-9cf2e02af1af","request_id":"d421ab52-37cd-9cf8-89e9-9cf2e02af1af"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-708a0771-5ed5-9b47-8411-7747c2d2f61d","request_id":"708a0771-5ed5-9b47-8411-7747c2d2f61d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e1743cee-f095-9554-8038-8829ccf8d13c","request_id":"e1743cee-f095-9554-8038-8829ccf8d13c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bc5db4e5-08fb-9294-9767-26f56b4c6de2","request_id":"bc5db4e5-08fb-9294-9767-26f56b4c6de2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c7dc73b3-ef52-9ddb-9b64-ba40ca09362c","request_id":"c7dc73b3-ef52-9ddb-9b64-ba40ca09362c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-609c3ca6-0ab8-9e58-b2ff-82b130e601ce","request_id":"609c3ca6-0ab8-9e58-b2ff-82b130e601ce"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-47cc1881-5af4-9731-a74f-cdc13339eb57","request_id":"47cc1881-5af4-9731-a74f-cdc13339eb57"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c6adab0f-b3da-92ca-8676-cd7ff69ca17b","request_id":"c6adab0f-b3da-92ca-8676-cd7ff69ca17b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: git clean -fd -e important.tmp{ "command": "git clean -fd -e important.tmp" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd -e important.tmp
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd -e important.tmp
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd --exclude=important.tmp{ "command": "git clean -fd --exclude=important.tmp" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd -e important.tmp
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -f -e important.tmp
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)