Remove ignored build artifacts with nested directories
Tests ability to remove ignored build artifacts in nested directories. Evaluates complex cleanup with directory hierarchies.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf '*.o *.pyc __pycache__/ build/ ' > .gitignore
  5. 05 echo 'src' > main.py
  6. 06 git add .gitignore main.py
  7. 07 git commit -m 'Initial commit'
  8. 08 echo 'obj' > module.o
  9. 09 mkdir -p __pycache__
  10. 10 echo 'cache' > __pycache__/main.pyc
  11. 11 mkdir -p build/dist
  12. 12 echo 'binary' > build/dist/app
  13. 13 echo 'untracked' > notes.md
Prompt
Remove all ignored files and directories while preserving untracked files. Output ONLY the git command, nothing else.
Expected
git clean -f -X -d

Scoped model quality, cost, API time, and token usage for git_clean/f011.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 94 in → 49 out (42 reasoning)
``` git clean -fdX ```
anthropic/claude-fable-5:high__json_schema PASS 100% 304 in → 50 out (36 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
anthropic/claude-fable-5:low PASS 100% 94 in → 18 out (30 reasoning)
git clean -fdX
anthropic/claude-fable-5:low__json_schema PASS 100% 304 in → 16 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
anthropic/claude-fable-5:max PASS 100% 94 in → 236 out (74 reasoning)
`git clean -fdX`
anthropic/claude-fable-5:max__json_schema PASS 100% 304 in → 213 out (102 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
anthropic/claude-fable-5:medium PASS 100% 94 in → 22 out (30 reasoning)
``` git clean -fdX ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 304 in → 16 out (0 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) {"command": "git clean -fXd"}
anthropic/claude-fable-5:xhigh PASS 100% 94 in → 59 out (41 reasoning)
``` git clean -fdX ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 304 in → 51 out (36 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
anthropic/claude-haiku-4.5:high PASS 100% 106 in → 752 out (526 reasoning)
``` git clean -fdX ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 269 in → 902 out (596 reasoning)
git clean -f -d -X
JSON Schema Structured Output
(raw) {"command": "git clean -f -d -X"}
anthropic/claude-haiku-4.5:low PASS 100% 106 in → 229 out (212 reasoning)
git clean -fdX
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 269 in → 383 out (367 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 269 in → 252 out (230 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
anthropic/claude-haiku-4.5:none PASS 100% 77 in → 19 out (0 reasoning)
``` git clean -fd -x --exclude=notes.md ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 239 in → 14 out (0 reasoning)
git clean -fd -X
JSON Schema Structured Output
(raw) {"command": "git clean -fd -X"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 106 in → 289 out (248 reasoning)
``` git clean -fdX ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 269 in → 571 out (517 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
anthropic/claude-opus-4.6:high PASS 100% 77 in → 111 out (86 reasoning)
``` git clean -fdX ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 240 in → 92 out (65 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
anthropic/claude-opus-4.6:low PASS 100% 77 in → 13 out (0 reasoning)
``` git clean -ffdX ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 240 in → 13 out (0 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) {"command":"git clean -fXd"}
anthropic/claude-opus-4.6:max PASS 100% 77 in → 113 out (92 reasoning)
``` git clean -fdX ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 240 in → 117 out (96 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) {"command":"git clean -fXd"}
anthropic/claude-opus-4.6:medium PASS 100% 77 in → 13 out (0 reasoning)
``` git clean -fXd ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 240 in → 13 out (0 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) {"command":"git clean -fXd"}
anthropic/claude-opus-4.6:none PASS 100% 77 in → 13 out (0 reasoning)
``` git clean -ffdX ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 240 in → 13 out (0 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) {"command":"git clean -fXd"}
anthropic/claude-opus-4.6:xhigh PASS 100% 77 in → 112 out (86 reasoning)
``` git clean -fXd ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 240 in → 91 out (64 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
anthropic/claude-opus-4.7:high PASS 100% 99 in → 14 out (0 reasoning)
`git clean -Xfd`
anthropic/claude-opus-4.7:high__json_schema PASS 100% 309 in → 18 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
anthropic/claude-opus-4.7:low PASS 100% 99 in → 12 out (0 reasoning)
git clean -Xfd
anthropic/claude-opus-4.7:low__json_schema PASS 100% 309 in → 19 out (0 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) {"command":"git clean -Xfd"}
anthropic/claude-opus-4.7:max PASS 100% 99 in → 16 out (0 reasoning)
``` git clean -fdX ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 309 in → 18 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
anthropic/claude-opus-4.7:medium PASS 100% 99 in → 12 out (0 reasoning)
git clean -Xfd
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 309 in → 18 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
anthropic/claude-opus-4.7:none PASS 100% 99 in → 13 out (0 reasoning)
`git clean -fdX`
anthropic/claude-opus-4.7:none__json_schema PASS 100% 309 in → 19 out (0 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) {"command":"git clean -Xfd"}
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 309 in → 18 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
anthropic/claude-opus-4.8:high PASS 100% 94 in → 10 out (0 reasoning)
git clean -Xdf
anthropic/claude-opus-4.8:high__json_schema PASS 100% 304 in → 17 out (0 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
anthropic/claude-opus-4.8:low PASS 100% 94 in → 11 out (0 reasoning)
`git clean -fdX`
anthropic/claude-opus-4.8:low__json_schema PASS 100% 304 in → 17 out (0 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
anthropic/claude-opus-4.8:max PASS 100% 94 in → 387 out (73 reasoning)
`git clean -fdX`
anthropic/claude-opus-4.8:max__json_schema PASS 100% 304 in → 256 out (28 reasoning)
git clean -dfX
JSON Schema Structured Output
(raw) {"command":"git clean -dfX"}
anthropic/claude-opus-4.8:medium PASS 100% 94 in → 10 out (0 reasoning)
git clean -Xdf
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 304 in → 17 out (0 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) {"command": "git clean -Xfd"}
anthropic/claude-opus-4.8:none PASS 100% 94 in → 13 out (0 reasoning)
``` git clean -fdX ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 304 in → 16 out (0 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
anthropic/claude-opus-4.8:xhigh PASS 100% 94 in → 9 out (0 reasoning)
git clean -fdX
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 304 in → 17 out (0 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
anthropic/claude-sonnet-4.6:high PASS 100% 77 in → 108 out (87 reasoning)
``` git clean -fdX ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 240 in → 113 out (89 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
anthropic/claude-sonnet-4.6:max PASS 100% 77 in → 164 out (125 reasoning)
``` git clean -Xdf ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 240 in → 154 out (119 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 77 in → 107 out (81 reasoning)
``` git clean -Xdf ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 240 in → 102 out (81 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
anthropic/claude-sonnet-5:high PASS 100% 94 in → 13 out (0 reasoning)
``` git clean -fdX ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 304 in → 16 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 304 in → 16 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
anthropic/claude-sonnet-5:none PASS 100% 94 in → 13 out (0 reasoning)
``` git clean -fdX ```
anthropic/claude-sonnet-5:xhigh PASS 100% 94 in → 74 out (37 reasoning)
git clean -X -d -f
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 304 in → 16 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
arcee-ai/trinity-large-thinking:high PASS 100% 62 in → 1,158 out (1,151 reasoning)
git clean -fdX
arcee-ai/trinity-large-thinking:xhigh PASS 100% 62 in → 2,011 out (2,026 reasoning)
git clean -Xfd
deepseek/deepseek-v4-flash-0731:high PASS 100% 139 in → 420 out (414 reasoning)
git clean -fdX
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 145 in → 170 out (160 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 152 in → 686 out (683 reasoning)
git clean -fdX
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 152 in → 520 out (554 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 60 in → 447 out (440 reasoning)
git clean -fdX
deepseek/deepseek-v4-flash-0731:none PASS 100% 60 in → 5 out (0 reasoning)
git clean -fdX
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 60 in → 13 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 145 in → 83 out (70 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
deepseek/deepseek-v4-flash:high PASS 100% 60 in → 294 out (284 reasoning)
git clean -f -d -X
deepseek/deepseek-v4-flash:low PASS 100% 60 in → 121 out (111 reasoning)
git clean -f -d -X
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 60 in → 340 out (362 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) {"command": "git clean -fXd"}
deepseek/deepseek-v4-flash:medium PASS 100% 60 in → 375 out (350 reasoning)
git clean -f -d -X
deepseek/deepseek-v4-flash:none PASS 100% 60 in → 6 out (0 reasoning)
git clean -fdX
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 60 in → 14 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 139 in → 573 out (566 reasoning)
git clean -fdX
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 139 in → 457 out (430 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
deepseek/deepseek-v4-pro:high PASS 100% 60 in → 303 out (296 reasoning)
git clean -Xdf
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 62 in → 413 out (401 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) {"command": "git clean -fXd"}
deepseek/deepseek-v4-pro:low PASS 100% 60 in → 368 out (370 reasoning)
git clean -fdX
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 247 in → 265 out (253 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
deepseek/deepseek-v4-pro:medium PASS 100% 60 in → 309 out (302 reasoning)
git clean -fdX
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 247 in → 332 out (320 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
deepseek/deepseek-v4-pro:none PASS 100% 60 in → 5 out (0 reasoning)
git clean -fdX
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 60 in → 14 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 139 in → 418 out (411 reasoning)
git clean -fdX
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 325 in → 415 out (403 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
google/gemini-3-flash-preview:high PASS 100% 64 in → 286 out (281 reasoning)
git clean -fdX
google/gemini-3-flash-preview:high__json_schema PASS 100% 121 in → 286 out (272 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
google/gemini-3-flash-preview:low PASS 100% 64 in → 234 out (229 reasoning)
git clean -fdX
google/gemini-3-flash-preview:low__json_schema PASS 100% 121 in → 328 out (319 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
google/gemini-3-flash-preview:medium PASS 100% 64 in → 289 out (284 reasoning)
git clean -fdX
google/gemini-3-flash-preview:medium__json_schema PASS 100% 65 in → 236 out (227 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
google/gemini-3-flash-preview:xhigh PASS 100% 64 in → 277 out (272 reasoning)
git clean -fdX
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 121 in → 141 out (132 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 64 in → 537 out (532 reasoning)
git clean -fdX
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 65 in → 409 out (399 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 65 in → 118 out (113 reasoning)
git clean -fdX
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 65 in → 120 out (105 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 64 in → 211 out (206 reasoning)
git clean -fXd
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 65 in → 348 out (338 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 64 in → 344 out (339 reasoning)
git clean -fdX
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 65 in → 275 out (261 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
google/gemini-3.1-pro-preview:high PASS 100% 64 in → 482 out (477 reasoning)
git clean -Xdf
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 121 in → 309 out (299 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) {"command": "git clean -Xfd"}
google/gemini-3.1-pro-preview:low PASS 100% 64 in → 192 out (187 reasoning)
git clean -Xfd
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 121 in → 125 out (110 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
google/gemini-3.1-pro-preview:medium PASS 100% 64 in → 462 out (457 reasoning)
git clean -Xdf
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 121 in → 276 out (266 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 64 in → 357 out (352 reasoning)
git clean -Xdf
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 65 in → 426 out (416 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command": "git clean -Xdf"}
google/gemini-3.5-flash-lite:high PASS 100% 64 in → 463 out (458 reasoning)
git clean -Xfd
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 121 in → 18 out (0 reasoning)
git clean -X -f -d
JSON Schema Structured Output
(raw) { "command": "git clean -X -f -d" }
google/gemini-3.5-flash-lite:max PASS 100% 65 in → 600 out (595 reasoning)
git clean -Xfd
google/gemini-3.5-flash-lite:medium PASS 100% 65 in → 233 out (228 reasoning)
git clean -Xfd
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 65 in → 293 out (283 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) {"command": "git clean -Xfd"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 64 in → 765 out (760 reasoning)
git clean -fdX
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 121 in → 357 out (347 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) {"command": "git clean -Xfd"}
google/gemini-3.5-flash:high PASS 100% 64 in → 689 out (684 reasoning)
git clean -fdX
google/gemini-3.5-flash:high__json_schema PASS 100% 121 in → 537 out (527 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
google/gemini-3.5-flash:low PASS 100% 64 in → 374 out (369 reasoning)
git clean -fdX
google/gemini-3.5-flash:low__json_schema PASS 100% 121 in → 269 out (254 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
google/gemini-3.5-flash:medium PASS 100% 64 in → 476 out (471 reasoning)
git clean -fdX
google/gemini-3.5-flash:medium__json_schema PASS 100% 121 in → 552 out (542 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
google/gemini-3.5-flash:xhigh PASS 100% 64 in → 522 out (517 reasoning)
git clean -fdX
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 121 in → 486 out (476 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) {"command": "git clean -fXd"}
google/gemini-3.6-flash:high PASS 100% 65 in → 538 out (533 reasoning)
git clean -fdX
google/gemini-3.6-flash:high__json_schema PASS 100% 65 in → 337 out (328 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
google/gemini-3.6-flash:low PASS 100% 65 in → 135 out (130 reasoning)
git clean -fdX
google/gemini-3.6-flash:max PASS 100% 64 in → 470 out (465 reasoning)
git clean -fdX
google/gemini-3.6-flash:max__json_schema PASS 100% 121 in → 445 out (430 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) { "command": "git clean -fXd" }
google/gemini-3.6-flash:medium PASS 100% 65 in → 441 out (436 reasoning)
git clean -fdX
google/gemini-3.6-flash:medium__json_schema PASS 100% 65 in → 312 out (303 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
google/gemini-3.6-flash:xhigh PASS 100% 65 in → 585 out (580 reasoning)
git clean -fdX
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 121 in → 492 out (483 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 80 in → 824 out (738 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
google/gemma-4-26b-a4b-it:low PASS 100% 80 in → 843 out (756 reasoning)
git clean -fdX
google/gemma-4-26b-a4b-it:medium PASS 100% 80 in → 1,322 out (1,212 reasoning)
git clean -fdX
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 80 in → 1,560 out (1,374 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
google/gemma-4-26b-a4b-it:none PASS 100% 77 in → 6 out (0 reasoning)
git clean -fdX
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 77 in → 16 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
google/gemma-4-31b-it:high PASS 100% 80 in → 272 out (236 reasoning)
git clean -fXd
google/gemma-4-31b-it:high__json_schema PASS 100% 80 in → 261 out (217 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
google/gemma-4-31b-it:low PASS 100% 80 in → 215 out (184 reasoning)
git clean -fdX
google/gemma-4-31b-it:low__json_schema PASS 100% 83 in → 274 out (263 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
google/gemma-4-31b-it:medium PASS 100% 80 in → 283 out (248 reasoning)
git clean -fdX
google/gemma-4-31b-it:medium__json_schema PASS 100% 79 in → 279 out (236 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
google/gemma-4-31b-it:none PASS 100% 77 in → 6 out (0 reasoning)
git clean -Xdf
google/gemma-4-31b-it:none__json_schema PASS 100% 77 in → 16 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
google/gemma-4-31b-it:xhigh PASS 100% 79 in → 349 out (315 reasoning)
git clean -fdX
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 80 in → 36 out (11 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
git clean -fdx -e notes.md
JSON Schema Structured Output
(raw) {"command":"git clean -fdx -e notes.md"}
minimax/minimax-m2.5:low PASS 100% 91 in → 308 out (321 reasoning)
git clean -fdX
minimax/minimax-m2.5:xhigh PASS 100% 91 in → 243 out (252 reasoning)
git clean -fdX
minimax/minimax-m2.7:high PASS 100% 93 in → 382 out (376 reasoning)
git clean -fdX
minimax/minimax-m2.7:low PASS 100% 93 in → 1,077 out (1,071 reasoning)
git clean -fXd
minimax/minimax-m2.7:low__json_schema PASS 100% 223 in → 428 out (413 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
minimax/minimax-m2.7:medium PASS 100% 93 in → 756 out (750 reasoning)
git clean -fdX
minimax/minimax-m2.7:medium__json_schema PASS 100% 223 in → 824 out (813 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
minimax/minimax-m2.7:xhigh PASS 100% 75 in → 195 out (207 reasoning)
``` git clean -fdX ```
minimax/minimax-m3:high PASS 100% 229 in → 178 out (180 reasoning)
git clean -fXd
minimax/minimax-m3:low PASS 100% 229 in → 99 out (111 reasoning)
`git clean -fXd`
minimax/minimax-m3:medium__json_schema PASS 100% 216 in → 13 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
mistralai/devstral-2512 PASS 100% 59 in → 11 out
```bash git clean -fdX ```
mistralai/devstral-2512__json_schema PASS 100% 59 in → 11 out
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
mistralai/mistral-medium-3-5:high PASS 100% 71 in → 2,404 out (2,635 reasoning)
git clean -fdX
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 59 in → 548 out (583 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command": "git clean -Xdf"}
mistralai/mistral-medium-3-5:low PASS 100% 71 in → 1,933 out (2,100 reasoning)
git clean -fdX
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 59 in → 3,137 out (3,341 reasoning)
git clean -fd -X
JSON Schema Structured Output
(raw) {"command": "git clean -fd -X"}
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 59 in → 1,804 out (1,956 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 59 in → 11 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 59 in → 1,662 out (1,740 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) {"command": "git clean -Xfd"}
moonshotai/kimi-k2.6:high PASS 100% 63 in → 3,383 out (3,377 reasoning)
git clean -Xfd
moonshotai/kimi-k2.6:low PASS 100% 63 in → 2,689 out (2,761 reasoning)
git clean -Xfd
moonshotai/kimi-k2.6:low__json_schema PASS 100% 63 in → 1,017 out (1,057 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) { "command": "git clean -Xfd" }
moonshotai/kimi-k2.6:medium PASS 100% 63 in → 2,969 out (2,964 reasoning)
git clean -Xfd
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 62 in → 1,754 out (1,755 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) { "command": "git clean -Xfd" }
moonshotai/kimi-k2.6:xhigh PASS 100% 63 in → 588 out (581 reasoning)
git clean -fdX
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 63 in → 968 out (1,065 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
moonshotai/kimi-k2.7-code:high PASS 100% 63 in → 362 out (352 reasoning)
git clean -X -d -f
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 63 in → 187 out (182 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) {"command": "git clean -fXd"}
moonshotai/kimi-k2.7-code:low PASS 100% 63 in → 184 out (174 reasoning)
git clean -X -d -f
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 62 in → 706 out (536 reasoning)
git clean -d -f -X
JSON Schema Structured Output
(raw) { "command": "git clean -d -f -X" }
moonshotai/kimi-k2.7-code:medium PASS 100% 63 in → 251 out (253 reasoning)
git clean -fXd
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 146 in → 208 out (197 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 62 in → 108 out (101 reasoning)
git clean -Xfd
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 63 in → 366 out (361 reasoning)
git clean -X -d -f
JSON Schema Structured Output
(raw) { "command": "git clean -X -d -f" }
moonshotai/kimi-k3:high PASS 100% 140 in → 195 out (175 reasoning)
git clean -fdX
moonshotai/kimi-k3:high__json_schema PASS 100% 140 in → 345 out (325 reasoning)
git clean -Xdf
moonshotai/kimi-k3:low PASS 100% 140 in → 180 out (160 reasoning)
git clean -fdX
moonshotai/kimi-k3:low__json_schema PASS 100% 140 in → 177 out (157 reasoning)
git clean -fdX
moonshotai/kimi-k3:max PASS 100% 140 in → 149 out (129 reasoning)
git clean -fdX
moonshotai/kimi-k3:max__json_schema PASS 100% 140 in → 204 out (184 reasoning)
git clean -fdX
moonshotai/kimi-k3:medium PASS 100% 140 in → 217 out (197 reasoning)
git clean -fdX
moonshotai/kimi-k3:medium__json_schema PASS 100% 140 in → 153 out (133 reasoning)
git clean -fXd
moonshotai/kimi-k3:xhigh PASS 100% 140 in → 205 out (182 reasoning)
``` git clean -fdX ```
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 140 in → 197 out (177 reasoning)
git clean -fdX
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 72 in → 444 out (492 reasoning)
git clean -Xf .
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 72 in → 504 out (542 reasoning)
git clean -fd -X
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 72 in → 6 out (0 reasoning)
git clean -Xfd
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 72 in → 175 out (161 reasoning)
git clean -Xfd
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 72 in → 403 out (374 reasoning)
git clean -X -f -d
JSON Schema Structured Output
(raw) { "command": "git clean -X -f -d" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 72 in → 1,670 out (1,613 reasoning)
git clean -f -X -d
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 72 in → 282 out (258 reasoning)
git clean -f -X -d
JSON Schema Structured Output
(raw) { "command": "git clean -f -X -d" }
openai/gpt-5.4-mini:high PASS 100% 58 in → 71 out (60 reasoning)
git clean -fdX
openai/gpt-5.4-mini:high__json_schema PASS 100% 93 in → 130 out (111 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4-mini:low PASS 100% 58 in → 42 out (31 reasoning)
git clean -fdX
openai/gpt-5.4-mini:low__json_schema PASS 100% 93 in → 82 out (63 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4-mini:medium PASS 100% 58 in → 78 out (67 reasoning)
git clean -fdX
openai/gpt-5.4-mini:medium__json_schema PASS 100% 93 in → 73 out (54 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4-mini:none PASS 100% 58 in → 9 out (0 reasoning)
git clean -fdX
openai/gpt-5.4-mini:none__json_schema PASS 100% 93 in → 23 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4-mini:xhigh PASS 100% 58 in → 163 out (152 reasoning)
git clean -fdX
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 93 in → 222 out (203 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4-nano:high PASS 100% 58 in → 112 out (101 reasoning)
git clean -fdX
openai/gpt-5.4-nano:high__json_schema PASS 100% 93 in → 98 out (79 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4-nano:low PASS 100% 58 in → 51 out (0 reasoning)
git clean -ndX -e '__pycache__' -e 'build' -e 'module.o' && git clean -fdX -e '__pycache__' -e 'build' -e 'module.o'
openai/gpt-5.4-nano:none__json_schema PASS 100% 93 in → 17 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4-nano:xhigh PASS 100% 58 in → 341 out (330 reasoning)
git clean -fdX
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 93 in → 603 out (584 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4:high PASS 100% 58 in → 71 out (60 reasoning)
git clean -fdX
openai/gpt-5.4:high__json_schema PASS 100% 93 in → 100 out (81 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4:low PASS 100% 58 in → 59 out (48 reasoning)
git clean -fdX
openai/gpt-5.4:low__json_schema PASS 100% 93 in → 48 out (29 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4:medium PASS 100% 58 in → 84 out (73 reasoning)
git clean -fdX
openai/gpt-5.4:medium__json_schema PASS 100% 93 in → 75 out (56 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4:none PASS 100% 58 in → 9 out (0 reasoning)
git clean -fdX
openai/gpt-5.4:none__json_schema PASS 100% 93 in → 17 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.4:xhigh PASS 100% 58 in → 102 out (91 reasoning)
git clean -fdX
openai/gpt-5.4:xhigh__json_schema PASS 100% 93 in → 99 out (80 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.5:high PASS 100% 58 in → 49 out (38 reasoning)
git clean -fdX
openai/gpt-5.5:high__json_schema PASS 100% 93 in → 155 out (136 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.5:low PASS 100% 58 in → 56 out (45 reasoning)
git clean -fdX
openai/gpt-5.5:low__json_schema PASS 100% 93 in → 53 out (34 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.5:medium PASS 100% 58 in → 47 out (36 reasoning)
git clean -fdX
openai/gpt-5.5:medium__json_schema PASS 100% 93 in → 115 out (96 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.5:none PASS 100% 58 in → 9 out (0 reasoning)
git clean -fdX
openai/gpt-5.5:none__json_schema PASS 100% 93 in → 17 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.5:xhigh PASS 100% 58 in → 216 out (205 reasoning)
git clean -fdX
openai/gpt-5.5:xhigh__json_schema PASS 100% 93 in → 273 out (254 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-luna:high PASS 100% 58 in → 60 out (49 reasoning)
git clean -fdX
openai/gpt-5.6-luna:high__json_schema PASS 100% 93 in → 66 out (47 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-luna:low PASS 100% 58 in → 40 out (29 reasoning)
git clean -fdX
openai/gpt-5.6-luna:low__json_schema PASS 100% 93 in → 52 out (33 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-luna:max PASS 100% 58 in → 109 out (98 reasoning)
git clean -fdX
openai/gpt-5.6-luna:max__json_schema PASS 100% 93 in → 114 out (95 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-luna:medium PASS 100% 58 in → 62 out (51 reasoning)
git clean -fdX
openai/gpt-5.6-luna:medium__json_schema PASS 100% 93 in → 53 out (34 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-luna:none PASS 100% 58 in → 9 out (0 reasoning)
git clean -fdX
openai/gpt-5.6-luna:none__json_schema PASS 100% 93 in → 17 out (0 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
openai/gpt-5.6-luna:xhigh PASS 100% 58 in → 69 out (58 reasoning)
git clean -fdX
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 93 in → 223 out (204 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-sol:high PASS 100% 58 in → 35 out (24 reasoning)
git clean -fdX
openai/gpt-5.6-sol:high__json_schema PASS 100% 93 in → 106 out (87 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-sol:low PASS 100% 58 in → 9 out (0 reasoning)
git clean -fdX
openai/gpt-5.6-sol:low__json_schema PASS 100% 93 in → 45 out (26 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-sol:max PASS 100% 58 in → 81 out (70 reasoning)
git clean -fdX
openai/gpt-5.6-sol:max__json_schema PASS 100% 93 in → 206 out (187 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-sol:medium PASS 100% 58 in → 9 out (0 reasoning)
git clean -fdX
openai/gpt-5.6-sol:medium__json_schema PASS 100% 93 in → 41 out (22 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-sol:none PASS 100% 58 in → 9 out (0 reasoning)
git clean -fdX
openai/gpt-5.6-sol:none__json_schema PASS 100% 93 in → 17 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-sol:xhigh PASS 100% 58 in → 9 out (0 reasoning)
git clean -fdX
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 93 in → 51 out (32 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-terra:high PASS 100% 58 in → 35 out (24 reasoning)
git clean -fdX
openai/gpt-5.6-terra:high__json_schema PASS 100% 93 in → 100 out (81 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-terra:low PASS 100% 58 in → 50 out (39 reasoning)
git clean -fdX
openai/gpt-5.6-terra:low__json_schema PASS 100% 93 in → 17 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-terra:max PASS 100% 58 in → 70 out (59 reasoning)
git clean -fdX
openai/gpt-5.6-terra:max__json_schema PASS 100% 93 in → 97 out (78 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-terra:medium PASS 100% 58 in → 32 out (21 reasoning)
git clean -fdX
openai/gpt-5.6-terra:medium__json_schema PASS 100% 93 in → 17 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-terra:none PASS 100% 58 in → 9 out (0 reasoning)
git clean -fdX
openai/gpt-5.6-terra:none__json_schema PASS 100% 93 in → 17 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-5.6-terra:xhigh PASS 100% 58 in → 37 out (26 reasoning)
git clean -fdX
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 93 in → 63 out (44 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-oss-120b:high PASS 100% 119 in → 329 out (365 reasoning)
git clean -fdX
openai/gpt-oss-120b:high__json_schema PASS 100% 106 in → 304 out (310 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-oss-120b:low PASS 100% 130 in → 41 out (0 reasoning)
git clean -fdX
openai/gpt-oss-120b:low__json_schema PASS 100% 121 in → 50 out (38 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-oss-120b:medium PASS 100% 119 in → 124 out (119 reasoning)
git clean -fdX
openai/gpt-oss-120b:medium__json_schema PASS 100% 119 in → 9 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
openai/gpt-oss-120b:xhigh PASS 100% 106 in → 435 out (484 reasoning)
git clean -fdX
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 121 in → 430 out (441 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
openai/gpt-oss-20b:high PASS 100% 119 in → 351 out (326 reasoning)
git clean -dfX
openai/gpt-oss-20b:high__json_schema PASS 100% 121 in → 687 out (730 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
openai/gpt-oss-20b:low PASS 100% 106 in → 94 out (78 reasoning)
git clean -fdX
openai/gpt-oss-20b:low__json_schema PASS 100% 169 in → 337 out (295 reasoning)
git clean -d -f -x -e notes.md
JSON Schema Structured Output
(raw) {"command":"git clean -d -f -x -e notes.md"}
openai/gpt-oss-20b:medium PASS 100% 117 in → 245 out (228 reasoning)
git clean -fdX
openai/gpt-oss-20b:medium__json_schema PASS 100% 119 in → 17 out (0 reasoning)
git clean -fdX --exclude=!*.md
JSON Schema Structured Output
(raw) { "command": "git clean -fdX --exclude=!*.md" }
openai/gpt-oss-20b:xhigh PASS 100% 119 in → 381 out (366 reasoning)
git clean -fdX
poolside/laguna-m.1:high PASS 100% 70 in → 553 out (545 reasoning)
git clean -fdX
poolside/laguna-m.1:high__json_schema PASS 100% 70 in → 289 out (272 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) { "command": "git clean -Xfd" }
poolside/laguna-m.1:low PASS 100% 70 in → 361 out (350 reasoning)
git clean -X -d -f
poolside/laguna-m.1:low__json_schema PASS 100% 70 in → 340 out (323 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) { "command": "git clean -Xdf" }
poolside/laguna-m.1:medium__json_schema PASS 100% 70 in → 640 out (627 reasoning)
git clean -dX -f
JSON Schema Structured Output
(raw) {"command": "git clean -dX -f"}
poolside/laguna-m.1:none PASS 100% 70 in → 7 out (0 reasoning)
git clean -fdX
poolside/laguna-m.1:none__json_schema PASS 100% 70 in → 11 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
poolside/laguna-m.1:xhigh PASS 100% 70 in → 269 out (261 reasoning)
git clean -Xdf
poolside/laguna-m.1:xhigh__json_schema PASS 100% 70 in → 309 out (297 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) {"command": "git clean -Xfd"}
poolside/laguna-xs-2.1:high PASS 100% 70 in → 290 out (284 reasoning)
git clean -dXf
poolside/laguna-xs-2.1:medium PASS 100% 70 in → 262 out (253 reasoning)
git clean -f -X -d
poolside/laguna-xs-2.1:none PASS 100% 70 in → 5 out (0 reasoning)
git clean -fdX
poolside/laguna-xs-2.1:xhigh PASS 100% 70 in → 264 out (258 reasoning)
git clean -fdX
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 70 in → 355 out (338 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
qwen/qwen3.6-27b:high__json_schema PASS 100% 66 in → 1,175 out (1,100 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
qwen/qwen3.6-27b:low PASS 100% 66 in → 1,376 out (1,297 reasoning)
git clean -Xfd
qwen/qwen3.6-27b:low__json_schema PASS 100% 66 in → 1,309 out (1 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) { "command": "git clean -Xdf" }
qwen/qwen3.6-27b:medium PASS 100% 66 in → 2,560 out (2,300 reasoning)
git clean -f -X -d
qwen/qwen3.6-27b:medium__json_schema PASS 100% 66 in → 1,467 out (1,359 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
qwen/qwen3.6-27b:none PASS 100% 68 in → 6 out (0 reasoning)
git clean -fdX
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 66 in → 1,029 out (909 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) { "command": "git clean -Xfd" }
qwen/qwen3.6-35b-a3b:high PASS 100% 66 in → 2,106 out (2,012 reasoning)
git clean -fdX
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 66 in → 1,094 out (1,011 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) { "command": "git clean -Xdf" }
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 66 in → 1,385 out (1,322 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) { "command": "git clean -Xdf" }
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 66 in → 1,455 out (1,312 reasoning)
git clean -dXf
JSON Schema Structured Output
(raw) { "command": "git clean -dXf" }
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 68 in → 115 out (0 reasoning)
find . -type f -name '*.pyc' -delete -o -type d -name '__pycache__' -exec rm -rf {} + -o -type f -name 'main.pyc' -delete -o -type d -name 'build' -exec rm -rf {} + -o -type f -name 'module.o' -delete -o -type f -name 'app' -delete 2>/dev/null; echo "Cleanup completed. Untracked files preserved."
JSON Schema Structured Output
(raw) { "command": "find . -type f -name '*.pyc' -delete -o -type d -name '__pycache__' -exec rm -rf {} + -o -type f -name 'main.pyc' -delete -o -type d -name 'build' -exec rm -rf {} + -o -type f -name 'module.o' -delete -o -type f -name 'app' -delete 2>/dev/null; echo \"Cleanup completed. Untracked files preserved.\"" }
qwen/qwen3.6-flash:high PASS 100% 66 in → 2,420 out (2,410 reasoning)
git clean -fdX
qwen/qwen3.6-flash:low PASS 100% 66 in → 1,845 out (1,835 reasoning)
git clean -fdX
qwen/qwen3.6-flash:medium PASS 100% 66 in → 1,475 out (1,465 reasoning)
git clean -Xdf
qwen/qwen3.6-flash:xhigh PASS 100% 66 in → 1,067 out (1,057 reasoning)
git clean -fdX
qwen/qwen3.7-flash:high PASS 100% 66 in → 1,520 out (1,510 reasoning)
git clean -fdX
qwen/qwen3.7-flash:low PASS 100% 66 in → 1,468 out (1,454 reasoning)
git clean -fdx -e notes.md
qwen/qwen3.7-flash:medium PASS 100% 66 in → 1,567 out (1,557 reasoning)
git clean -Xfd
qwen/qwen3.7-flash:none PASS 100% 68 in → 5 out (0 reasoning)
git clean -fdX
qwen/qwen3.7-max:high PASS 100% 66 in → 246 out (236 reasoning)
git clean -fdX
qwen/qwen3.7-max:low PASS 100% 66 in → 967 out (957 reasoning)
git clean -fdX
qwen/qwen3.7-max:medium PASS 100% 66 in → 980 out (970 reasoning)
git clean -fdX
qwen/qwen3.7-max:none PASS 100% 68 in → 5 out (0 reasoning)
git clean -fXd
qwen/qwen3.7-max:xhigh PASS 100% 66 in → 925 out (915 reasoning)
git clean -fdX
qwen/qwen3.7-plus:high PASS 100% 66 in → 1,474 out (1,464 reasoning)
git clean -fdX
qwen/qwen3.7-plus:low PASS 100% 66 in → 648 out (638 reasoning)
git clean -fdX
qwen/qwen3.7-plus:medium PASS 100% 66 in → 1,181 out (1,168 reasoning)
git clean -f -X -d
qwen/qwen3.7-plus:xhigh PASS 100% 66 in → 1,196 out (1,186 reasoning)
git clean -fdX
tencent/hy3:high PASS 100% 66 in → 736 out (729 reasoning)
git clean -fdX
tencent/hy3:high__json_schema PASS 100% 66 in → 2,719 out (2,717 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
tencent/hy3:medium PASS 100% 66 in → 615 out (608 reasoning)
git clean -fdX
tencent/hy3:medium__json_schema PASS 100% 66 in → 541 out (496 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) { "command": "git clean -fXd" }
tencent/hy3:none PASS 100% 69 in → 6 out (0 reasoning)
git clean -fdX
tencent/hy3:none__json_schema PASS 100% 69 in → 13 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
tencent/hy3:xhigh PASS 100% 66 in → 747 out (740 reasoning)
git clean -fXd
tencent/hy3:xhigh__json_schema PASS 100% 66 in → 1,040 out (998 reasoning)
git clean -fXd
JSON Schema Structured Output
(raw) { "command": "git clean -fXd" }
thinkingmachines/inkling-small:high PASS 100% 67 in → 121 out (110 reasoning)
git clean -fdX
thinkingmachines/inkling-small:high__json_schema PASS 100% 67 in → 237 out (222 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
thinkingmachines/inkling-small:low PASS 100% 67 in → 408 out (421 reasoning)
git clean -fdX
thinkingmachines/inkling-small:low__json_schema PASS 100% 67 in → 131 out (116 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
thinkingmachines/inkling-small:max PASS 100% 67 in → 225 out (214 reasoning)
git clean -fdX
thinkingmachines/inkling-small:max__json_schema PASS 100% 67 in → 209 out (191 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
thinkingmachines/inkling-small:medium PASS 100% 67 in → 131 out (134 reasoning)
git clean -fdX
thinkingmachines/inkling-small:medium__json_schema PASS 100% 67 in → 98 out (81 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
thinkingmachines/inkling-small:none PASS 100% 65 in → 11 out (0 reasoning)
git clean -f -d -X
thinkingmachines/inkling-small:xhigh PASS 100% 67 in → 196 out (204 reasoning)
git clean -fdX
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 67 in → 198 out (181 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
thinkingmachines/inkling:high PASS 100% 67 in → 404 out (456 reasoning)
git clean -fdX
thinkingmachines/inkling:high__json_schema PASS 100% 67 in → 212 out (198 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
thinkingmachines/inkling:low PASS 100% 67 in → 64 out (52 reasoning)
git clean -fdX
thinkingmachines/inkling:max PASS 100% 67 in → 359 out (341 reasoning)
git clean -fdX
thinkingmachines/inkling:max__json_schema PASS 100% 67 in → 771 out (751 reasoning)
git clean -f -d -X
JSON Schema Structured Output
(raw) { "command": "git clean -f -d -X" }
thinkingmachines/inkling:medium PASS 100% 67 in → 84 out (73 reasoning)
git clean -fdX
thinkingmachines/inkling:medium__json_schema PASS 100% 67 in → 349 out (332 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
thinkingmachines/inkling:xhigh PASS 100% 67 in → 192 out (181 reasoning)
git clean -fdX
x-ai/grok-4.3:high PASS 100% 237 in → 392 out (387 reasoning)
git clean -Xdf
x-ai/grok-4.3:high__json_schema PASS 100% 293 in → 838 out (829 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
x-ai/grok-4.3:low PASS 100% 243 in → 263 out (258 reasoning)
git clean -Xfd
x-ai/grok-4.3:max PASS 100% 237 in → 367 out (357 reasoning)
```bash git clean -Xdf ```
x-ai/grok-4.3:max__json_schema PASS 100% 293 in → 738 out (729 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) {"command":"git clean -Xfd"}
x-ai/grok-4.3:medium PASS 100% 243 in → 1,082 out (1,077 reasoning)
git clean -Xdf
x-ai/grok-4.3:medium__json_schema PASS 100% 299 in → 701 out (688 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) { "command": "git clean -Xfd" }
x-ai/grok-4.3:none PASS 100% 235 in → 5 out (0 reasoning)
git clean -Xfd
x-ai/grok-4.3:none__json_schema PASS 100% 291 in → 12 out (0 reasoning)
git clean -X -d -f
JSON Schema Structured Output
(raw) {"command":"git clean -X -d -f"}
x-ai/grok-4.3:xhigh PASS 100% 237 in → 777 out (767 reasoning)
```bash git clean -Xfd ```
x-ai/grok-4.3:xhigh__json_schema PASS 100% 293 in → 1,892 out (1,883 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) {"command":"git clean -Xdf"}
x-ai/grok-4.5:high PASS 100% 261 in → 310 out (301 reasoning)
``` git clean -fdX ```
x-ai/grok-4.5:high__json_schema PASS 100% 327 in → 499 out (489 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
x-ai/grok-4.5:low PASS 100% 261 in → 138 out (129 reasoning)
``` git clean -fdX ```
x-ai/grok-4.5:low__json_schema PASS 100% 327 in → 346 out (336 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
x-ai/grok-4.5:max PASS 100% 261 in → 429 out (420 reasoning)
``` git clean -fdX ```
x-ai/grok-4.5:max__json_schema PASS 100% 327 in → 418 out (408 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
x-ai/grok-4.5:medium PASS 100% 261 in → 322 out (313 reasoning)
``` git clean -fdX ```
x-ai/grok-4.5:medium__json_schema PASS 100% 327 in → 539 out (529 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command":"git clean -fdX"}
x-ai/grok-4.5:xhigh PASS 100% 261 in → 949 out (940 reasoning)
``` git clean -Xdf ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 327 in → 549 out (539 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
z-ai/glm-4.7-flash:none PASS 100% 58 in → 11 out (0 reasoning)
rm -rf __pycache__ build dist *.o
z-ai/glm-4.7-flash:none__json_schema PASS 100% 58 in → 13 out (0 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 58 in → 1,365 out (1,356 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) {"command": "git clean -Xfd"}
z-ai/glm-5.2:high PASS 100% 65 in → 285 out (263 reasoning)
git clean -Xfd
z-ai/glm-5.2:low PASS 100% 65 in → 169 out (162 reasoning)
git clean -Xfd
z-ai/glm-5.2:low__json_schema PASS 100% 65 in → 240 out (226 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) { "command": "git clean -fdX" }
z-ai/glm-5.2:medium PASS 100% 65 in → 508 out (501 reasoning)
git clean -fdX
z-ai/glm-5.2:medium__json_schema PASS 100% 67 in → 446 out (429 reasoning)
git clean -X -d -f
JSON Schema Structured Output
(raw) { "command": "git clean -X -d -f" }
z-ai/glm-5.2:none__json_schema PASS 100% 59 in → 15 out (0 reasoning)
git clean -dX -f
JSON Schema Structured Output
(raw) { "command": "git clean -dX -f" }
z-ai/glm-5.2:xhigh PASS 100% 65 in → 650 out (643 reasoning)
git clean -fXd
z-ai/glm-5.2:xhigh__json_schema PASS 100% 65 in → 13 out (12 reasoning)
git clean -Xdf
JSON Schema Structured Output
(raw) { "command": "git clean -Xdf" }
arcee-ai/trinity-mini:low FAIL 80% 62 in → 381 out (415 reasoning)
git clean -fdx
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
arcee-ai/trinity-mini:low__json_schema FAIL 80% 62 in → 386 out (409 reasoning)
git clean -fdx
JSON Schema Structured Output
(raw) { "command": "git clean -fdx" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
arcee-ai/trinity-mini:medium FAIL 80% 62 in → 310 out (335 reasoning)
git clean -fdx
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
arcee-ai/trinity-mini:medium__json_schema FAIL 80% 62 in → 389 out (444 reasoning)
git clean -fdx
JSON Schema Structured Output
(raw) { "command": "git clean -fdx" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
arcee-ai/trinity-mini:xhigh FAIL 80% 62 in → 379 out (415 reasoning)
git clean -fdx
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
arcee-ai/trinity-mini:xhigh__json_schema FAIL 80% 62 in → 421 out (458 reasoning)
git clean -fdx
JSON Schema Structured Output
(raw) { "command": "git clean -fdx" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
git clean -xdf
JSON Schema Structured Output
(raw) { "command": "git clean -xdf" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
inclusionai/ling-2.6-flash FAIL 80% 79 in → 12 out
```sh git clean -fd -x ```
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
minimax/minimax-m2.5:medium__json_schema FAIL 80% 91 in → 539 out (525 reasoning)
git clean -fdx
JSON Schema Structured Output
(raw) { "command": "git clean -fdx" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 80% 72 in → 18 out (0 reasoning)
git clean -fdx --exclude='!'
JSON Schema Structured Output
(raw) { "command": "git clean -fdx --exclude='!'" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
openai/gpt-5.4-nano:medium__json_schema FAIL 80% 93 in → 38 out (0 reasoning)
git clean -ndX -f --exclude=notes.md && git clean -d -f -X --exclude=notes.md
JSON Schema Structured Output
(raw) {"command":"git clean -ndX -f --exclude=notes.md && git clean -d -f -X --exclude=notes.md"}
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
poolside/laguna-xs-2.1:none__json_schema FAIL 80% 70 in → 21 out (0 reasoning)
git clean -fdx -e '!notes.md'
JSON Schema Structured Output
(raw) { "command": "git clean -fdx -e '!notes.md'" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
qwen/qwen3.6-27b:none__json_schema FAIL 80% 68 in → 19 out (0 reasoning)
git clean -x -f -d
JSON Schema Structured Output
(raw) { "command": "git clean -x -f -d" }
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
qwen/qwen3.7-plus:none FAIL 80% 68 in → 5 out (0 reasoning)
git clean -fdx
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
z-ai/glm-4.7-flash:low FAIL 80% 58 in → 501 out (487 reasoning)
git clean -fxd
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
z-ai/glm-4.7-flash:low__json_schema FAIL 80% 58 in → 1,666 out (1,666 reasoning)
git clean -xd -f
JSON Schema Structured Output
(raw) {"command": "git clean -xd -f"}
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
z-ai/glm-4.7-flash:medium FAIL 80% 58 in → 718 out (702 reasoning)
git clean -dxf
Failure: Failed assertions: [{'type': 'file_exists', 'path': 'notes.md'}]
anthropic/claude-haiku-4.5:medium FAIL 60% 106 in → 253 out (215 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-opus-4.7:xhigh FAIL 60% 99 in → 12 out (0 reasoning)
`git clean -fX`
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-sonnet-4.6:low FAIL 60% 77 in → 12 out (0 reasoning)
``` git clean -fX ```
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-sonnet-4.6:low__json_schema FAIL 60% 240 in → 12 out (0 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) {"command":"git clean -fX"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-sonnet-4.6:medium FAIL 60% 77 in → 12 out (0 reasoning)
``` git clean -fX ```
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-sonnet-4.6:medium__json_schema FAIL 60% 240 in → 12 out (0 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) {"command":"git clean -fX"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-sonnet-4.6:none FAIL 60% 77 in → 12 out (0 reasoning)
``` git clean -fX ```
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-sonnet-4.6:none__json_schema FAIL 60% 240 in → 12 out (0 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) {"command":"git clean -fX"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-sonnet-5:low FAIL 60% 94 in → 13 out (0 reasoning)
``` git clean -Xf ```
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-sonnet-5:medium FAIL 60% 94 in → 13 out (0 reasoning)
``` git clean -Xf ```
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
arcee-ai/trinity-large-thinking:low FAIL 60% 62 in → 1,032 out (1,033 reasoning)
git clean -X -f
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
arcee-ai/trinity-large-thinking:medium FAIL 60% 62 in → 995 out (992 reasoning)
git clean -X -f
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
deepseek/deepseek-v4-flash-0731:low FAIL 60% 60 in → 501 out (506 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 60% 60 in → 386 out (391 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) { "command": "git clean -fX" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
deepseek/deepseek-v4-flash-0731:medium__json_schema FAIL 60% 202 in → 138 out (120 reasoning)
git clean -f -X
JSON Schema Structured Output
(raw) {"command": "git clean -f -X"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
deepseek/deepseek-v4-flash-0731:xhigh FAIL 60% 60 in → 85 out (86 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
deepseek/deepseek-v4-flash:high__json_schema FAIL 60% 152 in → 12 out (0 reasoning)
git clean -f -X
JSON Schema Structured Output
(raw) {"command": "git clean -f -X"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
google/gemini-3.5-flash-lite:low FAIL 60% 64 in → 5 out (0 reasoning)
git clean -Xf
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
google/gemini-3.6-flash:low__json_schema FAIL 60% 65 in → 16 out (0 reasoning)
git clean -f -X
JSON Schema Structured Output
(raw) { "command": "git clean -f -X" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
google/gemma-4-26b-a4b-it:high FAIL 60% 80 in → 626 out (564 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
ibm-granite/granite-4.1-8b FAIL 60% 74 in → 6 out
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m2.5:high FAIL 60% 91 in → 169 out (162 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m2.5:high__json_schema FAIL 60% 91 in → 445 out (431 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) { "command": "git clean -fX" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m2.5:medium FAIL 60% 91 in → 177 out (186 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m3:high__json_schema FAIL 60% 216 in → 13 out (0 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) { "command": "git clean -fX" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m3:low__json_schema FAIL 60% 216 in → 13 out (0 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) { "command": "git clean -fX" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m3:medium FAIL 60% 229 in → 122 out (123 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m3:xhigh FAIL 60% 229 in → 93 out (101 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m3:xhigh__json_schema FAIL 60% 216 in → 14 out (0 reasoning)
git clean -f -X
JSON Schema Structured Output
(raw) { "command": "git clean -f -X" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
mistralai/mistral-medium-3-5:medium FAIL 60% 71 in → 623 out (648 reasoning)
git clean -f -X
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
mistralai/mistral-medium-3-5:xhigh FAIL 60% 71 in → 617 out (671 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 60% 72 in → 4,342 out (4,796 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) { "command": "git clean -fX" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 60% 72 in → 2,810 out (3,086 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) { "command": "git clean -fX" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema FAIL 60% 72 in → 425 out (452 reasoning)
git clean -X -f
JSON Schema Structured Output
(raw) { "command": "git clean -X -f" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 60% 72 in → 705 out (815 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 60% 72 in → 1,239 out (1,319 reasoning)
git clean -Xf
JSON Schema Structured Output
(raw) { "command": "git clean -Xf" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-super-120b-a12b:high__json_schema FAIL 60% 72 in → 202 out (180 reasoning)
git clean -f -X
JSON Schema Structured Output
(raw) { "command": "git clean -f -X" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-super-120b-a12b:low FAIL 60% 72 in → 224 out (210 reasoning)
git clean -X -f
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-super-120b-a12b:none FAIL 60% 72 in → 7 out (0 reasoning)
git clean -X -f
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-super-120b-a12b:none__json_schema FAIL 60% 72 in → 14 out (0 reasoning)
git clean -Xf
JSON Schema Structured Output
(raw) { "command": "git clean -Xf" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-super-120b-a12b:xhigh FAIL 60% 72 in → 170 out (184 reasoning)
git clean -f -X
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema FAIL 60% 72 in → 118 out (100 reasoning)
git clean -X -f
JSON Schema Structured Output
(raw) { "command": "git clean -X -f" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs-2.1:high__json_schema FAIL 60% 70 in → 240 out (229 reasoning)
git clean -Xf
JSON Schema Structured Output
(raw) {"command": "git clean -Xf"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs-2.1:low FAIL 60% 70 in → 330 out (325 reasoning)
git clean -Xf
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs-2.1:low__json_schema FAIL 60% 70 in → 294 out (281 reasoning)
git clean -X -f
JSON Schema Structured Output
(raw) {"command": "git clean -X -f"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs-2.1:medium__json_schema FAIL 60% 70 in → 332 out (321 reasoning)
git clean -Xf
JSON Schema Structured Output
(raw) {"command": "git clean -Xf"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs.2:high FAIL 60% 107 in → 632 out (623 reasoning)
git clean -X -f
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs.2:high__json_schema FAIL 60% 107 in → 173 out (160 reasoning)
git clean -X -f
JSON Schema Structured Output
(raw) {"command": "git clean -X -f"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs.2:low FAIL 60% 107 in → 222 out (213 reasoning)
git clean -X -f
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs.2:low__json_schema FAIL 60% 107 in → 189 out (176 reasoning)
git clean -X -f
JSON Schema Structured Output
(raw) {"command": "git clean -X -f"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs.2:xhigh FAIL 60% 107 in → 304 out (295 reasoning)
git clean -X -f
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs.2:xhigh__json_schema FAIL 60% 107 in → 210 out (197 reasoning)
git clean -X -f
JSON Schema Structured Output
(raw) {"command": "git clean -X -f"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
qwen/qwen3.6-27b:high FAIL 60% 66 in → 918 out (905 reasoning)
git clean -f -X
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
qwen/qwen3.6-35b-a3b:low FAIL 60% 66 in → 1,667 out (1,597 reasoning)
git clean -Xf
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
qwen/qwen3.6-35b-a3b:medium FAIL 60% 66 in → 1,694 out (1,575 reasoning)
git clean -X -f
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
qwen/qwen3.6-35b-a3b:none FAIL 60% 68 in → 7 out (0 reasoning)
git clean -X -f
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
qwen/qwen3.6-35b-a3b:xhigh FAIL 60% 66 in → 1,887 out (1,874 reasoning)
git clean -Xf
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
qwen/qwen3.6-35b-a3b:xhigh__json_schema FAIL 60% 66 in → 1,367 out (1,249 reasoning)
git clean -f -X
JSON Schema Structured Output
(raw) { "command": "git clean -f -X" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
tencent/hy3:low FAIL 60% 66 in → 1,174 out (1,167 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
tencent/hy3:low__json_schema FAIL 60% 66 in → 1,091 out (1,098 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) {"command":"git clean -fX"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
z-ai/glm-4.7-flash:medium__json_schema FAIL 60% 58 in → 834 out (805 reasoning)
git clean -Xf
JSON Schema Structured Output
(raw) {"command": "git clean -Xf"}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
z-ai/glm-5.2:none FAIL 60% 59 in → 6 out (0 reasoning)
git clean -fX
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-sonnet-5:low__json_schema FAIL 40% 304 in → 16 out (0 reasoning)
git clean -Xd
JSON Schema Structured Output
(raw) {"command": "git clean -Xd"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
anthropic/claude-sonnet-5:none__json_schema FAIL 40% 304 in → 15 out (0 reasoning)
git clean -Xd
JSON Schema Structured Output
(raw) {"command":"git clean -Xd"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
arcee-ai/trinity-mini:high__json_schema FAIL 40% 62 in → 368 out (383 reasoning)
gitclean -fdx
JSON Schema Structured Output
(raw) { "command": "gitclean -fdx" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
deepseek/deepseek-v4-flash:medium__json_schema FAIL 40% 152 in → 11 out (0 reasoning)
git clean -Xdn
JSON Schema Structured Output
(raw) {"command": "git clean -Xdn"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
google/gemini-3.5-flash-lite:high__json_schema FAIL 40% 121 in → 415 out (406 reasoning)
git clean -Xd
JSON Schema Structured Output
(raw) {"command": "git clean -Xd"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
google/gemini-3.5-flash-lite:max__json_schema FAIL 40% 65 in → 449 out (441 reasoning)
git clean -Xd
JSON Schema Structured Output
(raw) {"command":"git clean -Xd"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
google/gemma-4-26b-a4b-it:low__json_schema FAIL 40% 80 in → 330 out (293 reasoning)
git clean -Xd
JSON Schema Structured Output
(raw) { "command": "git clean -Xd" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
google/gemma-4-26b-a4b-it:xhigh FAIL 40% 80 in → 428 out (375 reasoning)
git clean -Xd
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
google/gemma-4-26b-a4b-it:xhigh__json_schema FAIL 40% 80 in → 408 out (349 reasoning)
git clean -Xd
JSON Schema Structured Output
(raw) { "command": "git clean -Xd" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
liquid/lfm-2-24b-a2b FAIL 40% 69 in → 24 out
git rm --ignore-unmatch -r __pycache__/main.pyc build/dist/app module.o
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m2.5:low__json_schema FAIL 40% 91 in → 509 out (572 reasoning)
git clean -X -d
JSON Schema Structured Output
(raw) { "command": "git clean -X -d" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m2.5:xhigh__json_schema FAIL 40% 91 in → 520 out (570 reasoning)
git clean -X -d
JSON Schema Structured Output
(raw) { "command": "git clean -X -d" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
minimax/minimax-m2.7:high__json_schema FAIL 40% 91 in → 17 out (17 reasoning)
git clean -fd --ignore-ignore
JSON Schema Structured Output
(raw) { "command": "git clean -fd --ignore-ignore" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
mistralai/mistral-medium-3-5:none FAIL 40% 71 in → 14 out (0 reasoning)
```sh git clean -fd --dry-run ```
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
nvidia/nemotron-3-nano-30b-a3b:high FAIL 40% 72 in → 691 out (727 reasoning)
gitclean -X -f -d
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
openai/gpt-5.4-nano:low__json_schema FAIL 40% 93 in → 20 out (0 reasoning)
git clean -fdX --dry-run
JSON Schema Structured Output
(raw) {"command":"git clean -fdX --dry-run"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
openai/gpt-5.4-nano:medium FAIL 40% 58 in → 14 out (0 reasoning)
git clean -fdX --dry-run -n
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
openai/gpt-5.4-nano:none FAIL 40% 58 in → 12 out (0 reasoning)
git clean -fdX --dry-run
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
openai/gpt-oss-20b:xhigh__json_schema FAIL 40% 119 in → 26 out (0 reasoning)
git rm --cached --ignore-unmatch * --exclude .gitignore?...??}
JSON Schema Structured Output
(raw) { "command":"git rm --cached --ignore-unmatch * --exclude .gitignore?...??}"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-m.1:medium FAIL 40% 70 in → 322 out (315 reasoning)
git clean -Xd
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs.2:medium FAIL 40% 107 in → 844 out (833 reasoning)
`git clean -d -x`
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs.2:medium__json_schema FAIL 40% 107 in → 2,382 out (2,364 reasoning)
git clean -f -x
JSON Schema Structured Output
(raw) { "command": "git clean -f -x" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}, {'type': 'file_exists', 'path': 'notes.md'}]
poolside/laguna-xs.2:none FAIL 40% 107 in → 6 out (0 reasoning)
git clean -i
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
poolside/laguna-xs.2:none__json_schema FAIL 40% 107 in → 10 out (0 reasoning)
git clean -n
JSON Schema Structured Output
(raw) {"command": "git clean -n"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
qwen/qwen3.6-27b:xhigh FAIL 40% 66 in → 1,420 out (1,365 reasoning)
git clean -Xd
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
qwen/qwen3.6-flash:none FAIL 40% 68 in → 4 out (0 reasoning)
git clean -X
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
qwen/qwen3.7-flash:max FAIL 40% 66 in → 930 out (921 reasoning)
git clean -X
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
qwen/qwen3.7-flash:xhigh FAIL 40% 66 in → 1,783 out (1,774 reasoning)
git clean -X
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
thinkingmachines/inkling:none FAIL 40% 67 in → 11 out (0 reasoning)
git clean -X -d -f<|end_message|>
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
x-ai/grok-4.3:low__json_schema FAIL 40% 299 in → 496 out (488 reasoning)
git clean -Xd
JSON Schema Structured Output
(raw) {"command":"git clean -Xd"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
z-ai/glm-4.7-flash:high FAIL 40% 58 in → 694 out (697 reasoning)
git clean -Xd
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}]
arcee-ai/trinity-mini:high FAIL 20% 62 in → 473 out (531 reasoning)
git clean -fd
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}, {'type': 'file_exists', 'path': 'notes.md'}]
z-ai/glm-4.7-flash:xhigh FAIL 20% 58 in → 502 out (470 reasoning)
git clean -f -d
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'module.o'}, {'type': 'dir_not_exists', 'path': '__pycache__'}, {'type': 'dir_not_exists', 'path': 'build'}, {'type': 'file_exists', 'path': 'notes.md'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git rm --ignore-unmatch -r . --exclude="**/__pycache__/*" --exclude="build/dist/app/*" --exclude="module.o"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-696b871f-de3b-9514-b123-aa97bd77228e","request_id":"696b871f-de3b-9514-b123-aa97bd77228e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d7c1927b-a4c2-9102-b3a0-7e9bd3e2dc99","request_id":"d7c1927b-a4c2-9102-b3a0-7e9bd3e2dc99"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-edc4010a-4f4e-9f6b-8ee6-69b0877a8883","request_id":"edc4010a-4f4e-9f6b-8ee6-69b0877a8883"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2255f285-0cec-9cba-99c8-730250984443","request_id":"2255f285-0cec-9cba-99c8-730250984443"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-339bd62c-fa9d-9f84-8c79-e7c36133cc14","request_id":"339bd62c-fa9d-9f84-8c79-e7c36133cc14"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-23865fae-ff74-9f4f-a662-d8dcfba5de90"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c98afdbb-a9cf-9a26-9af1-7f5df13f67b1"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7bc986e4-e243-9f90-9eda-dc2729eccfb3"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-23b0072d-6175-9976-ac9d-6c7df040f6c0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3a248b13-186c-94f0-95db-fe5848f8a8fc"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-75f4fa2f-bf46-9e3c-b18b-fe1dcb74802f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f9de0b19-99ad-9df3-b003-b5997acd3b40","request_id":"f9de0b19-99ad-9df3-b003-b5997acd3b40"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c535c3f9-e8c3-9533-bb97-fcd63576893c","request_id":"c535c3f9-e8c3-9533-bb97-fcd63576893c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e316f29c-0483-9951-b5bd-54a232016608","request_id":"e316f29c-0483-9951-b5bd-54a232016608"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0c0b68db-9284-98eb-ba9e-269faefec148","request_id":"0c0b68db-9284-98eb-ba9e-269faefec148"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f67339c6-0c25-9f61-8d18-cfde4f4394cc","request_id":"f67339c6-0c25-9f61-8d18-cfde4f4394cc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3feca1c0-f40a-9dad-b1e1-7fa8e9c93ba0","request_id":"3feca1c0-f40a-9dad-b1e1-7fa8e9c93ba0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5d4ea975-f608-912f-adec-3d38e1e8c0ad","request_id":"5d4ea975-f608-912f-adec-3d38e1e8c0ad"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2c69caf5-29ff-9f61-b2c7-41de15761662","request_id":"2c69caf5-29ff-9f61-b2c7-41de15761662"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3f8a14f7-6fc1-99de-9085-51fac0bdddb5","request_id":"3f8a14f7-6fc1-99de-9085-51fac0bdddb5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a14d22c6-930c-99a7-b4e6-c1c0cb214081","request_id":"a14d22c6-930c-99a7-b4e6-c1c0cb214081"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: git clean -f -d -X{"command": "git clean -f -d -X"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fdX
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -Xd -n{"command": "git clean -Xdi"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fdX
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -Xfd
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)