: GitBench

f004 — git_clean

Loading campaigns…
← git_clean
Remove ignored files only
Tests ability to remove only ignored files with git clean -fX. Evaluates understanding of gitignore interaction.
medium git-clean ignored gitignore
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf '*.log node_modules/ ' > .gitignore
  5. 05 git add .gitignore
  6. 06 git commit -m 'Add gitignore'
  7. 07 echo 'data' > app.log
  8. 08 mkdir -p node_modules/pkg
  9. 09 echo '{}' > node_modules/pkg/index.js
  10. 10 echo 'untracked' > untracked.txt
Prompt
Remove only ignored files and directories while keeping untracked files. Output ONLY the git command, nothing else.
Expected
git clean -f -X -d
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
deepseek/deepseek-v4-flash:high PASS 100% 57 in → 126 out (120 reasoning)
git clean -f -d -X
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 121 in → 134 out (130 reasoning)
git clean -fdX
JSON Schema Structured Output
(raw) {"command": "git clean -fdX"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 69 in → 688 out (748 reasoning)
git clean -X -fd .
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 69 in → 1,102 out (1,189 reasoning)
git clean -Xfd
JSON Schema Structured Output
(raw) { "command": "git clean -Xfd" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 69 in → 14 out (0 reasoning)
git clean -X -f app.log node_modules/pkg/index.js
deepseek/deepseek-v4-flash:none__json_schema FAIL 66.7% 59 in → 13 out (0 reasoning)
git clean -fX
JSON Schema Structured Output
(raw) { "command": "git clean -fX" }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'node_modules'}]
poolside/laguna-xs.2:high FAIL 66.7% 103 in → 152 out (143 reasoning)
git clean -X -f
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'node_modules'}]
deepseek/deepseek-v4-flash:none FAIL 33.3% 57 in → 5 out (0 reasoning)
git clean -n
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'app.log'}, {'type': 'dir_not_exists', 'path': 'node_modules'}]
mistralai/devstral-2512 FAIL 33.3% 56 in → 11 out
```bash git clean -fdX ```
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'app.log'}, {'type': 'dir_not_exists', 'path': 'node_modules'}]
mistralai/devstral-2512__json_schema FAIL 33.3% 56 in → 12 out
git clean -X -d
JSON Schema Structured Output
(raw) {"command": "git clean -X -d"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'app.log'}, {'type': 'dir_not_exists', 'path': 'node_modules'}]
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 33.3% 69 in → 11 out (0 reasoning)
git clean -Xn
JSON Schema Structured Output
(raw) {"command": "git clean -Xn"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'app.log'}, {'type': 'dir_not_exists', 'path': 'node_modules'}]
poolside/laguna-xs.2:high__json_schema FAIL 33.3% 103 in → 478 out (462 reasoning)
git clean -X
JSON Schema Structured Output
(raw) { "command": "git clean -X" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'app.log'}, {'type': 'dir_not_exists', 'path': 'node_modules'}]
poolside/laguna-xs.2:none FAIL 33.3% 103 in → 12 out (0 reasoning)
git clean -d --exclude='!' -i
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'app.log'}, {'type': 'dir_not_exists', 'path': 'node_modules'}]
poolside/laguna-xs.2:none__json_schema FAIL 33.3% 103 in → 11 out (0 reasoning)
git clean -Xdn
JSON Schema Structured Output
(raw) {"command": "git clean -Xdn"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'app.log'}, {'type': 'dir_not_exists', 'path': 'node_modules'}]