: GitBench

f003 — git_clean

Loading campaigns…
← git_clean
Remove untracked directories
Tests ability to remove untracked directories with git clean -fd. Evaluates recursive cleanup awareness.
easy git-clean directories recursive
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'tracked' > tracked.txt
  5. 05 git add tracked.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p build/output
  8. 08 echo 'artifact' > build/output/app.bin
  9. 09 mkdir -p tmp
  10. 10 echo 'temp' > tmp/cache.txt
Prompt
Remove all untracked files and directories from the working directory. Output ONLY the git command, nothing else.
Expected
git clean -f -d
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
deepseek/deepseek-v4-flash:high PASS 100% 57 in → 69 out (63 reasoning)
git clean -fd
deepseek/deepseek-v4-flash:none PASS 100% 57 in → 4 out (0 reasoning)
git clean -fd
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 57 in → 13 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
mistralai/devstral-2512__json_schema PASS 100% 54 in → 10 out
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 67 in → 173 out (180 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 67 in → 6 out (0 reasoning)
git clean -fdx
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 67 in → 15 out (0 reasoning)
git clean -d -f
JSON Schema Structured Output
(raw) { "command": "git clean -d -f" }
poolside/laguna-xs.2:high PASS 100% 106 in → 145 out (138 reasoning)
git clean -fd
poolside/laguna-xs.2:high__json_schema PASS 100% 106 in → 166 out (150 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) { "command": "git clean -fd" }
poolside/laguna-xs.2:none PASS 100% 106 in → 6 out (0 reasoning)
git clean -fd
poolside/laguna-xs.2:none__json_schema PASS 100% 106 in → 10 out (0 reasoning)
git clean -fd
JSON Schema Structured Output
(raw) {"command": "git clean -fd"}
mistralai/devstral-2512 FAIL 33.3% 54 in → 10 out
```bash git clean -fd ```
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'build'}, {'type': 'dir_not_exists', 'path': 'tmp'}]
nvidia/nemotron-3-nano-30b-a3b:high FAIL 33.3% 67 in → 32,768 out (66 reasoning)
(empty output)
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'build'}, {'type': 'dir_not_exists', 'path': 'tmp'}]
deepseek/deepseek-v4-flash:high__json_schema FAIL 0%
Invalid JSON. Output: gitclean -fd
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)