: GitBench

f008 — submodule_usage

Loading campaigns…
← submodule_usage
Deinit a submodule
Tests ability to deinitialize a submodule without removing it. Evaluates submodule-deinit command understanding.
medium submodule deinit deactivate
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
  10. 10 git -c protocol.file.allow=always submodule add ../lib-bare lib
  11. 11 git add .gitmodules lib
  12. 12 git commit -m 'Add lib submodule'
Prompt
Deinitialize the submodule 'lib' (remove its working tree but keep the config). Output ONLY the git command, nothing else.
Expected
git submodule deinit -f lib
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
deepseek/deepseek-v4-flash:high PASS 100% 87 in → 124 out (136 reasoning)
git submodule deinit lib
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 86 in → 844 out (827 reasoning)
git submodule deinit -f lib
JSON Schema Structured Output
(raw) { "command": "git submodule deinit -f lib" }
deepseek/deepseek-v4-flash:none PASS 100% 87 in → 7 out (0 reasoning)
git submodule deinit lib
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 88 in → 15 out (0 reasoning)
git submodule deinit lib
JSON Schema Structured Output
(raw) { "command": "git submodule deinit lib" }
mistralai/devstral-2512__json_schema PASS 100% 93 in → 14 out
git submodule deinit -f lib
JSON Schema Structured Output
(raw) {"command": "git submodule deinit -f lib"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 111 in → 1,160 out (1,251 reasoning)
git submodule deinit -f lib
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 110 in → 781 out (819 reasoning)
git submodule deinit -f lib
JSON Schema Structured Output
(raw) { "command": "git submodule deinit -f lib" }
poolside/laguna-xs.2:high__json_schema PASS 100% 146 in → 150 out (133 reasoning)
git submodule deinit lib
JSON Schema Structured Output
(raw) { "command": "git submodule deinit lib" }
mistralai/devstral-2512 FAIL 33.3% 93 in → 14 out
```bash git submodule deinit -f lib ```
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'lib/lib.py'}, {'type': 'git_output', 'command': 'git config --get submodule.lib.url', 'not_contains': 'lib-bare'}]
nvidia/nemotron-3-nano-30b-a3b:none FAIL 33.3% 109 in → 29 out (0 reasoning)
git submodule deinit --force --merge ../lib-bare/lib && git rm --force lib && rm -rf .git/modules/lib
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'lib/lib.py'}, {'type': 'git_output', 'command': 'git config --get submodule.lib.url', 'not_contains': 'lib-bare'}]
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 33.3% 105 in → 28 out (0 reasoning)
git submodule deinit -f --lib && git rm --cached --force --quiet lib
JSON Schema Structured Output
(raw) { "command": "git submodule deinit -f --lib && git rm --cached --force --quiet lib" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'lib/lib.py'}, {'type': 'git_output', 'command': 'git config --get submodule.lib.url', 'not_contains': 'lib-bare'}]
poolside/laguna-xs.2:high FAIL 33.3% 144 in → 98 out (91 reasoning)
git deinit lib
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'lib/lib.py'}, {'type': 'git_output', 'command': 'git config --get submodule.lib.url', 'not_contains': 'lib-bare'}]
poolside/laguna-xs.2:none FAIL 33.3% 144 in → 6 out (0 reasoning)
git deinit lib
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'lib/lib.py'}, {'type': 'git_output', 'command': 'git config --get submodule.lib.url', 'not_contains': 'lib-bare'}]
poolside/laguna-xs.2:none__json_schema FAIL 33.3% 141 in → 15 out (0 reasoning)
git deinit lib
JSON Schema Structured Output
(raw) { "command": "git deinit lib" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'lib/lib.py'}, {'type': 'git_output', 'command': 'git config --get submodule.lib.url', 'not_contains': 'lib-bare'}]