: GitBench
Remove a submodule
Tests ability to properly remove a submodule (deinit + rm). Evaluates complete submodule-removal procedure knowledge.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
  10. 10 git -c protocol.file.allow=always submodule add ../lib-bare lib
  11. 11 git add .gitmodules lib
  12. 12 git commit -m 'Add lib submodule'
Prompt
Remove the submodule 'lib' completely. Output ONLY the commands to run, one per line. The .gitmodules file should no longer reference 'lib' after removal.
Expected
git submodule deinit -f lib git rm -f lib rm -rf .git/modules/lib
Loading campaign evidence…
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 95 in → 438 out (397 reasoning)
git submodule deinit -f lib git rm -f lib rm -rf .git/modules/lib
JSON Schema Structured Output
(raw) { "commands": [ "git submodule deinit -f lib", "git rm -f lib", "rm -rf .git/modules/lib" ] }
deepseek/deepseek-v4-flash:none PASS 100% 96 in → 20 out (0 reasoning)
git submodule deinit lib git rm lib rm -rf .git/modules/lib
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 88 in → 49 out (0 reasoning)
git submodule deinit -f lib rm -rf .git/modules/lib git rm -f lib rm -f .gitmodules
JSON Schema Structured Output
(raw) { "commands": [ "git submodule deinit -f lib", "rm -rf .git/modules/lib", "git rm -f lib", "rm -f .gitmodules" ] }
mistralai/devstral-2512__json_schema PASS 100% 101 in → 78 out
git submodule deinit -f lib git rm -f lib rm -rf .git/modules/lib git config -f .gitmodules --remove-section submodule.lib git add .gitmodules git commit -m "Remove submodule 'lib'"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule deinit -f lib", "git rm -f lib", "rm -rf .git/modules/lib", "git config -f .gitmodules --remove-section submodule.lib", "git add .gitmodules", "git commit -m \"Remove submodule 'lib'\"" ] }
poolside/laguna-xs.2:high PASS 100% 152 in → 563 out (533 reasoning)
git submodule deinit lib git rm lib rm -rf .git/modules/lib git commit -m "Remove lib submodule"
deepseek/deepseek-v4-flash:high FAIL 75% 94 in → 2,384 out (2,512 reasoning)
git submodule deinit -f lib git rm -f lib git config -f .gitmodules --remove-section submodule.lib git config --remove-section submodule.lib git add .gitmodules
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '.git/modules/lib'}]
nvidia/nemotron-3-nano-30b-a3b:high FAIL 75% 119 in → 804 out (771 reasoning)
git submodule deinit -f lib git rm -f lib git commit -m "Remove lib submodule" sed -i '/^\[submodule "lib"\]/,/^\[/d' .gitmodules git add .gitmodules git commit -m "Update .gitmodules after submodule removal"
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '.git/modules/lib'}]
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 75% 115 in → 482 out (385 reasoning)
git submodule deinit -f lib git rm -f lib git commit -m "Remove lib submodule" sed -i '/\[submodule \"lib\"\]/,/url = \/\/\/../lib-bare/d' .gitmodules git add .gitmodules git commit -m "Update .gitmodules after submodule removal"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule deinit -f lib", "git rm -f lib", "git commit -m \"Remove lib submodule\"", "sed -i '/\\[submodule \\\"lib\\\"\\]/,/url = \\/\\/\\/../lib-bare/d' .gitmodules", "git add .gitmodules", "git commit -m \"Update .gitmodules after submodule removal\"" ] }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '.git/modules/lib'}]
nvidia/nemotron-3-nano-30b-a3b:none FAIL 75% 117 in → 64 out (0 reasoning)
git submodule deinit -f lib git rm --cached lib git commit -m "Remove submodule lib" git submodule sync rm -rf .git/modules/lib rm -rf lib git add .gitmodules .gitignore git commit -m "Cleanup submodule lib"
Failure: Failed assertions: [{'type': 'file_not_contains', 'path': '.gitmodules', 'value': 'lib'}]
poolside/laguna-xs.2:none FAIL 75% 154 in → 40 out (0 reasoning)
git rm lib git commit -m "Remove lib submodule" rm .gitmodules git add .gitmodules git commit -m "Remove lib entry from .gitmodules"
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '.git/modules/lib'}]
poolside/laguna-xs.2:none__json_schema FAIL 75% 153 in → 57 out (0 reasoning)
git rm lib git config --file .gitmodules --remove-section submodule.lib git add .gitmodules git commit -m "Remove lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git rm lib", "git config --file .gitmodules --remove-section submodule.lib", "git add .gitmodules", "git commit -m \"Remove lib submodule\"" ] }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': '.git/modules/lib'}]
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 50% 114 in → 106 out (0 reasoning)
git submodule deinit -f lib git rm -f --cached lib git commit -m "Remove lib submodule" git submodule foreach 'git rm -f --cached $name' git config -f .gitmodules --remove-section submodule.lib git rm .gitmodules git clean -fdx git gc --prune=now --aggressive
JSON Schema Structured Output
(raw) { "commands": [ "git submodule deinit -f lib", "git rm -f --cached lib", "git commit -m \"Remove lib submodule\"", "git submodule foreach 'git rm -f --cached $name'", "git config -f .gitmodules --remove-section submodule.lib", "git rm .gitmodules", "git clean -fdx", "git gc --prune=now --aggressive" ] }
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'lib'}, {'type': 'dir_not_exists', 'path': '.git/modules/lib'}]
poolside/laguna-xs.2:high__json_schema FAIL 50% 155 in → 993 out (952 reasoning)
git rm --cached lib git config --remove-section submodule.lib rm -rf .git/modules/lib git commit -m "Remove lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git rm --cached lib", "git config --remove-section submodule.lib", "rm -rf .git/modules/lib", "git commit -m \"Remove lib submodule\""]}
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'lib'}, {'type': 'file_not_contains', 'path': '.gitmodules', 'value': 'lib'}]
mistralai/devstral-2512 FAIL 0% 102 in → 59 out
```bash git submodule deinit -f lib git rm -f lib rm -rf .git/modules/lib git config -f .gitmodules --remove-section submodule.lib git add .gitmodules git commit -m "Remove submodule 'lib'" ```
Failure: Failed assertions: [{'type': 'dir_not_exists', 'path': 'lib'}, {'type': 'dir_not_exists', 'path': '.git/modules/lib'}, {'type': 'file_not_contains', 'path': '.gitmodules', 'value': 'lib'}, {'type': 'git_output', 'command': 'git submodule status', 'not_contains': 'lib'}]