: GitBench

f006 — submodule_usage

Loading campaigns…
← submodule_usage
List submodules
Tests ability to list all submodules in a repository. Evaluates submodule-listing command knowledge.
easy submodule list enumeration
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
  10. 10 git -c protocol.file.allow=always submodule add ../lib-bare lib
  11. 11 git add .gitmodules lib
  12. 12 git commit -m 'Add lib submodule'
Prompt
Run the command to list all configured submodules. Output ONLY the git command, nothing else.
Expected
git submodule
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
deepseek/deepseek-v4-flash:high PASS 100% 79 in → 215 out (209 reasoning)
git submodule status
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 86 in → 362 out (349 reasoning)
git submodule
JSON Schema Structured Output
(raw) { "command": "git submodule" }
deepseek/deepseek-v4-flash:none PASS 100% 77 in → 4 out (0 reasoning)
git submodule
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 81 in → 12 out (0 reasoning)
git submodule
JSON Schema Structured Output
(raw) { "command": "git submodule" }
mistralai/devstral-2512__json_schema PASS 100% 86 in → 10 out
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 99 in → 427 out (442 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 100 in → 5 out (0 reasoning)
git submodule status
poolside/laguna-xs.2:high PASS 100% 133 in → 139 out (133 reasoning)
git submodule status
poolside/laguna-xs.2:high__json_schema PASS 100% 133 in → 286 out (276 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs.2:none PASS 100% 132 in → 5 out (0 reasoning)
git submodule status
poolside/laguna-xs.2:none__json_schema PASS 100% 135 in → 9 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/devstral-2512 FAIL 0% 87 in → 10 out
```bash git submodule status ```
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
nvidia/nemotron-3-nano-30b-a3b:high FAIL 0% 100 in → 488 out (538 reasoning)
git submodule --list
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 100 in → 16 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) { "command": "git submodule status --recursive" }
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']