: GitBench

f001 — submodule_usage

Loading campaigns…
← submodule_usage
Add a submodule
Tests ability to add a git submodule to a repository. Evaluates basic submodule-add command knowledge.
easy submodule add basic
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
Prompt
Add the repository at '../lib-bare' as a git submodule at path 'lib'. Output ONLY the git command to run, nothing else.
Expected
git submodule add ../lib-bare lib
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 40 in → 82 out (63 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-flash:none PASS 100% 40 in → 10 out (0 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 42 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
mistralai/devstral-2512__json_schema PASS 100% 39 in → 15 out
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 52 in → 86 out (80 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 52 in → 69 out (52 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 52 in → 19 out (0 reasoning)
git submodule add '../lib-bare' lib
JSON Schema Structured Output
(raw) { "command": "git submodule add '../lib-bare' lib" }
poolside/laguna-xs.2:high PASS 100% 86 in → 84 out (74 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs.2:high__json_schema PASS 100% 86 in → 89 out (75 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
poolside/laguna-xs.2:none PASS 100% 86 in → 10 out (0 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs.2:none__json_schema PASS 100% 86 in → 14 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
deepseek/deepseek-v4-flash:high FAIL 0% 40 in → 100 out (90 reasoning)
gitsubmodule add ../lib-bare lib
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git submodule status', 'contains': 'lib'}]
mistralai/devstral-2512 FAIL 0% 39 in → 15 out
```bash git submodule add ../lib-bare lib ```
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git submodule status', 'contains': 'lib'}]
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 52 in → 19 out (0 reasoning)
git submodule add '../lib-bare' lib -m 'Add lib submodule'
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git submodule status', 'contains': 'lib'}]