: GitBench

f012 — submodule_usage

Loading campaigns…
← submodule_usage
Add submodule with specific branch
Tests ability to add a submodule tracking a specific branch. Evaluates branch-specific submodule configuration.
hard submodule branch tracking config
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib main' > lib.py && git add lib.py && git commit -m 'Initial lib on main' && git checkout -b develop && echo 'lib develop' > dev.py && git add dev.py && git commit -m 'Develop branch'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
Prompt
Add the repository at '../lib-bare' as a submodule at path 'lib', tracking the 'develop' branch. Output ONLY the git submodule command, nothing else.
Expected
git submodule add -b develop ../lib-bare lib
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
deepseek/deepseek-v4-flash:high PASS 100% 46 in → 143 out (140 reasoning)
git submodule add -b develop ../lib-bare lib
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 110 in → 67 out (49 reasoning)
git submodule add -b develop ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add -b develop ../lib-bare lib"}
deepseek/deepseek-v4-flash:none PASS 100% 46 in → 13 out (0 reasoning)
git submodule add -b develop ../lib-bare lib
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 46 in → 21 out (0 reasoning)
git submodule add -b develop ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add -b develop ../lib-bare lib" }
mistralai/devstral-2512__json_schema PASS 100% 46 in → 18 out
git submodule add -b develop ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add -b develop ../lib-bare lib"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 59 in → 154 out (154 reasoning)
git submodule add -b develop ../lib-bare lib
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 59 in → 84 out (69 reasoning)
git submodule add --branch develop ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add --branch develop ../lib-bare lib" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 59 in → 13 out (0 reasoning)
git submodule add -b develop ../lib-bare lib
poolside/laguna-xs.2:high PASS 100% 91 in → 177 out (164 reasoning)
git submodule add --branch develop ../lib-bare lib
poolside/laguna-xs.2:high__json_schema PASS 100% 91 in → 117 out (101 reasoning)
git submodule add -b develop ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add -b develop ../lib-bare lib"}
poolside/laguna-xs.2:none PASS 100% 91 in → 13 out (0 reasoning)
git submodule add -b develop ../lib-bare lib
poolside/laguna-xs.2:none__json_schema PASS 100% 91 in → 17 out (0 reasoning)
git submodule add -b develop ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add -b develop ../lib-bare lib"}
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 66.7% 59 in → 28 out (0 reasoning)
git submodule add ../lib-bare lib && git submodule foreach "git checkout develop"
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib && git submodule foreach \"git checkout develop\"" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'cat .gitmodules', 'contains': 'develop'}]
mistralai/devstral-2512 FAIL 0% 46 in → 18 out
```bash git submodule add -b develop ../lib-bare lib ```
Failure: Failed assertions: [{'type': 'dir_exists', 'path': 'lib'}, {'type': 'file_exists', 'path': 'lib/dev.py'}, {'type': 'git_output', 'command': 'cat .gitmodules', 'contains': 'develop'}]