: GitBench

f004 — submodule_usage

Loading campaigns…
← submodule_usage
Check submodule status
Tests ability to check submodule status for modifications. Evaluates submodule-status inspection.
easy submodule status inspection
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
  10. 10 git -c protocol.file.allow=always submodule add ../lib-bare lib
  11. 11 git add .gitmodules lib
  12. 12 git commit -m 'Add lib submodule'
Prompt
Run the command to show the status of all submodules. Output ONLY the git command, nothing else.
Expected
git submodule status
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
deepseek/deepseek-v4-flash:high PASS 100% 82 in → 213 out (208 reasoning)
git submodule status
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 83 in → 77 out (71 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:none PASS 100% 83 in → 5 out (0 reasoning)
git submodule status
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 80 in → 13 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
mistralai/devstral-2512__json_schema PASS 100% 90 in → 10 out
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 102 in → 206 out (226 reasoning)
git submodule status
poolside/laguna-xs.2:high PASS 100% 137 in → 85 out (79 reasoning)
git submodule status
poolside/laguna-xs.2:high__json_schema PASS 100% 139 in → 224 out (214 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs.2:none PASS 100% 135 in → 5 out (0 reasoning)
git submodule status
poolside/laguna-xs.2:none__json_schema PASS 100% 141 in → 9 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/devstral-2512 FAIL 0% 88 in → 10 out
```bash git submodule status ```
Failure: Expected 'git submodule status', got '```bash git submodule status ```'
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 0% 101 in → 207 out (227 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) { "command": "git submodule status --recursive" }
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 100 in → 7 out (0 reasoning)
git submodule status -ssh
Failure: Expected 'git submodule status', got 'git submodule status -ssh'
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 103 in → 16 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) { "command": "git submodule status --recursive" }
Failure: Expected 'git submodule status', got 'git submodule status --recursive'