: GitBench

f011 — submodule_usage

Loading campaigns…
← submodule_usage
View .gitmodules content
Tests ability to view .gitmodules file content for submodule configuration. Evaluates submodule-configuration inspection.
easy submodule gitmodules config inspection
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
  10. 10 git -c protocol.file.allow=always submodule add ../lib-bare lib
  11. 11 git add .gitmodules lib
  12. 12 git commit -m 'Add lib submodule'
Prompt
View the contents of the .gitmodules file to verify the submodule configuration. Output ONLY the command to run (e.g., cat .gitmodules), nothing else.
Expected
cat .gitmodules
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
deepseek/deepseek-v4-flash:high PASS 100% 93 in → 70 out (65 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 95 in → 115 out (105 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
deepseek/deepseek-v4-flash:none PASS 100% 96 in → 4 out (0 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 97 in → 13 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
mistralai/devstral-2512 PASS 100% 101 in → 5 out
cat .gitmodules
mistralai/devstral-2512__json_schema PASS 100% 102 in → 10 out
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 109 in → 359 out (399 reasoning)
cat .gitmodules
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 113 in → 106 out (104 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 115 in → 5 out (0 reasoning)
cat .gitmodules
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 116 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
poolside/laguna-xs.2:high PASS 100% 147 in → 56 out (49 reasoning)
cat .gitmodules
poolside/laguna-xs.2:high__json_schema PASS 100% 150 in → 64 out (48 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
poolside/laguna-xs.2:none PASS 100% 148 in → 6 out (0 reasoning)
cat .gitmodules
poolside/laguna-xs.2:none__json_schema PASS 100% 152 in → 10 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}