f002 — submodule_usage

: GitBench

← submodule_usage

Initialize submodules after clone

Tests ability to initialize submodules after cloning a repository. Evaluates understanding of submodule clone workflow.

medium submodule init clone workflow

Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

01 git init
02 git config user.email 'test@test.com'
03 git config user.name 'Test User'
04 echo 'App' > app.txt
05 git add app.txt
06 git commit -m 'Initial commit'
07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
08 git clone --bare ../lib-src ../lib-bare
09 git config protocol.file.allow always
10 git -c protocol.file.allow=always submodule add ../lib-bare lib
11 git add .gitmodules lib
12 git commit -m 'Add lib submodule'
13 rm -rf lib

Prompt

The submodule directory 'lib' was removed. Run the commands to re-initialize and update the submodule. Output ONLY the commands to run, one per line.

Expected

git submodule init git submodule update

Campaign Evidence

Loading campaign evidence…

Model Outputs (14)

deepseek/deepseek-v4-flash:high PASS 100% 92 in → 218 out (207 reasoning)

git submodule init git submodule update

deepseek/deepseek-v4-flash:none PASS 100% 88 in → 12 out (0 reasoning)

git submodule init lib git submodule update lib

deepseek/deepseek-v4-flash:none__json_schema PASS 100% 92 in → 23 out (0 reasoning)

git submodule init git submodule update

JSON Schema Structured Output

(raw) { "commands": [ "git submodule init", "git submodule update" ] }

mistralai/devstral-2512__json_schema PASS 100% 99 in → 24 out

git submodule init git submodule update

JSON Schema Structured Output

(raw) { "commands": [ "git submodule init", "git submodule update" ] }

nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 109 in → 471 out (525 reasoning)

git submodule init git submodule update

nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 112 in → 504 out (527 reasoning)

git submodule sync git submodule update --init --recursive

JSON Schema Structured Output

(raw) { "commands": [ "git submodule sync", "git submodule update --init --recursive" ] }

nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 115 in → 10 out (0 reasoning)

git submodule init git submodule update

nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 107 in → 29 out (0 reasoning)

git submodule sync git submodule update --init --recursive

JSON Schema Structured Output

(raw) { "commands": [ "git submodule sync", "git submodule update --init --recursive" ] }

poolside/laguna-xs.2:high PASS 100% 147 in → 119 out (109 reasoning)

git submodule init git submodule update

poolside/laguna-xs.2:high__json_schema PASS 100% 149 in → 212 out (197 reasoning)

git submodule init git submodule update

JSON Schema Structured Output

(raw) {"commands": ["git submodule init", "git submodule update"]}

poolside/laguna-xs.2:none__json_schema PASS 100% 148 in → 14 out (0 reasoning)

git submodule init git submodule update

JSON Schema Structured Output

(raw) {"commands": ["git submodule init", "git submodule update"]}

poolside/laguna-xs.2:none FAIL 50% 146 in → 83 out (0 reasoning)

git init lib git -C lib add remote origin ../lib-bare git -C lib fetch origin git -C lib checkout -b main f4996651d37385df003b13fb0955f4d40b2daacf git add lib git commit -m "Re-add lib submodule"

Failure: Failed assertions: [{'type': 'file_content', 'path': 'lib/lib.py', 'value': 'lib code'}]

deepseek/deepseek-v4-flash:high__json_schema FAIL 0%

Invalid JSON. Output:

JSON Schema Structured Output

Structured Output Error

Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)

Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)

mistralai/devstral-2512 FAIL 0% 98 in → 15 out

```bash git submodule init git submodule update ```

Failure: Failed assertions: [{'type': 'dir_exists', 'path': 'lib'}, {'type': 'file_content', 'path': 'lib/lib.py', 'value': 'lib code'}]