: GitBench

f004 — commit_messages

Loading campaigns…
← commit_messages
File deleted
Tests ability to generate a commit message for a file deletion. Evaluates handling of removal operations.
easy commit-message deletion remove
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'to be deleted' > redundant.txt
  5. 05 git add redundant.txt
  6. 06 git commit -m 'Add file'
  7. 07 git rm redundant.txt
Prompt
Generate a concise commit message (max 50 characters) describing the changes in this diff. Focus on the action taken (e.g., 'Add', 'Fix', 'Update', 'Remove').
Expected
Remove redundant.txt
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
mistralai/devstral-2512 PASS 96% 118 in → 4 out
Remove redundant.txt
mistralai/devstral-2512__json_schema PASS 95% 118 in → 10 out
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
poolside/laguna-xs.2:high__json_schema PASS 94.3% 160 in → 174 out (163 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 94% 131 in → 445 out (324 reasoning)
Delete redundant.txt
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 93.7% 131 in → 12 out (0 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
poolside/laguna-xs.2:none PASS 93.7% 160 in → 6 out (0 reasoning)
Remove redundant.txt file
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 93.3% 131 in → 108 out (98 reasoning)
Remove redundant.txt
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant.txt" }
deepseek/deepseek-v4-flash:high PASS 92.7% 108 in → 104 out (97 reasoning)
Remove redundant.txt
deepseek/deepseek-v4-flash:none PASS 91.7% 108 in → 4 out (0 reasoning)
Remove redundant file
deepseek/deepseek-v4-flash:none__json_schema PASS 91.7% 108 in → 12 out (0 reasoning)
Remove redundant file
JSON Schema Structured Output
(raw) { "commit_message": "Remove redundant file" }
poolside/laguna-xs.2:high PASS 91% 160 in → 175 out (169 reasoning)
Remove redundant.txt
poolside/laguna-xs.2:none__json_schema PASS 90% 160 in → 11 out (0 reasoning)
Remove redundant.txt file
JSON Schema Structured Output
(raw) {"commit_message": "Remove redundant.txt file"}
nvidia/nemotron-3-nano-30b-a3b:none PASS 87.7% 131 in → 6 out (0 reasoning)
Remove file: redundant.txt
deepseek/deepseek-v4-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)