Documentation update
Tests ability to generate a commit message for a documentation update (README). Evaluates recognizing documentation-only changes.
Baseline Repository
These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.
- 01
git init - 02
git config user.email 'test@test.com' - 03
git config user.name 'Test User' - 04
printf '# Project\nThis is a project.\n' > README.md - 05
git add README.md - 06
git commit -m 'Initial commit' - 07
printf '# Project\nThis is a project.\n\n## Installation\nRun `pip install` to install dependencies.\n' > README.md - 08
git add README.md
Prompt
Generate a concise commit message (max 50 characters) describing the changes in this diff. Focus on the action taken (e.g., 'Add', 'Fix', 'Update').
Expected
Add installation section to README
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
Add installation instructions to README
JSON Schema
Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
Add installation instructions to README
JSON Schema
Structured Output
(raw) { "commit_message": "Add installation instructions to README" }
Add installation section to README
JSON Schema
Structured Output
(raw) {"commit_message": "Add installation section to README"}
Add installation instructions to README
JSON Schema
Structured Output
(raw) {"commit_message": "Add installation instructions to README"}
Add Installation section to README
JSON Schema
Structured Output
(raw) {"commit_message": "Add Installation section to README"}
Add Installation section to README
Update README with installation instructions
Add installation instructions to README
"Add installation instructions to README"
Add Installation section to README
Add Installation section to README
JSON Schema
Structured Output
(raw) {
"commit_message": "Add Installation section to README"
}
Update: Add Installation section
JSON Schema
Structured Output
(raw) { "commit_message": "Update: Add Installation section" }
Add Installation section with pip install instructions.
Update README with installation instructions