Search commit messages using git log --grep
Tests ability to search commit messages using git log --grep. Evaluates understanding of log-search vs file-search.
Baseline Repository
These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.
- 01
git init - 02
git config user.email 'test@test.com' - 03
git config user.name 'Test User' - 04
echo 'alpha' > alpha.txt - 05
git add alpha.txt - 06
git commit -m 'Add alpha module' - 07
echo 'beta' > beta.txt - 08
git add beta.txt - 09
git commit -m 'Fix beta parsing bug' - 10
echo 'gamma' > gamma.txt - 11
git add gamma.txt - 12
git commit -m 'Add gamma feature' - 13
echo 'updated alpha' > alpha.txt - 14
git add alpha.txt - 15
git commit -m 'Fix alpha edge case' - 16
echo 'git log --oneline --grep=Fix' > .grep_command - 17
git add .grep_command - 18
git commit -m 'Add grep sentinel'
Prompt
Here is the output of a git log --grep command run on this repository. How many commits contain the word 'Fix' in their message? Output ONLY the number, nothing else.
Expected
2
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
2
2
JSON Schema
Structured Output
(raw) {
"count": 2
}
2
2
JSON Schema
Structured Output
(raw) {"count": 2}
2
2
JSON Schema
Structured Output
(raw) {
"count": 2
}
2
2
JSON Schema
Structured Output
(raw) { "count": 2 }
2
2
JSON Schema
Structured Output
(raw) {
"count": 2
}
2
2
JSON Schema
Structured Output
(raw) {
"count": 2
}
Invalid structured output. Output: 2
JSON Schema
Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
3
Failure: Expected '2', got '3'