Path-limited search restricting to a subdirectory
Tests ability to restrict search to a subdirectory path. Evaluates path-limited search scoping.
Baseline Repository
These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.
- 01
git init - 02
git config user.email 'test@test.com' - 03
git config user.name 'Test User' - 04
mkdir -p src tests - 05
echo 'def connect(): return True' > src/db.py - 06
echo 'def test_connect(): assert connect()' > tests/test_db.py - 07
echo 'def helper(): pass' > src/utils.py - 08
echo 'def test_helper(): assert helper()' > tests/test_utils.py - 09
git add . - 10
git commit -m 'Add source and test files' - 11
echo 'git grep def -- src/' > .grep_command - 12
git add .grep_command - 13
git commit -m 'Add grep sentinel'
Prompt
Here is the output of a git grep command limited to the src/ directory. How many function definitions appear in the output? Output ONLY the number, nothing else.
Expected
2
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
2
2
JSON Schema
Structured Output
(raw) { "count": 2 }
2
2
JSON Schema
Structured Output
(raw) {"count": 2}
2
2
JSON Schema
Structured Output
(raw) {
"count": 2
}
2
2
JSON Schema
Structured Output
(raw) {
"count": 2
}
2
2
JSON Schema
Structured Output
(raw) {
"count": 2
}
2
2
JSON Schema
Structured Output
(raw) {
"count": 2
}
Invalid structured output. Output: 2
JSON Schema
Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
3
Failure: Expected '2', got '3'