Multiple patterns with git grep -e
Tests ability to search for multiple patterns with git grep -e. Evaluates OR-pattern matching in search.
Baseline Repository
These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.
- 01
git init - 02
git config user.email 'test@test.com' - 03
git config user.name 'Test User' - 04
mkdir -p src - 05
echo 'import requests import json import os API_URL = "https://api.example.com" def fetch_data(endpoint): resp = requests.get(f"{API_URL}/{endpoint}") return json.loads(resp.text) def save_to_file(data, path): with open(path, "w") as f: json.dump(data, f) def load_from_file(path): with open(path) as f: return json.load(f) ' > src/io.py - 06
git add . - 07
git commit -m 'Add IO module' - 08
echo 'git grep -e json -e requests' > .grep_command - 09
git add .grep_command - 10
git commit -m 'Add grep sentinel'
Prompt
Here is the output of a git grep -e command that searches for multiple patterns ('json' and 'requests'). How many total matching lines appear? Output ONLY the number, nothing else.
Expected
6
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
6
6
6
JSON Schema
Structured Output
(raw) {
"count": 6
}
6
JSON Schema
Structured Output
(raw) {"count": 6}
6
6
JSON Schema
Structured Output
(raw) {
"count": 6
}
6
JSON Schema
Structured Output
(raw) { "count": 6 }
6
6
JSON Schema
Structured Output
(raw) {"count":6}
Invalid JSON. Output:
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
3
Failure: Expected numeric answer '6', got '3'
7
Failure: Expected numeric answer '6', got '7'
7
JSON Schema
Structured Output
(raw) {"count": 7}
Failure: Expected numeric answer '6', got '7'