Experimental commits that should be combined
Tests ability to identify experimental commits that should be combined into a single coherent change. Evaluates recognizing exploratory work patterns.
Baseline Repository
These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.
- 01
git init - 02
git config user.email 'test@test.com' - 03
git config user.name 'Test User' - 04
echo 'base' > base.txt - 05
git add base.txt - 06
git commit -m 'Base setup' - 07
echo 'exp1' > experiment.py - 08
git add experiment.py - 09
git commit -m 'Experiment v1' - 10
echo 'exp2' > experiment.py - 11
git add experiment.py - 12
git commit -m 'Experiment v2' - 13
echo 'final' > experiment.py - 14
git add experiment.py - 15
git commit -m 'Experiment final'
Prompt
Find the intermediate commits that should be marked as squash/fixup to clean up experimental work. These are consecutive commits with similar purpose that were split during development. Do not include the final experiment commit as one of the commits to squash. Output ONLY the selected commit subject lines, one per line, with no explanation.
Expected
Experiment v1
Experiment v2
Fixture Metrics
Scoped model quality, cost, API time, and token usage for commit_squash/f005.
Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
```
Experiment v2
Experiment v1
```
Failure: Extra selected commit messages: ['```', '```']
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v1
Experiment v2
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": ["Experiment v2", "Experiment v1"]
}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": ["Experiment v2", "Experiment v1"] }
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v1",
"Experiment v2"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": [ "Experiment v2", "Experiment v1" ] }
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v1",
"Experiment v2"
]
}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v1",
"Experiment v2"
]
}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v1",
"Experiment v2"
]
}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v1",
"Experiment v2"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v1",
"Experiment v2"
]
}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw)
{
"commits"
:
[
"Experiment v2"
,
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": ["Experiment v2", "Experiment v1"] }
Experiment v2
Experiment v1
Experiment v1
Experiment v2
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": ["Experiment v2", "Experiment v1"] }
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v1
Experiment v2
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": [
"Experiment v2",
"Experiment v1"
]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits":["Experiment v1","Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
Experiment final
Experiment v2
Experiment v1
Base setup
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment final",
"Experiment v2",
"Experiment v1",
"Base setup"
]
}
Failure: Extra selected commit messages: ['Experiment final', 'Base setup']
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": ["Experiment v1", "Experiment v2"]
}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v1",
"Experiment v2"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v1
Experiment v2
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": [
"Experiment v2",
"Experiment v1"
] }
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": [
"Experiment v2",
"Experiment v1"
] }
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": ["Experiment v2", "Experiment v1"] }
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v1",
"Experiment v2"
]
}
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": [ "Experiment v2", "Experiment v1" ] }
Experiment v2
Experiment v1
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) { "commits": [ "Experiment v1", "Experiment v2" ] }
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": [ "Experiment v2", "Experiment v1" ] }
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v1
Experiment v2
Experiment v1
Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v1", "Experiment v2"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) { "commits": ["Experiment v2", "Experiment v1"] }
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["Experiment v2","Experiment v1"]}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v1
Experiment v2
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"Experiment v2",
"Experiment v1"
]
}
Experiment v2
Experiment v1
Experiment v2
Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["Experiment v2", "Experiment v1"]}
Experiment v2
Experiment v1
Experiment v1
Failure: Missing expected commit messages: ['Experiment v2']
Experiment v1
Failure: Missing expected commit messages: ['Experiment v2']
00e72f8 Experiment v1
c6d5957 Experiment v2
JSON Schema
Structured Output
(raw) {"commits": ["00e72f8 Experiment v1", "c6d5957 Experiment v2"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v1', 'Experiment v2']
43c5eae Experiment v2
7c4ab6c Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
e301775 Experiment v2
5cfc741 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
0840e03 Experiment v2
67297f9 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
bb368be Experiment v2
e2248eb Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
fb96ed2 Experiment v2
e34e869 Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["fb96ed2 Experiment v2","e34e869 Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
501f436 Experiment v2
9be91bb Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["501f436 Experiment v2","9be91bb Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
Invalid JSON. Output:
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
6abb9f9 Experiment final
bbe45fd Experiment v2
759ce4c Experiment v1
f490f23 Base setup
JSON Schema
Structured Output
(raw) {
"commits": [
"6abb9f9 Experiment final",
"bbe45fd Experiment v2",
"759ce4c Experiment v1",
"f490f23 Base setup"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment final', 'Experiment v2', 'Experiment v1', 'Base setup']
21ffe78 Experiment final
1261592 Experiment v2
d31417b Experiment v1
51ce9c0 Base setup
JSON Schema
Structured Output
(raw) {
"commits": [
"21ffe78 Experiment final",
"1261592 Experiment v2",
"d31417b Experiment v1",
"51ce9c0 Base setup"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment final', 'Experiment v2', 'Experiment v1', 'Base setup']
0e8b685 Experiment final
80a2ef2 Experiment v2
ec7f10b Experiment v1
9256d0d Base setup
JSON Schema
Structured Output
(raw) {
"commits": [
"0e8b685 Experiment final",
"80a2ef2 Experiment v2",
"ec7f10b Experiment v1",
"9256d0d Base setup"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment final', 'Experiment v2', 'Experiment v1', 'Base setup']
Experiment v2Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2']
f8e6708 Experiment final
b1d25c3 Experiment v2
4f1c038 Experiment v1
dba38d7 Base setup
JSON Schema
Structured Output
(raw) {
"commits": [
"f8e6708 Experiment final",
"b1d25c3 Experiment v2",
"4f1c038 Experiment v1",
"dba38d7 Base setup"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment final', 'Experiment v2', 'Experiment v1', 'Base setup']
ceafe21 Experiment v2
c4bd66f Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
5675e7b Experiment v2
1c7f935 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
75e673c Experiment v1
e3eb0e1 Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": [
"75e673c Experiment v1",
"e3eb0e1 Experiment v2"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v1', 'Experiment v2']
f238273 Experiment v2
aa53008 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
11211b5
47f2afc
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
9457253 Experiment v2
a17f38f Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"9457253 Experiment v2",
"a17f38f Experiment v1"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
da6fa6d Experiment v2
cb6f67e Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
714516e Experiment v1
7ce3dd6 Experiment v2
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v1', 'Experiment v2']
Invalid JSON. Output: Experiment v2
Experiment v1
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
2593cd7
9a91a24
JSON Schema
Structured Output
(raw) {"commits":["2593cd7","9a91a24"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
32d2519 Experiment v2
b85172e Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["32d2519 Experiment v2", "b85172e Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
87c3560 Experiment v2
2310c14 Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"87c3560 Experiment v2",
"2310c14 Experiment v1"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
6a583b9 Experiment v1
92c8fb1 Experiment v2
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v1', 'Experiment v2']
cadb922 Experiment v2
4dfe3b2 Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"cadb922 Experiment v2",
"4dfe3b2 Experiment v1"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
77a53d4 Experiment v2
45b5693 Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["77a53d4 Experiment v2", "45b5693 Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
(empty output)
JSON Schema
Structured Output
(raw) {"commits":[]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']
b4d34a4 Experiment v2
151d5b1 Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["b4d34a4 Experiment v2", "151d5b1 Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
Invalid JSON. Output: a2acebf Experiment v2
ddcfd24 Experiment v1
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
f78cd80 Experiment v2
64a9ff5 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
9b25abc Experiment v2
c1ab1a8 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
c22f949 Experiment v2
128572b Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["c22f949 Experiment v2", "128572b Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
Invalid JSON. Output:
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
84843a7 Experiment final
90d25a6 Experiment v2
65ff9b5 Experiment v1
2ec0658 Base setup
JSON Schema
Structured Output
(raw) {
"commits": [
"84843a7 Experiment final",
"90d25a6 Experiment v2",
"65ff9b5 Experiment v1",
"2ec0658 Base setup"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment final', 'Experiment v2', 'Experiment v1', 'Base setup']
4216951 Experiment v2
209a4bc Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"4216951 Experiment v2",
"209a4bc Experiment v1"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
a5e792c Experiment v2
ae083ba Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
ad474a2 Experiment v2
5d21d6e Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
6a79516 Experiment v2
67c01bf Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
4618933 Experiment v2
b8b8477 Experiment v1
JSON Schema
Structured Output
(raw)
{
"commits": [
"4618933 Experiment v2",
"b8b8477 Experiment v1"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
268595f Experiment v2
220d999 Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"268595f Experiment v2",
"220d999 Experiment v1"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
d4c7e44 Experiment v2
ff86feb Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
1330928 Experiment v1
42ac56a Experiment v2
JSON Schema
Structured Output
(raw) {
"commits": ["1330928 Experiment v1", "42ac56a Experiment v2"]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v1', 'Experiment v2']
ad72556 Experiment v2
824a1fc Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["ad72556 Experiment v2","824a1fc Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
0ca54c6 Experiment v2
2fcbd31 Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["0ca54c6 Experiment v2","2fcbd31 Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
bc6e4ff Experiment v2
3e2a39e Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["bc6e4ff Experiment v2","3e2a39e Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
d76babc Experiment v2
3359a00 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
ab96caa Experiment v2
91b3801 Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"ab96caa Experiment v2",
"91b3801 Experiment v1"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
68d0e27 Experiment v2
970dbdd Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
b066686 Experiment v2
c6962c1 Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["b066686 Experiment v2", "c6962c1 Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
2eba30d Experiment v2
921dab5 Experiment v1
JSON Schema
Structured Output
(raw) {"commits": ["2eba30d Experiment v2", "921dab5 Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-325ac47b-2ece-909d-876d-6bc14bbe97ea","request_id":"325ac47b-2ece-909d-876d-6bc14bbe97ea"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d1fb61dc-4f70-97d7-a94d-411eda429cfc","request_id":"d1fb61dc-4f70-97d7-a94d-411eda429cfc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-151f5090-1130-92a0-a8c4-ce92a4a46a5d","request_id":"151f5090-1130-92a0-a8c4-ce92a4a46a5d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-da4d7f69-1f02-90ad-82ad-86f7082178a2","request_id":"da4d7f69-1f02-90ad-82ad-86f7082178a2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b406bcb2-ec68-9c66-85a3-9af73f43e85b","request_id":"b406bcb2-ec68-9c66-85a3-9af73f43e85b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-db2eb0b4-4e19-9381-b956-ab443e2ad275"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-22460b30-2382-9cae-bcb8-14721ea8991e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a3392e4c-1d45-9f74-9f28-6a60a010aeff"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-990743c4-b38b-9602-bec7-5db299bd98e7"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-010098f2-b11d-90d5-9fba-2c607fe8790a"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-de4d2075-4d96-99d6-b185-173eeeab99cf"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-330ec3b5-44d9-9aaf-917d-b47c7347da39","request_id":"330ec3b5-44d9-9aaf-917d-b47c7347da39"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8247ace6-2da4-9e96-a668-c14edf6f77d1","request_id":"8247ace6-2da4-9e96-a668-c14edf6f77d1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-41b831f2-f439-928a-a89d-a42fb6fb8a7f","request_id":"41b831f2-f439-928a-a89d-a42fb6fb8a7f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-45e445ed-616d-9327-a898-488ca3b4167b","request_id":"45e445ed-616d-9327-a898-488ca3b4167b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5ed3a12c-d27b-9738-822e-12984f6db5e9","request_id":"5ed3a12c-d27b-9738-822e-12984f6db5e9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7c66f58f-20e0-9a4f-b8bd-df7d01605316","request_id":"7c66f58f-20e0-9a4f-b8bd-df7d01605316"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-992223ad-5109-98ee-9fe0-867c09dfacf5","request_id":"992223ad-5109-98ee-9fe0-867c09dfacf5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
e96ca12 Experiment v2
0cf30a2 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-aa582034-a442-9247-b2b5-8fbd3dd4b24e","request_id":"aa582034-a442-9247-b2b5-8fbd3dd4b24e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5ce4a7b3-d0dd-91e9-83b0-6519452d3c6f","request_id":"5ce4a7b3-d0dd-91e9-83b0-6519452d3c6f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema
Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-53bc3e22-ed23-934e-8149-63abcc14a2ac","request_id":"53bc3e22-ed23-934e-8149-63abcc14a2ac"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
08d05de Experiment v2
dcb8cb0 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
Invalid JSON. Output: Experiment v1
Experiment v2{
"commits": [
"67e55ea Experiment v1",
"d43cc31 Experiment v2"
]
}
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
66850bc Experiment v2
b0856d7 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
a204df1 Experiment v2
292a314 Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["a204df1 Experiment v2","292a314 Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
617d882 Experiment v2
569a933 Experiment v1
JSON Schema
Structured Output
(raw) {"commits":["617d882 Experiment v2","569a933 Experiment v1"]}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
Invalid JSON. Output:
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Experiment v1
Experiment v2
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
ece2d45 Experiment v2
dcae09d Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
dddcfc8 Experiment v2
07a7a72 Experiment v1
JSON Schema
Structured Output
(raw) { "commits": ["dddcfc8 Experiment v2", "07a7a72 Experiment v1"] }
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
18f017e Experiment v2
f5e2120 Experiment v1
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
7ee2f74 Experiment v2
af4aa60 Experiment v1
JSON Schema
Structured Output
(raw) {
"commits": [
"7ee2f74 Experiment v2",
"af4aa60 Experiment v1"
]
}
Failure: Missing expected commit messages: ['Experiment v1', 'Experiment v2']; Extra selected commit messages: ['Experiment v2', 'Experiment v1']
Invalid JSON. Output: Experiment v2
Experiment v1
JSON Schema
Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)