: GitBench

f007 — cherry_pick

Loading campaigns…
← cherry_pick
Three-way: conflicting port and protocol settings in same line range
Tests ability to resolve a three-way conflict with port and protocol settings. Evaluates preference-based conflict resolution with multiple configuration dimensions.
hard cherry-pick conflict-resolution three-way port protocol
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'PORT=8080 PROTOCOL=http' > config.txt
  5. 05 git add config.txt
  6. 06 git commit -m 'Initial config'
  7. 07 git checkout -b branch-a
  8. 08 printf 'PORT=8443 PROTOCOL=https' > config.txt
  9. 09 git add config.txt
  10. 10 git commit -m 'Use HTTPS port'
  11. 11 git checkout main
  12. 12 printf 'PORT=9090 PROTOCOL=http' > config.txt
  13. 13 git add config.txt
  14. 14 git commit -m 'Change to non-standard port'
  15. 15 git cherry-pick $(git rev-parse branch-a)
Prompt
Resolve the cherry-pick conflict in config.txt. The base PORT was 8080/PROTOCOL=http. The incoming commit changed both to PORT=8443/PROTOCOL=https. The current branch (main) has PORT=9090/PROTOCOL=http. Prefer the HTTPS configuration when there is a conflict. Provide ONLY the resolved config.txt content.
Expected
PORT=8443 PROTOCOL=https
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
deepseek/deepseek-v4-flash:high PASS 100% 128 in → 116 out (106 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 131 in → 139 out (123 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
deepseek/deepseek-v4-flash:none PASS 100% 129 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 130 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
mistralai/devstral-2512__json_schema PASS 100% 139 in → 19 out
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 154 in → 206 out (195 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 152 in → 42 out (18 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 153 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 154 in → 22 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
poolside/laguna-xs.2:high PASS 100% 180 in → 181 out (168 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs.2:high__json_schema PASS 100% 177 in → 195 out (176 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
poolside/laguna-xs.2:none PASS 100% 179 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs.2:none__json_schema PASS 100% 178 in → 17 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
mistralai/devstral-2512 FAIL 0% 140 in → 30 out
Here's the resolved `config.txt` content preferring the HTTPS configuration: ``` PORT=8443 PROTOCOL=https ```
Failure: Expected 'PORT=8443 PROTOCOL=https ', got 'Here's the resolved `config.txt` content preferring the HTTPS configuration: ``` PORT=8443 PROTOCOL=https ```'