Multi-line hunk: config block conflict
Tests ability to resolve a configuration block merge conflict with port and host changes. Evaluates infrastructure-merge judgment.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'server: port: 8080 host: localhost' > config.yaml
  5. 05 git add config.yaml
  6. 06 git commit -m 'Initial config'
  7. 07 git checkout -b prod
  8. 08 printf 'server: port: 443 host: localhost' > config.yaml
  9. 09 git add config.yaml
  10. 10 git commit -m 'Use HTTPS port'
  11. 11 git checkout main
  12. 12 printf 'server: port: 8080 host: 0.0.0.0' > config.yaml
  13. 13 git add config.yaml
  14. 14 git commit -m 'Listen on all interfaces'
  15. 15 git merge prod
Prompt
Resolve the merge conflict in config.yaml. The base had port: 8080 and host: localhost. Branch A changed port to 443 (HTTPS) keeping localhost. Branch B kept port 8080 but changed host to 0.0.0.0. Provide ONLY the resolved file content.
Expected
server: port: 443 host: 0.0.0.0

Scoped model quality, cost, API time, and token usage for merge_conflicts/f006.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high__json_schema PASS 100% 400 in → 60 out (25 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-fable-5:low__json_schema PASS 100% 400 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-fable-5:medium__json_schema PASS 100% 400 in → 60 out (25 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 400 in → 122 out (73 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:high PASS 100% 162 in → 332 out (273 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 330 in → 586 out (467 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:low PASS 100% 162 in → 341 out (304 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 330 in → 700 out (518 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
anthropic/claude-haiku-4.5:medium PASS 100% 162 in → 1,975 out (880 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 330 in → 728 out (586 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
anthropic/claude-haiku-4.5:none PASS 100% 133 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 300 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 162 in → 393 out (335 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 330 in → 1,490 out (789 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:high PASS 100% 133 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 301 in → 216 out (145 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:low PASS 100% 133 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 301 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:max PASS 100% 133 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 301 in → 223 out (148 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:medium PASS 100% 133 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 301 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:none PASS 100% 133 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 301 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 133 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 301 in → 218 out (131 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:high PASS 100% 185 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
anthropic/claude-opus-4.7:high__json_schema PASS 100% 405 in → 39 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:low PASS 100% 185 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
anthropic/claude-opus-4.7:low__json_schema PASS 100% 405 in → 39 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:max PASS 100% 185 in → 332 out (159 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 405 in → 302 out (66 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:medium PASS 100% 185 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 405 in → 39 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:none PASS 100% 185 in → 32 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.7:none__json_schema PASS 100% 405 in → 40 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 185 in → 32 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 405 in → 39 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 400 in → 157 out (64 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:low PASS 100% 180 in → 86 out (49 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 400 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:max PASS 100% 180 in → 705 out (114 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 400 in → 540 out (177 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 400 in → 104 out (59 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 400 in → 36 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 180 in → 498 out (129 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 400 in → 184 out (53 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 301 in → 220 out (138 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 133 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 301 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 301 in → 400 out (200 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 133 in → 159 out (93 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 301 in → 80 out (29 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 133 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 301 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 301 in → 279 out (162 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:high PASS 100% 180 in → 71 out (24 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 400 in → 75 out (37 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:low PASS 100% 180 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 400 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:medium PASS 100% 180 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 400 in → 68 out (27 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:none PASS 100% 180 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 400 in → 36 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 180 in → 81 out (43 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 400 in → 100 out (51 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 120 in → 2,412 out (2,404 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-large-thinking:low PASS 100% 120 in → 1,045 out (1,025 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
arcee-ai/trinity-large-thinking:medium PASS 100% 120 in → 1,236 out (1,214 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
arcee-ai/trinity-large-thinking:xhigh PASS 100% 120 in → 1,089 out (1,071 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-mini:high PASS 100% 120 in → 690 out (701 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-mini:high__json_schema PASS 100% 120 in → 510 out (492 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
arcee-ai/trinity-mini:low PASS 100% 120 in → 330 out (326 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-mini:low__json_schema PASS 100% 120 in → 622 out (595 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 202 in → 503 out (435 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 202 in → 266 out (211 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 123 in → 240 out (189 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 129 in → 608 out (519 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 202 in → 273 out (240 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 202 in → 1,352 out (1,255 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 123 in → 341 out (320 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 129 in → 838 out (689 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 123 in → 19 out (0 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 123 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 202 in → 185 out (148 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 202 in → 2,892 out (2,783 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash:high PASS 100% 122 in → 269 out (248 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 125 in → 198 out (166 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash:low PASS 100% 123 in → 247 out (204 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 220 in → 27 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
deepseek/deepseek-v4-flash:medium PASS 100% 123 in → 352 out (331 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 125 in → 265 out (233 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash:none PASS 100% 123 in → 25 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 123 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 202 in → 594 out (520 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 202 in → 544 out (496 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-pro:high PASS 100% 123 in → 703 out (678 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:low PASS 100% 123 in → 341 out (320 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 123 in → 719 out (639 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-pro:medium PASS 100% 123 in → 366 out (346 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 315 in → 266 out (237 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
deepseek/deepseek-v4-pro:none PASS 100% 123 in → 25 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 125 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 202 in → 803 out (782 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 393 in → 623 out (594 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3-flash-preview:high PASS 100% 126 in → 398 out (376 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:high__json_schema PASS 100% 186 in → 1,258 out (1,228 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3-flash-preview:low PASS 100% 126 in → 239 out (217 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:low__json_schema PASS 100% 186 in → 926 out (897 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3-flash-preview:medium PASS 100% 126 in → 551 out (529 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:medium__json_schema PASS 100% 186 in → 2,105 out (2,076 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3-flash-preview:xhigh PASS 100% 126 in → 449 out (427 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 186 in → 919 out (889 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 126 in → 663 out (641 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:low PASS 100% 126 in → 156 out (134 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 186 in → 163 out (128 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 126 in → 292 out (270 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 127 in → 599 out (577 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 127 in → 1,953 out (1,924 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-pro-preview:high PASS 100% 126 in → 905 out (883 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 186 in → 1,441 out (1,410 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.1-pro-preview:low PASS 100% 126 in → 321 out (299 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 186 in → 177 out (140 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
google/gemini-3.1-pro-preview:medium PASS 100% 126 in → 342 out (320 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 186 in → 955 out (926 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 126 in → 361 out (339 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 186 in → 1,468 out (1,433 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash-lite:high PASS 100% 126 in → 529 out (507 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 127 in → 1,526 out (1,496 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash-lite:low PASS 100% 127 in → 22 out (0 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 186 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash-lite:max PASS 100% 127 in → 682 out (660 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 186 in → 1,140 out (1,110 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash-lite:medium PASS 100% 127 in → 357 out (335 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 186 in → 765 out (730 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 126 in → 716 out (694 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 186 in → 811 out (776 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.5-flash:high__json_schema PASS 100% 186 in → 1,114 out (1,083 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.5-flash:low PASS 100% 126 in → 412 out (390 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash:low__json_schema PASS 100% 186 in → 339 out (304 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.5-flash:medium PASS 100% 126 in → 517 out (495 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash:medium__json_schema PASS 100% 186 in → 544 out (513 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.5-flash:xhigh PASS 100% 126 in → 764 out (742 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 186 in → 1,046 out (1,016 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.6-flash:high PASS 100% 127 in → 659 out (637 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:high__json_schema PASS 100% 127 in → 805 out (775 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.6-flash:low PASS 100% 127 in → 394 out (372 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:low__json_schema PASS 100% 186 in → 35 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.6-flash:max PASS 100% 127 in → 766 out (744 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:max__json_schema PASS 100% 127 in → 1,035 out (1,004 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.6-flash:medium PASS 100% 127 in → 597 out (575 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:medium__json_schema PASS 100% 186 in → 594 out (559 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.6-flash:xhigh PASS 100% 127 in → 600 out (578 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 186 in → 708 out (679 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemma-4-26b-a4b-it:high PASS 100% 142 in → 1,482 out (1,158 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 142 in → 1,871 out (1,566 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:low PASS 100% 141 in → 988 out (795 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 141 in → 975 out (727 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:medium PASS 100% 142 in → 1,312 out (1,064 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 142 in → 740 out (555 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:none PASS 100% 139 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 139 in → 36 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 142 in → 1,075 out (861 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 142 in → 1,330 out (1,065 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-31b-it:high PASS 100% 142 in → 323 out (219 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:high__json_schema PASS 100% 142 in → 242 out (136 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-31b-it:low PASS 100% 142 in → 397 out (293 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:low__json_schema PASS 100% 141 in → 308 out (200 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-31b-it:medium__json_schema PASS 100% 145 in → 300 out (269 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemma-4-31b-it:none PASS 100% 139 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:none__json_schema PASS 100% 139 in → 36 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-31b-it:xhigh PASS 100% 142 in → 313 out (1 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 141 in → 236 out (152 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
inclusionai/ling-2.6-flash__json_schema PASS 100% 146 in → 35 out
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
minimax/minimax-m2.5:high PASS 100% 150 in → 525 out (504 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.5:high__json_schema PASS 100% 150 in → 352 out (320 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.5:low PASS 100% 150 in → 630 out (576 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.5:low__json_schema PASS 100% 150 in → 675 out (656 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.5:medium__json_schema PASS 100% 158 in → 494 out (419 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
minimax/minimax-m2.5:xhigh PASS 100% 158 in → 1,746 out (1,721 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 150 in → 951 out (927 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.7:high PASS 100% 152 in → 717 out (693 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
minimax/minimax-m2.7:high__json_schema PASS 100% 150 in → 29 out (29 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.7:low PASS 100% 150 in → 326 out (314 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
minimax/minimax-m2.7:low__json_schema PASS 100% 283 in → 713 out (686 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
minimax/minimax-m2.7:medium PASS 100% 150 in → 519 out (497 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.7:medium__json_schema PASS 100% 283 in → 707 out (680 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
minimax/minimax-m2.7:xhigh PASS 100% 150 in → 568 out (564 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 283 in → 778 out (742 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n" }
minimax/minimax-m3:high__json_schema PASS 100% 275 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
minimax/minimax-m3:low PASS 100% 288 in → 222 out (180 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m3:low__json_schema PASS 100% 275 in → 29 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m3:medium PASS 100% 288 in → 393 out (320 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m3:medium__json_schema PASS 100% 275 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
minimax/minimax-m3:xhigh__json_schema PASS 100% 275 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
mistralai/devstral-2512 PASS 100% 130 in → 27 out
```yaml server: port: 443 host: 0.0.0.0 ```
mistralai/devstral-2512__json_schema PASS 100% 130 in → 30 out
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:high PASS 100% 142 in → 624 out (495 reasoning)
server: port: 443 host: 0.0.0.0
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 130 in → 450 out (334 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:low PASS 100% 142 in → 518 out (433 reasoning)
server: port: 443 host: 0.0.0.0
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 130 in → 695 out (597 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:medium PASS 100% 142 in → 415 out (355 reasoning)
server: port: 443 host: 0.0.0.0
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 130 in → 979 out (777 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:none PASS 100% 142 in → 27 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 130 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 142 in → 447 out (345 reasoning)
server: port: 443 host: 0.0.0.0
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 130 in → 602 out (447 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
moonshotai/kimi-k2.6:high PASS 100% 119 in → 378 out (354 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 118 in → 827 out (649 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 118 in → 738 out (591 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 203 in → 226 out (197 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
moonshotai/kimi-k2.7-code:low PASS 100% 118 in → 315 out (289 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 203 in → 175 out (146 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
moonshotai/kimi-k2.7-code:medium PASS 100% 119 in → 382 out (356 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 118 in → 877 out (232 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 119 in → 556 out (525 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
moonshotai/kimi-k3:high PASS 100% 196 in → 326 out (287 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:high__json_schema PASS 100% 196 in → 596 out (557 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:low PASS 100% 196 in → 620 out (581 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:low__json_schema PASS 100% 196 in → 499 out (460 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:max PASS 100% 196 in → 260 out (226 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k3:max__json_schema PASS 100% 196 in → 774 out (735 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:medium PASS 100% 196 in → 412 out (378 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k3:medium__json_schema PASS 100% 196 in → 581 out (542 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:xhigh PASS 100% 196 in → 212 out (173 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 196 in → 585 out (546 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 143 in → 1,420 out (1,242 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 143 in → 1,243 out (1,023 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 143 in → 682 out (593 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 143 in → 813 out (685 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 143 in → 712 out (597 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 143 in → 349 out (257 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 143 in → 241 out (197 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 143 in → 319 out (262 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 143 in → 234 out (192 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 143 in → 239 out (192 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 143 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 143 in → 509 out (448 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 143 in → 234 out (180 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-5.4-mini:high PASS 100% 115 in → 182 out (157 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:high__json_schema PASS 100% 151 in → 361 out (326 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4-mini:low PASS 100% 115 in → 102 out (77 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:low__json_schema PASS 100% 151 in → 123 out (87 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-mini:medium PASS 100% 115 in → 113 out (88 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:medium__json_schema PASS 100% 151 in → 132 out (97 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4-mini:none PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:none__json_schema PASS 100% 151 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-mini:xhigh PASS 100% 115 in → 541 out (516 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 151 in → 551 out (516 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4-nano:high PASS 100% 115 in → 230 out (205 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-nano:high__json_schema PASS 100% 151 in → 230 out (194 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-nano:low PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-nano:low__json_schema PASS 100% 151 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-nano:medium PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-nano:medium__json_schema PASS 100% 151 in → 83 out (47 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-nano:none__json_schema PASS 100% 151 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 115 in → 742 out (717 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 151 in → 460 out (424 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4:high PASS 100% 115 in → 109 out (84 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:high__json_schema PASS 100% 151 in → 113 out (78 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4:low PASS 100% 115 in → 56 out (31 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:low__json_schema PASS 100% 151 in → 62 out (26 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4:medium PASS 100% 115 in → 60 out (35 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:medium__json_schema PASS 100% 151 in → 108 out (72 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4:none PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:none__json_schema PASS 100% 151 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4:xhigh PASS 100% 115 in → 172 out (147 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:xhigh__json_schema PASS 100% 151 in → 128 out (93 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.5:high PASS 100% 115 in → 132 out (107 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:high__json_schema PASS 100% 151 in → 329 out (294 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.5:low PASS 100% 115 in → 73 out (48 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:low__json_schema PASS 100% 151 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.5:medium PASS 100% 115 in → 104 out (79 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:medium__json_schema PASS 100% 151 in → 111 out (75 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.5:none PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:none__json_schema PASS 100% 151 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.5:xhigh PASS 100% 115 in → 159 out (134 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:xhigh__json_schema PASS 100% 151 in → 216 out (180 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:high PASS 100% 115 in → 118 out (93 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:high__json_schema PASS 100% 151 in → 129 out (93 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:low PASS 100% 115 in → 119 out (94 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:low__json_schema PASS 100% 151 in → 117 out (81 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:max PASS 100% 115 in → 214 out (189 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:max__json_schema PASS 100% 151 in → 229 out (194 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-luna:medium PASS 100% 115 in → 106 out (81 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 151 in → 129 out (93 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:xhigh PASS 100% 115 in → 117 out (92 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 151 in → 132 out (97 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:high PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:high__json_schema PASS 100% 151 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:low PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:low__json_schema PASS 100% 151 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:max PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:max__json_schema PASS 100% 151 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:medium PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:medium__json_schema PASS 100% 151 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-sol:none PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:none__json_schema PASS 100% 151 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-sol:xhigh PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 151 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:high PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:high__json_schema PASS 100% 151 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:low PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:low__json_schema PASS 100% 151 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:max PASS 100% 115 in → 88 out (63 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:max__json_schema PASS 100% 151 in → 283 out (248 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:medium PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 151 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:none PASS 100% 115 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:none__json_schema PASS 100% 151 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:xhigh PASS 100% 115 in → 60 out (35 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 151 in → 224 out (189 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-oss-120b:high PASS 100% 176 in → 733 out (677 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-120b:high__json_schema PASS 100% 178 in → 677 out (675 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
openai/gpt-oss-120b:low PASS 100% 174 in → 136 out (94 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
openai/gpt-oss-120b:low__json_schema PASS 100% 176 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
openai/gpt-oss-120b:medium PASS 100% 176 in → 230 out (193 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-120b:medium__json_schema PASS 100% 174 in → 215 out (173 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-120b:xhigh PASS 100% 174 in → 862 out (881 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 174 in → 771 out (713 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-20b:high PASS 100% 181 in → 1,013 out (992 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:high__json_schema PASS 100% 163 in → 688 out (663 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
openai/gpt-oss-20b:low PASS 100% 163 in → 90 out (54 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:medium PASS 100% 176 in → 380 out (336 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:medium__json_schema PASS 100% 278 in → 621 out (583 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-oss-20b:xhigh PASS 100% 163 in → 893 out (872 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 163 in → 582 out (536 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-m.1:high PASS 100% 137 in → 643 out (618 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:high__json_schema PASS 100% 137 in → 433 out (399 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-m.1:low PASS 100% 137 in → 493 out (468 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:low__json_schema PASS 100% 137 in → 1,083 out (1,049 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-m.1:medium PASS 100% 137 in → 763 out (738 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:medium__json_schema PASS 100% 137 in → 1,187 out (1,152 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-m.1:none PASS 100% 137 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
poolside/laguna-m.1:none__json_schema PASS 100% 137 in → 35 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-m.1:xhigh PASS 100% 137 in → 757 out (732 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:xhigh__json_schema PASS 100% 137 in → 827 out (792 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-xs-2.1:high PASS 100% 137 in → 1,467 out (1,440 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:high__json_schema PASS 100% 137 in → 1,299 out (1,264 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-xs-2.1:low PASS 100% 137 in → 610 out (584 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:low__json_schema PASS 100% 137 in → 918 out (889 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-xs-2.1:medium PASS 100% 137 in → 723 out (700 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 137 in → 836 out (797 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-xs-2.1:none PASS 100% 137 in → 22 out (0 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:none__json_schema PASS 100% 137 in → 29 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-xs-2.1:xhigh PASS 100% 137 in → 1,665 out (1,639 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 137 in → 2,460 out (2,427 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-xs.2:high PASS 100% 174 in → 344 out (314 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
poolside/laguna-xs.2:high__json_schema PASS 100% 174 in → 252 out (220 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n\n"}
poolside/laguna-xs.2:low PASS 100% 174 in → 426 out (401 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-xs.2:low__json_schema PASS 100% 174 in → 603 out (573 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-xs.2:medium PASS 100% 174 in → 298 out (268 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
poolside/laguna-xs.2:medium__json_schema PASS 100% 174 in → 501 out (471 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-xs.2:none PASS 100% 174 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-xs.2:none__json_schema PASS 100% 174 in → 29 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 174 in → 513 out (478 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-27b:high__json_schema PASS 100% 132 in → 2,000 out (1,702 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-27b:low PASS 100% 132 in → 1,558 out (1,317 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:medium PASS 100% 132 in → 2,061 out (1,820 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:medium__json_schema PASS 100% 132 in → 1,277 out (1 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-27b:none PASS 100% 134 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:none__json_schema PASS 100% 134 in → 37 out (0 reasoning)
# config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "# config.yaml\nserver:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-27b:xhigh PASS 100% 132 in → 1,604 out (1,405 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 132 in → 1,433 out (1,193 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:high PASS 100% 132 in → 1,874 out (1,656 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 132 in → 1,324 out (1,100 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:low PASS 100% 132 in → 1,330 out (1,130 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 132 in → 1,098 out (906 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 132 in → 1,330 out (1,147 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 132 in → 1,542 out (1,369 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 134 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 132 in → 1,442 out (1,212 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 132 in → 1,454 out (1,230 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-flash:high PASS 100% 132 in → 1,441 out (1,414 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-flash:low PASS 100% 132 in → 468 out (441 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-flash:medium PASS 100% 132 in → 1,362 out (1,335 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-flash:none PASS 100% 134 in → 27 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.6-flash:xhigh PASS 100% 132 in → 1,496 out (1,469 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:high PASS 100% 132 in → 1,768 out (1,741 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:low PASS 100% 132 in → 973 out (946 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:max PASS 100% 132 in → 1,566 out (1,539 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:medium PASS 100% 132 in → 1,117 out (1,090 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:none PASS 100% 134 in → 27 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-flash:xhigh PASS 100% 132 in → 1,441 out (1,414 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-max:high PASS 100% 132 in → 359 out (327 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-max:low PASS 100% 132 in → 841 out (814 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-max:medium PASS 100% 132 in → 331 out (299 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-max:none PASS 100% 134 in → 26 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.7-plus:high PASS 100% 132 in → 924 out (897 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-plus:low PASS 100% 132 in → 201 out (169 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-plus:medium PASS 100% 132 in → 196 out (164 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-plus:none PASS 100% 134 in → 22 out (0 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-plus:xhigh PASS 100% 132 in → 197 out (170 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:high PASS 100% 125 in → 2,040 out (1,940 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:high__json_schema PASS 100% 125 in → 3,391 out (3,317 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
tencent/hy3:low PASS 100% 125 in → 1,130 out (1,108 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:medium PASS 100% 125 in → 712 out (690 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:medium__json_schema PASS 100% 125 in → 1,336 out (1,144 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
tencent/hy3:none PASS 100% 128 in → 20 out (0 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:none__json_schema PASS 100% 128 in → 29 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
tencent/hy3:xhigh PASS 100% 125 in → 1,421 out (1,399 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:xhigh__json_schema PASS 100% 125 in → 1,365 out (1,278 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling-small:high PASS 100% 124 in → 291 out (262 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:high__json_schema PASS 100% 124 in → 670 out (645 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:low PASS 100% 124 in → 58 out (27 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:low__json_schema PASS 100% 124 in → 61 out (28 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
thinkingmachines/inkling-small:max PASS 100% 124 in → 385 out (344 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:max__json_schema PASS 100% 124 in → 623 out (589 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
thinkingmachines/inkling-small:medium PASS 100% 124 in → 149 out (115 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:medium__json_schema PASS 100% 124 in → 292 out (255 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
thinkingmachines/inkling-small:xhigh PASS 100% 124 in → 1,165 out (1,155 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 124 in → 348 out (313 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling:low PASS 100% 124 in → 49 out (24 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:max PASS 100% 124 in → 1,049 out (1,024 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:max__json_schema PASS 100% 124 in → 829 out (795 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
thinkingmachines/inkling:medium__json_schema PASS 100% 124 in → 417 out (383 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
thinkingmachines/inkling:none PASS 100% 122 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:xhigh PASS 100% 124 in → 696 out (666 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:xhigh__json_schema PASS 100% 124 in → 426 out (391 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.3:high PASS 100% 294 in → 1,781 out (1,762 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:high__json_schema PASS 100% 351 in → 1,680 out (1,655 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
x-ai/grok-4.3:low PASS 100% 300 in → 448 out (429 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:low__json_schema PASS 100% 357 in → 556 out (526 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.3:max PASS 100% 294 in → 1,790 out (1,771 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:max__json_schema PASS 100% 351 in → 1,710 out (1,681 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
x-ai/grok-4.3:medium PASS 100% 300 in → 859 out (840 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:medium__json_schema PASS 100% 357 in → 938 out (908 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.3:none PASS 100% 292 in → 24 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.3:none__json_schema PASS 100% 349 in → 30 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
x-ai/grok-4.3:xhigh PASS 100% 294 in → 3,661 out (3,642 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:xhigh__json_schema PASS 100% 351 in → 1,596 out (1,567 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
x-ai/grok-4.5:high PASS 100% 316 in → 265 out (241 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:high__json_schema PASS 100% 385 in → 430 out (397 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:low PASS 100% 316 in → 213 out (189 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:low__json_schema PASS 100% 385 in → 266 out (233 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:max PASS 100% 316 in → 329 out (305 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:max__json_schema PASS 100% 385 in → 539 out (506 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:medium PASS 100% 316 in → 294 out (270 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:medium__json_schema PASS 100% 385 in → 266 out (233 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:xhigh PASS 100% 316 in → 353 out (334 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.5:xhigh__json_schema PASS 100% 385 in → 604 out (571 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
z-ai/glm-4.7-flash:high PASS 100% 119 in → 3,304 out (3,035 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-4.7-flash:high__json_schema PASS 100% 119 in → 960 out (851 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-4.7-flash:low PASS 100% 119 in → 3,861 out (3,593 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
z-ai/glm-4.7-flash:medium PASS 100% 119 in → 1,125 out (983 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 119 in → 2,222 out (2,049 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-4.7-flash:xhigh PASS 100% 119 in → 5,092 out (4,672 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
z-ai/glm-5.2:high PASS 100% 126 in → 547 out (465 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:high__json_schema PASS 100% 302 in → 183 out (154 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
z-ai/glm-5.2:low PASS 100% 126 in → 495 out (472 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:low__json_schema PASS 100% 126 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-5.2:medium PASS 100% 126 in → 228 out (206 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:none PASS 100% 120 in → 20 out (0 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:none__json_schema PASS 100% 296 in → 29 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
z-ai/glm-5.2:xhigh PASS 100% 126 in → 1,014 out (0 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:xhigh__json_schema PASS 100% 126 in → 560 out (449 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
anthropic/claude-fable-5:high FAIL 0% 180 in → 5 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['config.yaml']
anthropic/claude-fable-5:low FAIL 0% 180 in → 1 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['config.yaml']
anthropic/claude-fable-5:max FAIL 0% 180 in → 10 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
anthropic/claude-fable-5:medium FAIL 0% 180 in → 3 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['config.yaml']
anthropic/claude-fable-5:xhigh FAIL 0% 180 in → 3 out (0 reasoning)
(empty output)
Failure: Content mismatch: ['config.yaml']
anthropic/claude-opus-4.8:high FAIL 0% 180 in → 334 out (82 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Reasoning:** The two branches changed different fields relative to the base (port 8080, host localhost): - Branch A: changed `port` → `443`, left `host` as `localhost`. - Branch B: left `port` as `8080`, changed `host` → `0.0.0.0`. Since each branch modified a distinct setting, the correct merge preserves both intended changes: the new HTTPS port (`443`) from A and the new bind address (`0.0.0.0`) from B.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-opus-4.8:medium FAIL 0% 180 in → 264 out (69 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Reasoning:** Each branch modified a different field, so a correct semantic merge should preserve both intentional changes: - Branch A's change: `port: 443` (HTTPS) - Branch B's change: `host: 0.0.0.0` (bind to all interfaces) This avoids discarding either team's work.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-opus-4.8:none FAIL 0% 180 in → 284 out (0 reasoning)
Looking at the two branches: - **Branch A** changed port to 443 (HTTPS) and kept host as localhost - **Branch B** kept port 8080 but changed host to 0.0.0.0 Each branch changed a *different* setting from the base, so both intentional changes should be preserved: - Port: 443 (Branch A's HTTPS change) - Host: 0.0.0.0 (Branch B's change) ```yaml --- config.yaml server: port: 443 host: 0.0.0.0 ``` **Note:** This combines both non-conflicting intents (HTTPS port + binding to all interfaces). However, please verify this is the desired outcome, as binding an HTTPS service to `0.0.0.0` exposes it publicly — confirm that's intended for your deployment and that TLS/certificates are configured for port 443.
Failure: Expected file heading before file content
anthropic/claude-sonnet-4.6:high FAIL 0% 133 in → 562 out (297 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Resolution logic (3-way merge):** | Field | Base | HEAD (Branch B) | prod (Branch A) | Resolution | |-------|------|-----------------|-----------------|------------| | `port` | 8080 | 8080 (unchanged) | **443** (changed) | `443` — take Branch A's change | | `host` | localhost | **0.0.0.0** (changed) | localhost (unchanged) | `0.0.0.0` — take Branch B's change | Each branch modified a **different field**, so both changes are applied independently. There is no true conflict — this is a clean non-overlapping merge.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-sonnet-4.6:max FAIL 0% 133 in → 649 out (189 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Resolution reasoning:** | Setting | Base | HEAD | prod | Resolution | |---------|------|------|------|------------| | `port` | 8080 | 8080 (unchanged) | **443** | **443** — prod made an intentional change to HTTPS; HEAD didn't touch it | | `host` | localhost | **0.0.0.0** | localhost (unchanged) | **0.0.0.0** — HEAD made an intentional change; prod didn't touch it | Each branch modified a **different setting**, so both changes are independent and non-conflicting. The correct resolution is to accept both: the HTTPS port from `prod` and the bind-all host from `HEAD`. This is a classic case where accepting only one side would silently discard a real, intentional change from the other branch.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 133 in → 423 out (123 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Resolution logic:** | Field | Base | HEAD (Branch B) | prod (Branch A) | Resolution | |-------|------|-----------------|-----------------|------------| | `port` | 8080 | 8080 (unchanged) | **443** (changed) | `443` | | `host` | localhost | **0.0.0.0** (changed) | localhost (unchanged) | `0.0.0.0` | Each branch made a **distinct, non-overlapping change** to a different field. Both changes are accepted — `port: 443` from the `prod` branch and `host: 0.0.0.0` from `HEAD`. This is a clean three-way merge with no true conflict.
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:medium FAIL 0% 120 in → 435 out (440 reasoning)
443 0.0.0.0
Failure: Content mismatch: ['config.yaml']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 120 in → 1,766 out (1,839 reasoning)
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: localhost" }
Failure: Content mismatch: ['config.yaml']
arcee-ai/trinity-mini:xhigh FAIL 0% 120 in → 494 out (460 reasoning)
server: port: 443host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 120 in → 893 out (973 reasoning)
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: localhost\n" }
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 127 in → 902 out (871 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server: port: 443 host: 0.0.0.0" }
Failure: Content mismatch: ['config.yaml']
google/gemini-3.5-flash:high FAIL 0% 126 in → 991 out (970 reasoning)
server port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
google/gemma-4-31b-it:medium FAIL 0% 142 in → 278 out (190 reasoning)
server: port: 443 hostBCBShost: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
ibm-granite/granite-4.1-8b FAIL 0% 124 in → 13 out
server: port: 443 host: localhost
Failure: Content mismatch: ['config.yaml']
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: localhost" }
Failure: Content mismatch: ['config.yaml']
inclusionai/ling-2.6-flash FAIL 0% 146 in → 30 out
```yaml --- server: port: 443 host: 0.0.0.0 ```
Failure: Content mismatch: ['config.yaml']
liquid/lfm-2-24b-a2b FAIL 0% 136 in → 14 out
server: port: 443 host: localhost
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: ```yaml server: port: 443 host: localhost ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:medium FAIL 0% 155 in → 593 out (533 reasoning)
yaml server: port: 443 host: 0.0.0.0 ```
Failure: Content mismatch: ['config.yaml']
minimax/minimax-m3:high FAIL 0% 288 in → 426 out (338 reasoning)
```yaml --- config.yaml server: port: 443 host: 0.0.0.0 ```
Failure: Expected file heading before file content
minimax/minimax-m3:xhigh FAIL 0% 288 in → 532 out (449 reasoning)
```yaml --- server: port: 443 host: 0.0.0.0 ```
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:low FAIL 0% 119 in → 965 out (879 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
moonshotai/kimi-k2.6:low__json_schema FAIL 0% 119 in → 361 out (354 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"resolved_content":""}
Failure: Content mismatch: ['config.yaml']
moonshotai/kimi-k2.6:medium FAIL 0% 119 in → 567 out (546 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
moonshotai/kimi-k2.6:xhigh FAIL 0% 119 in → 957 out (931 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
moonshotai/kimi-k2.7-code:high FAIL 0% 119 in → 393 out (372 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
moonshotai/kimi-k2.7-code:xhigh FAIL 0% 119 in → 245 out (188 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-nano-30b-a3b:low FAIL 0% 143 in → 1,205 out (1,068 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 143 in → 19 out (0 reasoning)
--- config.yaml server: port: 443 host: localhost
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 143 in → 28 out (0 reasoning)
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: localhost" }
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 0% 143 in → 894 out (793 reasoning)
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: localhost" }
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-super-120b-a12b:high__json_schema FAIL 0% 143 in → 682 out (596 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "---\nconfig.yaml\nserver:\n port: 443\n host: 0.0.0.0\n" }
Failure: Expected file heading before file content
nvidia/nemotron-3-super-120b-a12b:none FAIL 0% 143 in → 23 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.4-nano:none FAIL 0% 115 in → 16 out (0 reasoning)
server: port: 443 host: localhost
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.6-luna:none FAIL 0% 115 in → 16 out (0 reasoning)
server: port: 443 host: localhost
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.6-luna:none__json_schema FAIL 0% 151 in → 27 out (0 reasoning)
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: localhost\n"}
Failure: Content mismatch: ['config.yaml']
openai/gpt-oss-20b:low__json_schema FAIL 0% 176 in → 105 out (71 reasoning)
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: localhost\n" }
Failure: Content mismatch: ['config.yaml']
poolside/laguna-xs.2:xhigh FAIL 0% 174 in → 303 out (269 reasoning)
```yaml --- config.yaml server: port: 443 host: 0.0.0.0 ```
Failure: Expected file heading before file content
qwen/qwen3.6-27b:high FAIL 0% 132 in → 2,187 out (1,845 reasoning)
--- server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-35b-a3b:none FAIL 0% 134 in → 15 out (0 reasoning)
server: port: 443 host: localhost
Failure: Content mismatch: ['config.yaml']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-05b9abf2-44ff-9e69-9378-f41461bf19da","request_id":"05b9abf2-44ff-9e69-9378-f41461bf19da"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-29e9de1f-dbef-9901-bb29-d8383d47ae2d","request_id":"29e9de1f-dbef-9901-bb29-d8383d47ae2d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-90fc40ce-3a34-9911-bbf5-cee0d7273754","request_id":"90fc40ce-3a34-9911-bbf5-cee0d7273754"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dfc3793e-da6c-9dc3-bd8a-78c9823715cb","request_id":"dfc3793e-da6c-9dc3-bd8a-78c9823715cb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-52018b9c-6c4c-9aa2-b0fa-ad4a34998c23","request_id":"52018b9c-6c4c-9aa2-b0fa-ad4a34998c23"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2092ac34-1c47-9616-ba33-005a7c5f9970"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-25ceb8c0-fccd-9ff0-829c-d054e43ef4e5"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d68c9aeb-0c47-9926-bf5e-0273b001e1d4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-81fa0214-c207-948e-9708-e9d4291382ef"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-bd04d7cd-2f4b-9089-9eff-9f34347583eb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-37d14c9c-c956-946d-9f6e-b475e6959fec"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d6d924cf-3b08-9d56-9350-d956c7fea23d","request_id":"d6d924cf-3b08-9d56-9350-d956c7fea23d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-96ec92a7-4274-92d1-a100-c46f11f11d55","request_id":"96ec92a7-4274-92d1-a100-c46f11f11d55"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e53815fe-ce08-9106-8804-cba9fa335915","request_id":"e53815fe-ce08-9106-8804-cba9fa335915"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e2cdc388-b042-9aee-a538-2c7589880577","request_id":"e2cdc388-b042-9aee-a538-2c7589880577"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh FAIL 0% 132 in → 297 out (261 reasoning)
```yaml --- config.yaml server: port: 443 host: 0.0.0.0 ```
Failure: Expected file heading before file content
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bd262afe-6c87-96ef-8633-67ff667e2ddf","request_id":"bd262afe-6c87-96ef-8633-67ff667e2ddf"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-12dff2f6-716b-926b-b512-249221c7247d","request_id":"12dff2f6-716b-926b-b512-249221c7247d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6fe31a0b-2a60-988c-b367-b88d7bfaac51","request_id":"6fe31a0b-2a60-988c-b367-b88d7bfaac51"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c404a9ca-f056-9a8c-a783-415a759b3e42","request_id":"c404a9ca-f056-9a8c-a783-415a759b3e42"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-603346da-caba-979f-b339-8476c218e02a","request_id":"603346da-caba-979f-b339-8476c218e02a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bd36ee28-1e31-9564-a6b6-cc3605ce1ebc","request_id":"bd36ee28-1e31-9564-a6b6-cc3605ce1ebc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:low__json_schema FAIL 0% 125 in → 1,158 out (976 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "resolved_content": "" }
Failure: Content mismatch: ['config.yaml']
thinkingmachines/inkling-small:none FAIL 0% 124 in → 26 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0<|end_message|>
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: --- config.yaml server: port: 443 host: 0.0.0.0{"resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high FAIL 0% 124 in → 710 out (683 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
Failure: Expected file heading before file content
Invalid JSON. Output: server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium FAIL 0% 124 in → 856 out (827 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
Failure: Expected file heading before file content
Invalid JSON. Output: server: port: 443 host: 0.0.0.0{ "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 119 in → 21 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 119 in → 33 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 8080\n host: 0.0.0.0\n" }
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)