Multi-line hunk: config block conflict
Tests ability to resolve a config block conflict with port and host changes. Evaluates infrastructure configuration merging.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'server: port: 8080 host: localhost' > config.yaml
  5. 05 git add config.yaml
  6. 06 git commit -m 'Initial config'
  7. 07 git checkout -b prod
  8. 08 printf 'server: port: 443 host: localhost' > config.yaml
  9. 09 git add config.yaml
  10. 10 git commit -m 'Use HTTPS port'
  11. 11 git checkout main
  12. 12 printf 'server: port: 8080 host: 0.0.0.0' > config.yaml
  13. 13 git add config.yaml
  14. 14 git commit -m 'Listen on all interfaces'
  15. 15 git cherry-pick $(git rev-parse prod)
Prompt
Resolve the cherry-pick conflict in config.yaml. The base had port: 8080 and host: localhost. The incoming commit changed port to 443 (HTTPS) keeping localhost. The current branch kept port 8080 but changed host to 0.0.0.0. Provide ONLY the resolved file content.
Expected
server: port: 443 host: 0.0.0.0

Scoped model quality, cost, API time, and token usage for cherry_pick/f006.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 200 in → 53 out (42 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-fable-5:low PASS 100% 199 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-fable-5:low__json_schema PASS 100% 420 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-fable-5:max PASS 100% 202 in → 412 out (76 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-fable-5:max__json_schema PASS 100% 420 in → 436 out (162 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-fable-5:medium__json_schema PASS 100% 418 in → 57 out (20 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-fable-5:xhigh PASS 100% 199 in → 98 out (53 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 419 in → 73 out (31 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:high PASS 100% 174 in → 422 out (348 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 344 in → 397 out (311 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:low PASS 100% 178 in → 323 out (262 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 346 in → 525 out (452 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:medium PASS 100% 176 in → 398 out (361 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 343 in → 336 out (266 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:none PASS 100% 147 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 313 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 176 in → 338 out (288 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 344 in → 404 out (342 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:high PASS 100% 149 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 317 in → 79 out (29 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:low PASS 100% 148 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 316 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 315 in → 201 out (133 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:medium PASS 100% 148 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 317 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:none PASS 100% 148 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 316 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 148 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 316 in → 80 out (28 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:high PASS 100% 207 in → 80 out (27 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.7:high__json_schema PASS 100% 425 in → 39 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:low PASS 100% 204 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
anthropic/claude-opus-4.7:low__json_schema PASS 100% 425 in → 39 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:max PASS 100% 205 in → 404 out (125 reasoning)
server: port: 443 host: 0.0.0.0
anthropic/claude-opus-4.7:max__json_schema PASS 100% 424 in → 262 out (103 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:medium PASS 100% 203 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 425 in → 39 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:none__json_schema PASS 100% 424 in → 39 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 205 in → 89 out (38 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 423 in → 39 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 419 in → 130 out (68 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:low PASS 100% 202 in → 85 out (48 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 420 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:max__json_schema PASS 100% 420 in → 282 out (84 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:medium PASS 100% 200 in → 185 out (85 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 420 in → 127 out (76 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 421 in → 36 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 202 in → 339 out (128 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 422 in → 104 out (52 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 316 in → 149 out (79 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 149 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 317 in → 38 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 316 in → 241 out (158 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 314 in → 85 out (32 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 146 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 315 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 314 in → 186 out (109 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:high PASS 100% 201 in → 68 out (41 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 420 in → 86 out (41 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:low PASS 100% 199 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 420 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:medium PASS 100% 200 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 421 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:none PASS 100% 201 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 419 in → 46 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 199 in → 98 out (56 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 419 in → 108 out (38 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 129 in → 1,875 out (1,859 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-large-thinking:low PASS 100% 129 in → 1,381 out (1,365 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
arcee-ai/trinity-large-thinking:medium PASS 100% 133 in → 1,445 out (1,433 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
arcee-ai/trinity-large-thinking:xhigh PASS 100% 129 in → 1,605 out (1,592 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
arcee-ai/trinity-mini:high PASS 100% 132 in → 341 out (341 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-mini:medium PASS 100% 130 in → 752 out (769 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-mini:medium__json_schema PASS 100% 130 in → 438 out (422 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
arcee-ai/trinity-mini:xhigh PASS 100% 130 in → 457 out (462 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 129 in → 398 out (389 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 135 in → 414 out (412 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 136 in → 219 out (176 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 213 in → 139 out (119 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 134 in → 445 out (426 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 215 in → 255 out (210 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 417 in → 1,211 out (1,181 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 135 in → 480 out (459 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 135 in → 328 out (288 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 136 in → 20 out (0 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 134 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 137 in → 345 out (302 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 135 in → 576 out (512 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash:high PASS 100% 134 in → 130 out (103 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 136 in → 427 out (395 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash:low PASS 100% 136 in → 549 out (522 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 135 in → 403 out (358 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash:medium PASS 100% 133 in → 394 out (394 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 232 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash:none PASS 100% 137 in → 26 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 136 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 215 in → 686 out (665 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 216 in → 553 out (521 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-pro:high PASS 100% 135 in → 768 out (747 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 326 in → 227 out (197 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
deepseek/deepseek-v4-pro:low PASS 100% 136 in → 299 out (263 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:medium PASS 100% 135 in → 280 out (259 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 326 in → 380 out (352 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
deepseek/deepseek-v4-pro:none PASS 100% 137 in → 25 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 135 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 215 in → 714 out (693 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 404 in → 2,657 out (2,624 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
google/gemini-3-flash-preview:high PASS 100% 139 in → 345 out (323 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:high__json_schema PASS 100% 201 in → 1,198 out (1,169 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3-flash-preview:low PASS 100% 140 in → 285 out (263 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:low__json_schema PASS 100% 202 in → 365 out (331 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3-flash-preview:medium PASS 100% 141 in → 203 out (181 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:medium__json_schema PASS 100% 141 in → 503 out (474 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3-flash-preview:xhigh PASS 100% 141 in → 303 out (281 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 202 in → 1,007 out (978 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 142 in → 413 out (391 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 199 in → 1,618 out (1,588 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 142 in → 170 out (148 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 142 in → 159 out (124 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 142 in → 319 out (297 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 140 in → 619 out (586 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 142 in → 418 out (396 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 140 in → 1,609 out (1,580 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-pro-preview:high PASS 100% 139 in → 677 out (655 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 200 in → 1,536 out (1,502 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-pro-preview:low PASS 100% 140 in → 328 out (306 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 202 in → 552 out (522 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-pro-preview:medium PASS 100% 142 in → 320 out (298 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 201 in → 160 out (123 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 141 in → 586 out (564 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 201 in → 1,331 out (1,296 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash-lite:high PASS 100% 143 in → 327 out (305 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 142 in → 1,361 out (1,332 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash-lite:low PASS 100% 142 in → 22 out (0 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 201 in → 35 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.5-flash-lite:max PASS 100% 141 in → 571 out (549 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 201 in → 2,588 out (2,559 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash-lite:medium PASS 100% 142 in → 298 out (276 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 140 in → 591 out (556 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 141 in → 313 out (291 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 143 in → 1,225 out (1,195 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash:high PASS 100% 142 in → 722 out (700 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash:high__json_schema PASS 100% 200 in → 1,101 out (1,072 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash:low PASS 100% 142 in → 263 out (241 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash:low__json_schema PASS 100% 199 in → 286 out (251 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.5-flash:medium PASS 100% 141 in → 515 out (493 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash:medium__json_schema PASS 100% 201 in → 636 out (605 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.5-flash:xhigh PASS 100% 138 in → 587 out (565 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 200 in → 958 out (928 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.6-flash:high PASS 100% 141 in → 518 out (496 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:high__json_schema PASS 100% 200 in → 1,117 out (1,086 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.6-flash:low PASS 100% 142 in → 203 out (181 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:low__json_schema PASS 100% 142 in → 388 out (359 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.6-flash:max PASS 100% 143 in → 1,132 out (1,110 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:max__json_schema PASS 100% 141 in → 659 out (628 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.6-flash:medium PASS 100% 142 in → 534 out (512 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:medium__json_schema PASS 100% 143 in → 704 out (673 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.6-flash:xhigh PASS 100% 141 in → 475 out (453 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 142 in → 1,109 out (1,078 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemma-4-26b-a4b-it:high PASS 100% 156 in → 370 out (249 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 157 in → 352 out (255 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:low PASS 100% 156 in → 404 out (307 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 157 in → 436 out (312 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:medium PASS 100% 156 in → 655 out (491 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 157 in → 426 out (272 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:none PASS 100% 154 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 154 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 157 in → 517 out (424 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 156 in → 560 out (396 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-31b-it:high PASS 100% 158 in → 224 out (132 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
google/gemma-4-31b-it:high__json_schema PASS 100% 158 in → 317 out (1 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-31b-it:low PASS 100% 156 in → 379 out (254 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:low__json_schema PASS 100% 157 in → 307 out (187 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-31b-it:medium PASS 100% 161 in → 259 out (236 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:medium__json_schema PASS 100% 157 in → 292 out (185 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-31b-it:none PASS 100% 154 in → 28 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:none__json_schema PASS 100% 152 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemma-4-31b-it:xhigh PASS 100% 158 in → 220 out (126 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 157 in → 256 out (151 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
inclusionai/ling-2.6-flash PASS 100% 160 in → 28 out
```yaml server: port: 443 host: 0.0.0.0 ```
inclusionai/ling-2.6-flash__json_schema PASS 100% 161 in → 35 out
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
minimax/minimax-m2.5:high PASS 100% 164 in → 415 out (391 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.5:high__json_schema PASS 100% 163 in → 277 out (231 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
minimax/minimax-m2.5:low PASS 100% 163 in → 1,250 out (1,223 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.5:low__json_schema PASS 100% 161 in → 826 out (797 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.5:medium PASS 100% 162 in → 893 out (877 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.5:medium__json_schema PASS 100% 162 in → 262 out (234 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.5:xhigh PASS 100% 171 in → 553 out (527 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 161 in → 257 out (224 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.7:high PASS 100% 164 in → 542 out (518 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
minimax/minimax-m2.7:high__json_schema PASS 100% 160 in → 416 out (373 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
minimax/minimax-m2.7:low PASS 100% 162 in → 675 out (649 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m2.7:low__json_schema PASS 100% 162 in → 30 out (30 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.7:medium PASS 100% 162 in → 343 out (325 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m2.7:medium__json_schema PASS 100% 296 in → 405 out (377 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
minimax/minimax-m2.7:xhigh PASS 100% 165 in → 239 out (214 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 295 in → 670 out (642 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
minimax/minimax-m3:high PASS 100% 300 in → 270 out (264 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m3:high__json_schema PASS 100% 286 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m3:low PASS 100% 300 in → 218 out (176 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m3:low__json_schema PASS 100% 286 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
minimax/minimax-m3:medium PASS 100% 299 in → 505 out (504 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m3:medium__json_schema PASS 100% 287 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m3:xhigh PASS 100% 300 in → 524 out (497 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m3:xhigh__json_schema PASS 100% 287 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
mistralai/devstral-2512 PASS 100% 145 in → 27 out
```yaml server: port: 443 host: 0.0.0.0 ```
mistralai/devstral-2512__json_schema PASS 100% 145 in → 30 out
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:high PASS 100% 157 in → 514 out (444 reasoning)
server: port: 443 host: 0.0.0.0
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 145 in → 331 out (277 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:low PASS 100% 155 in → 730 out (624 reasoning)
server: port: 443 host: 0.0.0.0
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 144 in → 2,246 out (2,175 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:medium PASS 100% 157 in → 465 out (377 reasoning)
server: port: 443 host: 0.0.0.0
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 144 in → 340 out (307 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:none PASS 100% 156 in → 27 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 144 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 156 in → 578 out (508 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.6:high PASS 100% 130 in → 576 out (555 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.6:high__json_schema PASS 100% 130 in → 1,336 out (1,243 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
moonshotai/kimi-k2.6:low__json_schema PASS 100% 312 in → 1,304 out (1,276 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 132 in → 527 out (499 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 129 in → 1,487 out (1,118 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 131 in → 244 out (214 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 133 in → 249 out (231 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
moonshotai/kimi-k2.7-code:medium PASS 100% 129 in → 425 out (404 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.7-code:xhigh PASS 100% 131 in → 271 out (250 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 218 in → 222 out (193 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
moonshotai/kimi-k3:low PASS 100% 207 in → 530 out (491 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:low__json_schema PASS 100% 206 in → 337 out (303 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k3:max PASS 100% 209 in → 280 out (246 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k3:max__json_schema PASS 100% 208 in → 341 out (302 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:medium PASS 100% 208 in → 367 out (328 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:medium__json_schema PASS 100% 305 in → 324 out (283 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 208 in → 487 out (448 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 157 in → 1,716 out (1,690 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 157 in → 243 out (199 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 156 in → 387 out (334 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 157 in → 294 out (242 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 158 in → 426 out (376 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 158 in → 256 out (205 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 155 in → 436 out (397 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 157 in → 193 out (144 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 157 in → 26 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 156 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 158 in → 251 out (211 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 156 in → 335 out (281 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
openai/gpt-5.4-mini:high PASS 100% 126 in → 163 out (138 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:high__json_schema PASS 100% 161 in → 199 out (164 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4-mini:low PASS 100% 126 in → 112 out (87 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:low__json_schema PASS 100% 163 in → 184 out (149 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4-mini:medium PASS 100% 129 in → 114 out (89 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:medium__json_schema PASS 100% 162 in → 185 out (150 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4-mini:none PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:none__json_schema PASS 100% 163 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-mini:xhigh PASS 100% 127 in → 274 out (249 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 164 in → 266 out (231 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4-nano:high PASS 100% 126 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-nano:high__json_schema PASS 100% 162 in → 328 out (292 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-nano:low__json_schema PASS 100% 163 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-nano:medium PASS 100% 128 in → 217 out (192 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-nano:medium__json_schema PASS 100% 162 in → 80 out (44 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-nano:none__json_schema PASS 100% 162 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 127 in → 1,109 out (1,084 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 164 in → 468 out (432 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4:high PASS 100% 125 in → 86 out (61 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:high__json_schema PASS 100% 164 in → 115 out (80 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4:low PASS 100% 127 in → 53 out (28 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:low__json_schema PASS 100% 163 in → 64 out (29 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4:medium PASS 100% 126 in → 56 out (31 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:medium__json_schema PASS 100% 163 in → 70 out (34 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4:none PASS 100% 126 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:none__json_schema PASS 100% 164 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4:xhigh PASS 100% 127 in → 114 out (89 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:xhigh__json_schema PASS 100% 162 in → 129 out (94 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.5:high PASS 100% 128 in → 139 out (114 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:high__json_schema PASS 100% 164 in → 254 out (219 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.5:low PASS 100% 128 in → 61 out (36 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:low__json_schema PASS 100% 164 in → 69 out (33 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.5:medium PASS 100% 129 in → 101 out (76 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:medium__json_schema PASS 100% 162 in → 309 out (274 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.5:none PASS 100% 129 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:none__json_schema PASS 100% 161 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.5:xhigh PASS 100% 128 in → 132 out (107 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:xhigh__json_schema PASS 100% 164 in → 552 out (516 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:high PASS 100% 127 in → 115 out (90 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:high__json_schema PASS 100% 163 in → 131 out (96 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-luna:low PASS 100% 128 in → 54 out (29 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:low__json_schema PASS 100% 164 in → 67 out (31 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:max PASS 100% 126 in → 150 out (125 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:max__json_schema PASS 100% 164 in → 209 out (174 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-luna:medium PASS 100% 127 in → 59 out (34 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 161 in → 70 out (34 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:none PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:none__json_schema PASS 100% 163 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-luna:xhigh PASS 100% 129 in → 138 out (113 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 162 in → 132 out (96 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-sol:high PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:high__json_schema PASS 100% 163 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:low PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:low__json_schema PASS 100% 163 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:max PASS 100% 125 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:max__json_schema PASS 100% 164 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:medium PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:medium__json_schema PASS 100% 164 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:none PASS 100% 126 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:none__json_schema PASS 100% 162 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:xhigh PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 164 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:high PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:high__json_schema PASS 100% 164 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:low PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:low__json_schema PASS 100% 165 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:max PASS 100% 128 in → 88 out (63 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:max__json_schema PASS 100% 162 in → 245 out (210 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:medium PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 163 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:none PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:none__json_schema PASS 100% 163 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:xhigh PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 163 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-oss-120b:high PASS 100% 174 in → 667 out (666 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-120b:high__json_schema PASS 100% 191 in → 434 out (411 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
openai/gpt-oss-120b:low PASS 100% 188 in → 85 out (44 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
openai/gpt-oss-120b:low__json_schema PASS 100% 173 in → 99 out (50 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-120b:medium PASS 100% 176 in → 264 out (230 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
openai/gpt-oss-120b:medium__json_schema PASS 100% 187 in → 183 out (133 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-120b:xhigh PASS 100% 186 in → 729 out (742 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 185 in → 394 out (371 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-20b:high PASS 100% 175 in → 860 out (807 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:high__json_schema PASS 100% 186 in → 1,148 out (1,154 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-20b:low PASS 100% 191 in → 114 out (73 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:low__json_schema PASS 100% 187 in → 152 out (116 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-oss-20b:medium PASS 100% 189 in → 440 out (421 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:medium__json_schema PASS 100% 239 in → 432 out (400 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-oss-20b:xhigh PASS 100% 174 in → 959 out (936 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 189 in → 714 out (682 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
poolside/laguna-m.1:high PASS 100% 151 in → 394 out (369 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:high__json_schema PASS 100% 152 in → 887 out (852 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-m.1:low PASS 100% 150 in → 1,232 out (1,208 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:low__json_schema PASS 100% 152 in → 538 out (504 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-m.1:medium PASS 100% 151 in → 739 out (714 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:medium__json_schema PASS 100% 152 in → 1,185 out (1,150 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-m.1:none PASS 100% 150 in → 28 out (0 reasoning)
``` server: port: 443 host: 0.0.0.0 ```
poolside/laguna-m.1:none__json_schema PASS 100% 149 in → 29 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-m.1:xhigh__json_schema PASS 100% 151 in → 539 out (504 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-xs-2.1:high PASS 100% 152 in → 518 out (491 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:high__json_schema PASS 100% 151 in → 1,677 out (1,642 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-xs-2.1:low PASS 100% 151 in → 1,207 out (1,185 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:low__json_schema PASS 100% 151 in → 1,250 out (1,214 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-xs-2.1:medium PASS 100% 149 in → 590 out (567 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 151 in → 544 out (509 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-xs-2.1:none__json_schema PASS 100% 152 in → 35 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-xs-2.1:xhigh PASS 100% 151 in → 485 out (458 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 152 in → 770 out (734 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-xs.2:high PASS 100% 187 in → 343 out (313 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
poolside/laguna-xs.2:high__json_schema PASS 100% 187 in → 321 out (284 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
poolside/laguna-xs.2:low PASS 100% 187 in → 261 out (236 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-xs.2:low__json_schema PASS 100% 189 in → 378 out (348 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-xs.2:medium PASS 100% 188 in → 320 out (290 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
poolside/laguna-xs.2:medium__json_schema PASS 100% 189 in → 330 out (295 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-xs.2:none PASS 100% 187 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-xs.2:none__json_schema PASS 100% 188 in → 29 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-xs.2:xhigh PASS 100% 188 in → 309 out (279 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 188 in → 289 out (259 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
qwen/qwen3.6-27b:high PASS 100% 147 in → 1,645 out (1,422 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:high__json_schema PASS 100% 147 in → 1,456 out (1,335 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-27b:low PASS 100% 147 in → 1,796 out (1,602 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:low__json_schema PASS 100% 148 in → 2,023 out (1,733 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-27b:medium PASS 100% 148 in → 1,215 out (1,091 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:none PASS 100% 150 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:none__json_schema PASS 100% 149 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-27b:xhigh PASS 100% 147 in → 1,727 out (1,493 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 145 in → 1,338 out (1 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:high PASS 100% 145 in → 973 out (845 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 148 in → 454 out (370 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:low PASS 100% 146 in → 985 out (899 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 147 in → 1,353 out (1,185 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 147 in → 1,242 out (1,089 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 147 in → 1,268 out (1,097 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
qwen/qwen3.6-35b-a3b:none PASS 100% 150 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 150 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 147 in → 1,173 out (1,146 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 147 in → 1,370 out (1,172 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-flash:high PASS 100% 148 in → 1,184 out (1,157 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-flash:low PASS 100% 145 in → 1,500 out (1,473 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-flash:medium PASS 100% 148 in → 1,271 out (1,244 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-flash:none PASS 100% 150 in → 22 out (0 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-flash:xhigh PASS 100% 148 in → 1,409 out (1,382 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:high PASS 100% 148 in → 1,597 out (1,565 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-flash:low PASS 100% 147 in → 1,635 out (1,608 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:max PASS 100% 146 in → 1,344 out (1,317 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:medium PASS 100% 147 in → 1,491 out (1,464 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:none PASS 100% 148 in → 27 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-flash:xhigh PASS 100% 148 in → 1,253 out (1,226 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-max:high PASS 100% 146 in → 209 out (182 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-max:low PASS 100% 147 in → 477 out (450 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-max:medium PASS 100% 147 in → 206 out (174 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-max:none PASS 100% 150 in → 26 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.7-max:xhigh PASS 100% 147 in → 220 out (188 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-plus:high PASS 100% 145 in → 598 out (571 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-plus:low PASS 100% 144 in → 205 out (173 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-plus:medium PASS 100% 146 in → 182 out (150 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-plus:none PASS 100% 150 in → 22 out (0 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-plus:xhigh PASS 100% 147 in → 182 out (150 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
tencent/hy3:high PASS 100% 136 in → 1,845 out (1,824 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:high__json_schema PASS 100% 137 in → 1,527 out (1,464 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
tencent/hy3:low PASS 100% 138 in → 499 out (477 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:low__json_schema PASS 100% 137 in → 894 out (759 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
tencent/hy3:medium PASS 100% 136 in → 1,152 out (1,015 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:medium__json_schema PASS 100% 137 in → 1,146 out (980 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
tencent/hy3:none PASS 100% 141 in → 20 out (0 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:none__json_schema PASS 100% 140 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
tencent/hy3:xhigh PASS 100% 138 in → 6,185 out (6,164 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:xhigh__json_schema PASS 100% 138 in → 1,964 out (1,833 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling-small:high PASS 100% 135 in → 306 out (290 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:high__json_schema PASS 100% 136 in → 158 out (119 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling-small:low PASS 100% 137 in → 54 out (25 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:low__json_schema PASS 100% 136 in → 63 out (28 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling-small:max PASS 100% 136 in → 134 out (105 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:max__json_schema PASS 100% 136 in → 163 out (128 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
thinkingmachines/inkling-small:medium PASS 100% 136 in → 207 out (170 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:medium__json_schema PASS 100% 137 in → 145 out (107 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling-small:none PASS 100% 133 in → 26 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:xhigh PASS 100% 136 in → 208 out (179 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:high PASS 100% 137 in → 1,009 out (983 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:low PASS 100% 138 in → 50 out (24 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:low__json_schema PASS 100% 137 in → 53 out (18 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling:max PASS 100% 135 in → 879 out (854 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:max__json_schema PASS 100% 137 in → 795 out (760 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling:medium PASS 100% 135 in → 194 out (167 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:none PASS 100% 133 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:xhigh PASS 100% 136 in → 720 out (711 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:xhigh__json_schema PASS 100% 136 in → 421 out (386 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.3:high PASS 100% 304 in → 664 out (645 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:high__json_schema PASS 100% 362 in → 1,331 out (1,301 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.3:low PASS 100% 311 in → 592 out (573 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:low__json_schema PASS 100% 367 in → 417 out (392 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
x-ai/grok-4.3:max PASS 100% 304 in → 506 out (482 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.3:max__json_schema PASS 100% 362 in → 557 out (528 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
x-ai/grok-4.3:medium PASS 100% 311 in → 606 out (587 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:medium__json_schema PASS 100% 368 in → 771 out (742 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
x-ai/grok-4.3:none PASS 100% 304 in → 24 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.3:none__json_schema PASS 100% 359 in → 29 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0"}
x-ai/grok-4.3:xhigh PASS 100% 303 in → 664 out (645 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:xhigh__json_schema PASS 100% 363 in → 635 out (610 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
x-ai/grok-4.5:high PASS 100% 327 in → 407 out (383 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:high__json_schema PASS 100% 397 in → 453 out (420 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:low PASS 100% 327 in → 156 out (132 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:low__json_schema PASS 100% 396 in → 352 out (319 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:max PASS 100% 327 in → 229 out (205 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:max__json_schema PASS 100% 397 in → 567 out (534 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:medium PASS 100% 329 in → 254 out (230 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:medium__json_schema PASS 100% 397 in → 265 out (235 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
x-ai/grok-4.5:xhigh PASS 100% 327 in → 273 out (249 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 397 in → 438 out (408 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
z-ai/glm-4.7-flash:high__json_schema PASS 100% 132 in → 575 out (496 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-4.7-flash:none PASS 100% 132 in → 21 out (0 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-4.7-flash:none__json_schema PASS 100% 133 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-4.7-flash:xhigh PASS 100% 131 in → 1,617 out (1,481 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 131 in → 709 out (658 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-5.2:high PASS 100% 138 in → 477 out (455 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:high__json_schema PASS 100% 139 in → 164 out (108 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-5.2:low PASS 100% 139 in → 201 out (174 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
z-ai/glm-5.2:low__json_schema PASS 100% 141 in → 433 out (401 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-5.2:medium PASS 100% 139 in → 233 out (206 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
z-ai/glm-5.2:medium__json_schema PASS 100% 313 in → 183 out (154 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
z-ai/glm-5.2:none PASS 100% 133 in → 21 out (0 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:none__json_schema PASS 100% 133 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
z-ai/glm-5.2:xhigh PASS 100% 138 in → 193 out (171 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:xhigh__json_schema PASS 100% 314 in → 178 out (148 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
anthropic/claude-fable-5:medium FAIL 0% 200 in → 55 out (6 reasoning)
(empty output)
Failure: Content mismatch: ['config.yaml']
anthropic/claude-opus-4.6:max FAIL 0% 146 in → 404 out (172 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Reasoning:** Each side made one independent, non-overlapping change relative to the base: | Field | Base | HEAD (current) | Incoming | Resolved | |-------|------|----------------|----------|----------| | `port` | 8080 | 8080 *(unchanged)* | **443** | **443** | | `host` | localhost | **0.0.0.0** | localhost *(unchanged)* | **0.0.0.0** | The correct resolution accepts **both** intentional changes: - **port → 443** from the cherry-picked commit (HTTPS port) - **host → 0.0.0.0** from the current branch (bind to all interfaces)
Failure: Content mismatch: ['config.yaml']
anthropic/claude-opus-4.7:none FAIL 0% 206 in → 40 out (0 reasoning)
```yaml --- config.yaml server: port: 443 host: 0.0.0.0 ```
Failure: Expected file heading before file content
anthropic/claude-opus-4.8:high FAIL 0% 202 in → 397 out (111 reasoning)
```yaml --- config.yaml server: port: 443 host: 0.0.0.0 ``` **Reasoning:** For each field, take the side that actually changed it from the base: - **port**: base was `8080`; incoming changed it to `443` (HTTPS), current kept `8080`. The meaningful change is `443`. - **host**: base was `localhost`; incoming kept `localhost`, current changed it to `0.0.0.0`. The meaningful change is `0.0.0.0`. This merges both intended changes without discarding either.
Failure: Expected file heading before file content
anthropic/claude-opus-4.8:max FAIL 0% 200 in → 780 out (135 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Reasoning:** Each side modified a *different* line relative to the base (`port: 8080`, `host: localhost`): - The incoming commit ("Use HTTPS port") changed **only the port** → `443`, leaving host untouched. - The current branch (HEAD) changed **only the host** → `0.0.0.0`, leaving port untouched. Since the two changes don't semantically conflict, the correct resolution combines both intents: the HTTPS port `443` from the incoming commit and the `0.0.0.0` bind address from HEAD.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-opus-4.8:none FAIL 0% 201 in → 388 out (0 reasoning)
Looking at this conflict, I need to reason about intent rather than blindly picking one side: - **Incoming commit's intent**: Switch to HTTPS (port 443). This is the *functional change* being cherry-picked. - **Current branch's intent**: Bind to all interfaces (host `0.0.0.0`) instead of just localhost. This is an *independent change* on a different key. These two changes don't actually conflict semantically — they touch different concerns (protocol/port vs. bind address). Git flagged them only because they're adjacent lines. The correct resolution combines both intents: ```yaml server: port: 443 host: 0.0.0.0 ``` **Rationale:** Take the port from the cherry-picked commit (the whole point of the pick is to adopt HTTPS/443) and keep the host from the current branch (`0.0.0.0`), since that was a deliberate change here and reverting to `localhost` would silently undo it. Note: If binding `0.0.0.0` on port 443 is intentional, make sure the process has the privileges/TLS config to serve HTTPS on that interface — but that's a follow-up concern, not part of resolving the merge.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-sonnet-4.6:high FAIL 0% 147 in → 470 out (191 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Resolution logic:** | Setting | Base | HEAD (current) | Incoming cherry-pick | Change type | Resolution | |---------|------|----------------|----------------------|-------------|------------| | `port` | 8080 | 8080 | **443** | Cherry-pick modified this | **443** — apply the cherry-pick's intent | | `host` | localhost | **0.0.0.0** | localhost | Current branch modified this | **0.0.0.0** — preserve independent local change | The cherry-pick's sole purpose was updating the port to 443. The current branch independently changed the host to `0.0.0.0`. These are **non-overlapping changes**, so both are accepted: the cherry-picked port change is applied on top of the current branch's host change.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-sonnet-4.6:max FAIL 0% 147 in → 499 out (214 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Resolution logic:** | Field | Base | HEAD (current) | Incoming | Change? | Resolution | |-------|------|----------------|----------|---------|------------| | `port` | 8080 | 8080 | **443** | Incoming modified | `443` | | `host` | localhost | **0.0.0.0** | localhost | HEAD modified | `0.0.0.0` | Each side modified a **different field** independently, so both changes are applied: - `port: 443` — taken from the cherry-picked commit (its sole purpose was the HTTPS port change) - `host: 0.0.0.0` — taken from HEAD (independent change to bind all interfaces, unrelated to the cherry-pick)
Failure: Content mismatch: ['config.yaml']
anthropic/claude-sonnet-4.6:medium FAIL 0% 147 in → 213 out (77 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Resolution logic:** Both changes are independent modifications to different keys — the incoming commit (`d18f896`) introduced `port: 443`, and the current HEAD branch introduced `host: 0.0.0.0`. Neither change conflicts semantically, so both are preserved. The base values (`port: 8080`, `host: localhost`) are replaced by their respective branch modifications.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 148 in → 428 out (175 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Resolution rationale:** Both changes are to **independent fields** and should be combined: | Field | Base | HEAD (current) | Incoming commit | Resolution | |-------|------|----------------|-----------------|------------| | `port` | 8080 | 8080 (unchanged) | **443** (modified) | **443** — take the cherry-picked change | | `host` | localhost | **0.0.0.0** (modified) | localhost (unchanged) | **0.0.0.0** — keep the current branch change | Since the cherry-picked commit only intentionally touched `port` (keeping `host` as the base value), and the current branch only intentionally touched `host`, both changes are non-conflicting in intent and should be merged together.
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high__json_schema FAIL 0% 130 in → 1,287 out (1,324 reasoning)
server: port: 8080 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 8080\n host: localhost\n" }
Failure: Content mismatch: ['config.yaml']
arcee-ai/trinity-mini:low FAIL 0% 132 in → 618 out (645 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 131 in → 464 out (436 reasoning)
--- server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "---\nserver:\n port: 443\n host: 0.0.0.0\n"}
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
ibm-granite/granite-4.1-8b FAIL 0% 135 in → 13 out
server: port: 443 host: localhost
Failure: Content mismatch: ['config.yaml']
--- config.yaml server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: localhost\n" }
Failure: Content mismatch: ['config.yaml']
liquid/lfm-2-24b-a2b FAIL 0% 152 in → 20 out
```yaml server: port: 443 host: localhost ```
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: ```yaml server: port: 443 host: localhost ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
mistralai/mistral-medium-3-5:xhigh__json_schema FAIL 0% 144 in → 493 out (416 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "---\nconfig.yaml\nserver:\n port: 443\n host: 0.0.0.0"}
Failure: Expected file heading before file content
moonshotai/kimi-k2.6:low FAIL 0% 130 in → 787 out (766 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
moonshotai/kimi-k2.6:medium FAIL 0% 131 in → 1,211 out (1,190 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
moonshotai/kimi-k2.6:xhigh FAIL 0% 132 in → 728 out (707 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
moonshotai/kimi-k2.7-code:high FAIL 0% 130 in → 546 out (551 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
moonshotai/kimi-k2.7-code:low FAIL 0% 132 in → 243 out (222 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k3:high FAIL 0% 208 in → 247 out (207 reasoning)
```yaml --- server: port: 443 host: 0.0.0.0 ```
Failure: Content mismatch: ['config.yaml']
(empty output)
JSON Schema Structured Output
Failure: OpenAI-compatible response contained provider error: Provider returned error
(empty output)
Failure: OpenAI-compatible response contained provider error: Provider returned error
nvidia/nemotron-3-nano-30b-a3b:low FAIL 0% 158 in → 604 out (511 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
Failure: Expected file heading before file content
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 158 in → 1,387 out (1,361 reasoning)
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: localhost" }
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-nano-30b-a3b:medium FAIL 0% 156 in → 1,157 out (1,031 reasoning)
server: port: 443 host: localhost
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema FAIL 0% 156 in → 773 out (771 reasoning)
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: localhost" }
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 158 in → 23 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 156 in → 28 out (0 reasoning)
server: port: 8080 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 8080\n host: localhost\n" }
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 157 in → 784 out (749 reasoning)
server: port: 443 host: localhost
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 0% 158 in → 782 out (706 reasoning)
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: localhost" }
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.4-nano:low FAIL 0% 128 in → 89 out (63 reasoning)
server: port: 8080 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.4-nano:none FAIL 0% 127 in → 16 out (0 reasoning)
server: port: 443 host: localhost
Failure: Content mismatch: ['config.yaml']
poolside/laguna-m.1:xhigh FAIL 0% 152 in → 599 out (568 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0 ---
Failure: Content mismatch: ['config.yaml']
poolside/laguna-xs-2.1:none FAIL 0% 151 in → 46 out (0 reasoning)
server: port: 443 host: 0.0.0.0 </think>server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: --- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a55415e1-f324-9bd2-949d-b8e5a217aae1","request_id":"a55415e1-f324-9bd2-949d-b8e5a217aae1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-77cf038d-72b9-931d-bbe3-84848bdefcca","request_id":"77cf038d-72b9-931d-bbe3-84848bdefcca"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4355d2d3-e64a-985c-8603-064c5683f776","request_id":"4355d2d3-e64a-985c-8603-064c5683f776"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-838eae22-c1b1-91c7-8a4d-505db534bda6","request_id":"838eae22-c1b1-91c7-8a4d-505db534bda6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2b2eafa2-5747-91c7-8a50-516dffccb3be","request_id":"2b2eafa2-5747-91c7-8a50-516dffccb3be"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-797a5745-f307-961d-96b6-f27033991019"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0edd35c5-0d86-9194-9c94-0d9485f23a23"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1bb9c3c1-708a-9fe0-b510-d7b1b54bde92"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f4c10349-3a4f-9df9-b16f-0dbfdac06722"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-44c8f7d2-1d1b-9109-a372-64ffaa41e357"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ea0cb168-a66c-998f-b2fa-acceefbdb63e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9ba8e00b-edfb-9de0-91da-593abf82d66a","request_id":"9ba8e00b-edfb-9de0-91da-593abf82d66a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0c0e0358-e710-9c52-b6ac-55175a8bd146","request_id":"0c0e0358-e710-9c52-b6ac-55175a8bd146"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0a5469d3-327d-92d7-9e92-4a957b5a7f20","request_id":"0a5469d3-327d-92d7-9e92-4a957b5a7f20"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9b0bde4e-eac7-9747-b785-97edfe85452c","request_id":"9b0bde4e-eac7-9747-b785-97edfe85452c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7206e72d-a0cc-9163-91d4-1e2de00133cf","request_id":"7206e72d-a0cc-9163-91d4-1e2de00133cf"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-960f0372-4061-96b0-b965-3ab4e1397fcb","request_id":"960f0372-4061-96b0-b965-3ab4e1397fcb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bd06d414-1e88-9d62-be94-80cfa1d9e22f","request_id":"bd06d414-1e88-9d62-be94-80cfa1d9e22f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-45607e15-a866-93df-98cb-153ba1c6623a","request_id":"45607e15-a866-93df-98cb-153ba1c6623a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f72dc8f4-d659-97c4-9812-54c35e11dbf1","request_id":"f72dc8f4-d659-97c4-9812-54c35e11dbf1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b28a5f39-1682-9959-9ca0-8ffd6177ee5d","request_id":"b28a5f39-1682-9959-9ca0-8ffd6177ee5d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: --- config.yaml server: port: 443 host: 0.0.0.0{"resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling-small:xhigh__json_schema FAIL 0% 135 in → 411 out (377 reasoning)
server:\n port: 443\n host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\\n port: 443\\n host: 0.0.0.0"}
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: --- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: server: port: 443 host: 0.0.0.0{ "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:high FAIL 0% 132 in → 3,515 out (3,570 reasoning)
```yaml server: port: 443 host: localhost ```
Failure: Content mismatch: ['config.yaml']
z-ai/glm-4.7-flash:low FAIL 0% 133 in → 612 out (613 reasoning)
```yaml server: port: 443 host: localhost ```
Failure: Content mismatch: ['config.yaml']
z-ai/glm-4.7-flash:low__json_schema FAIL 0% 132 in → 1,442 out (1,404 reasoning)
server: port: 443 host: localhost
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: localhost" }
Failure: Content mismatch: ['config.yaml']
z-ai/glm-4.7-flash:medium FAIL 0% 130 in → 3,599 out (3,579 reasoning)
```yaml server: port: 443 host: localhost ```
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)