Multi-file: conflict in settings.json only
Tests ability to resolve a JSON config conflict preserving valid JSON syntax. Evaluates structured-data conflict resolution.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf '{ "name": "app", "version": "1.0.0" }' > settings.json
  5. 05 printf 'DB_HOST=localhost DB_PORT=5432' > database.ini
  6. 06 git add settings.json database.ini
  7. 07 git commit -m 'Initial config'
  8. 08 git checkout -b v2
  9. 09 printf '{ "name": "app", "version": "2.0.0" }' > settings.json
  10. 10 git add settings.json
  11. 11 git commit -m 'Bump version to 2.0.0'
  12. 12 git checkout main
  13. 13 printf '{ "name": "MyApp", "version": "1.0.0" }' > settings.json
  14. 14 git add settings.json
  15. 15 git commit -m 'Rename app to MyApp'
  16. 16 git cherry-pick $(git rev-parse v2)
Prompt
Resolve the cherry-pick conflict in settings.json. The base had name: app, version: 1.0.0. The incoming commit (v2 branch) kept name: app but bumped version to 2.0.0. The current branch (main) renamed name to MyApp but kept version 1.0.0. Provide ONLY the resolved settings.json content as valid JSON.
Expected
{ "name": "MyApp", "version": "2.0.0" }

Scoped model quality, cost, API time, and token usage for cherry_pick/f011.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high__json_schema PASS 100% 448 in → 75 out (22 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-fable-5:low PASS 100% 228 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-fable-5:low__json_schema PASS 100% 448 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-fable-5:max PASS 100% 229 in → 351 out (69 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-fable-5:max__json_schema PASS 100% 446 in → 351 out (53 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-fable-5:medium PASS 100% 229 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 446 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-fable-5:xhigh PASS 100% 226 in → 68 out (42 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 448 in → 93 out (41 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-haiku-4.5:high PASS 100% 211 in → 327 out (228 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 377 in → 830 out (560 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-haiku-4.5:low PASS 100% 209 in → 376 out (282 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 377 in → 384 out (285 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-haiku-4.5:medium PASS 100% 210 in → 387 out (312 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 378 in → 355 out (257 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-haiku-4.5:none PASS 100% 181 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 347 in → 39 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 209 in → 263 out (186 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.6:high PASS 100% 180 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 348 in → 97 out (35 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:low PASS 100% 180 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 350 in → 38 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 347 in → 227 out (137 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:medium PASS 100% 180 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 349 in → 38 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:none PASS 100% 178 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 350 in → 38 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:xhigh PASS 100% 182 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 348 in → 97 out (35 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.7:high PASS 100% 233 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:high__json_schema PASS 100% 454 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.7:low PASS 100% 234 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:low__json_schema PASS 100% 454 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.7:max PASS 100% 231 in → 338 out (81 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 453 in → 347 out (132 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-opus-4.7:medium PASS 100% 233 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 454 in → 54 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-opus-4.7:none PASS 100% 231 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:none__json_schema PASS 100% 455 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.7:xhigh PASS 100% 234 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 453 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.8:high PASS 100% 230 in → 135 out (77 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 448 in → 83 out (17 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-opus-4.8:low PASS 100% 229 in → 72 out (19 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 448 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-opus-4.8:max PASS 100% 230 in → 472 out (111 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 450 in → 323 out (134 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 449 in → 84 out (17 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 446 in → 49 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.8:xhigh PASS 100% 228 in → 261 out (65 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 448 in → 105 out (53 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 348 in → 91 out (35 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-4.6:low PASS 100% 182 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 348 in → 39 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 347 in → 338 out (179 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-4.6:medium PASS 100% 179 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 350 in → 39 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-4.6:none PASS 100% 182 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 350 in → 39 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 180 in → 101 out (57 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 349 in → 89 out (36 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-5:high PASS 100% 229 in → 90 out (43 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 450 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-sonnet-5:low PASS 100% 228 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 448 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-sonnet-5:medium PASS 100% 228 in → 63 out (32 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 448 in → 50 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-5:none PASS 100% 228 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 446 in → 51 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 228 in → 105 out (50 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 448 in → 110 out (38 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 161 in → 548 out (528 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-large-thinking:low PASS 100% 160 in → 1,393 out (1,382 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-large-thinking:medium PASS 100% 159 in → 701 out (682 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-large-thinking:xhigh PASS 100% 159 in → 552 out (533 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-mini:high PASS 100% 158 in → 473 out (460 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-mini:low PASS 100% 158 in → 657 out (628 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-mini:low__json_schema PASS 100% 159 in → 651 out (644 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
arcee-ai/trinity-mini:medium PASS 100% 159 in → 455 out (447 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 239 in → 143 out (121 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 246 in → 163 out (114 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 161 in → 244 out (208 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 160 in → 157 out (103 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content" : "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 251 in → 227 out (194 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 240 in → 79 out (42 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 160 in → 268 out (246 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 162 in → 172 out (115 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 160 in → 26 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 161 in → 41 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 161 in → 130 out (89 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 161 in → 228 out (185 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
deepseek/deepseek-v4-flash:high PASS 100% 161 in → 154 out (113 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 160 in → 412 out (372 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash:low PASS 100% 161 in → 147 out (101 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 164 in → 294 out (257 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash:medium PASS 100% 160 in → 229 out (206 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash:none PASS 100% 159 in → 26 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 230 in → 34 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 241 in → 612 out (575 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-pro:high PASS 100% 161 in → 464 out (441 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 160 in → 36 out (35 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-pro:low PASS 100% 161 in → 200 out (178 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 161 in → 36 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-pro:medium PASS 100% 160 in → 269 out (246 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 161 in → 293 out (256 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-pro:none PASS 100% 161 in → 25 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 353 in → 36 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 240 in → 772 out (749 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 431 in → 756 out (721 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3-flash-preview:high PASS 100% 163 in → 334 out (311 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3-flash-preview:low PASS 100% 164 in → 315 out (292 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3-flash-preview:medium PASS 100% 165 in → 258 out (235 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3-flash-preview:xhigh PASS 100% 165 in → 285 out (262 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-flash-lite-preview:high PASS 100% 165 in → 303 out (280 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-flash-lite-preview:low PASS 100% 163 in → 156 out (133 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 225 in → 171 out (137 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 166 in → 237 out (214 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 163 in → 449 out (426 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-pro-preview:high PASS 100% 163 in → 393 out (370 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-pro-preview:low PASS 100% 165 in → 307 out (284 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-pro-preview:medium PASS 100% 165 in → 559 out (536 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-pro-preview:xhigh PASS 100% 163 in → 632 out (609 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash-lite:high PASS 100% 162 in → 521 out (498 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash-lite:low PASS 100% 162 in → 23 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 166 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemini-3.5-flash-lite:max PASS 100% 163 in → 681 out (658 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash-lite:medium PASS 100% 166 in → 319 out (296 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 162 in → 748 out (725 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash:high PASS 100% 164 in → 624 out (601 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash:high__json_schema PASS 100% 224 in → 1,146 out (1,116 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.5-flash:low PASS 100% 163 in → 239 out (216 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash:low__json_schema PASS 100% 223 in → 316 out (281 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemini-3.5-flash:medium PASS 100% 164 in → 390 out (367 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash:medium__json_schema PASS 100% 225 in → 514 out (483 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.5-flash:xhigh PASS 100% 164 in → 399 out (376 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 223 in → 545 out (515 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.6-flash:high PASS 100% 166 in → 533 out (510 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.6-flash:high__json_schema PASS 100% 223 in → 905 out (875 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.6-flash:low PASS 100% 165 in → 111 out (88 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.6-flash:low__json_schema PASS 100% 225 in → 255 out (225 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.6-flash:max PASS 100% 165 in → 641 out (618 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.6-flash:max__json_schema PASS 100% 165 in → 956 out (926 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.6-flash:medium PASS 100% 164 in → 436 out (413 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.6-flash:medium__json_schema PASS 100% 223 in → 633 out (603 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.6-flash:xhigh PASS 100% 164 in → 548 out (525 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 223 in → 1,005 out (975 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemma-4-26b-a4b-it:high PASS 100% 180 in → 317 out (217 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-26b-a4b-it:low PASS 100% 181 in → 453 out (332 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-26b-a4b-it:medium PASS 100% 181 in → 409 out (311 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-26b-a4b-it:none PASS 100% 178 in → 29 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-26b-a4b-it:xhigh PASS 100% 180 in → 620 out (479 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-31b-it:high PASS 100% 178 in → 341 out (252 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-31b-it:low PASS 100% 176 in → 234 out (163 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-31b-it:medium PASS 100% 181 in → 243 out (158 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-31b-it:none PASS 100% 176 in → 29 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-31b-it:xhigh PASS 100% 181 in → 235 out (159 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 182 in → 238 out (206 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
ibm-granite/granite-4.1-8b PASS 100% 170 in → 26 out
```json { "name": "MyApp", "version": "2.0.0" } ```
inclusionai/ling-2.6-flash PASS 100% 184 in → 25 out
{ "name": "MyApp", "version": "2.0.0" }
inclusionai/ling-2.6-flash__json_schema PASS 100% 185 in → 40 out
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m2.5:high PASS 100% 192 in → 345 out (287 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.5:low PASS 100% 192 in → 495 out (458 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.5:medium PASS 100% 191 in → 1,868 out (1,685 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.5:medium__json_schema PASS 100% 205 in → 284 out (244 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m2.5:xhigh PASS 100% 192 in → 461 out (438 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 205 in → 517 out (456 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m2.7:high PASS 100% 196 in → 290 out (268 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.7:medium PASS 100% 193 in → 291 out (268 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.7:xhigh PASS 100% 193 in → 512 out (445 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 323 in → 1,309 out (1,274 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m3:high__json_schema PASS 100% 279 in → 35 out (33 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m3:low PASS 100% 329 in → 303 out (251 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
minimax/minimax-m3:low__json_schema PASS 100% 318 in → 34 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m3:medium__json_schema PASS 100% 317 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m3:xhigh PASS 100% 331 in → 242 out (182 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
minimax/minimax-m3:xhigh__json_schema PASS 100% 279 in → 34 out (32 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
mistralai/devstral-2512 PASS 100% 163 in → 26 out
```json { "name": "MyApp", "version": "2.0.0" } ```
mistralai/devstral-2512__json_schema PASS 100% 164 in → 35 out
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
mistralai/mistral-medium-3-5:high PASS 100% 175 in → 372 out (314 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 163 in → 303 out (271 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
mistralai/mistral-medium-3-5:low PASS 100% 176 in → 393 out (334 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 161 in → 540 out (476 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
mistralai/mistral-medium-3-5:medium PASS 100% 177 in → 603 out (541 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 163 in → 421 out (356 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
mistralai/mistral-medium-3-5:none PASS 100% 177 in → 26 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 163 in → 33 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 175 in → 297 out (245 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 163 in → 343 out (285 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
moonshotai/kimi-k2.6:high__json_schema PASS 100% 160 in → 696 out (720 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
moonshotai/kimi-k2.6:low__json_schema PASS 100% 345 in → 1,427 out (1,396 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
moonshotai/kimi-k2.6:medium PASS 100% 159 in → 345 out (316 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 159 in → 33 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
moonshotai/kimi-k2.6:xhigh PASS 100% 158 in → 315 out (288 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 159 in → 869 out (834 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n" }
moonshotai/kimi-k2.7-code:high PASS 100% 159 in → 228 out (205 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 160 in → 155 out (118 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
moonshotai/kimi-k2.7-code:low PASS 100% 160 in → 297 out (276 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 160 in → 171 out (134 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 244 in → 264 out (232 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 160 in → 224 out (189 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
moonshotai/kimi-k3:high PASS 100% 238 in → 279 out (243 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
moonshotai/kimi-k3:high__json_schema PASS 100% 238 in → 189 out (149 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:low PASS 100% 237 in → 191 out (151 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:low__json_schema PASS 100% 237 in → 216 out (176 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:max PASS 100% 236 in → 249 out (209 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:max__json_schema PASS 100% 236 in → 182 out (142 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:medium__json_schema PASS 100% 238 in → 367 out (331 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
moonshotai/kimi-k3:xhigh PASS 100% 237 in → 200 out (160 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 238 in → 202 out (162 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 177 in → 816 out (717 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 177 in → 398 out (336 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 177 in → 458 out (415 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 178 in → 295 out (239 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 175 in → 544 out (474 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 177 in → 22 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 178 in → 432 out (378 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 176 in → 431 out (361 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 176 in → 213 out (163 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 176 in → 169 out (127 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 177 in → 155 out (123 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 177 in → 269 out (227 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 178 in → 133 out (105 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 177 in → 207 out (168 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 177 in → 22 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 174 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
openai/gpt-5.4-mini:high PASS 100% 157 in → 173 out (146 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-mini:high__json_schema PASS 100% 192 in → 134 out (95 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-mini:low PASS 100% 158 in → 41 out (14 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-mini:low__json_schema PASS 100% 192 in → 70 out (31 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-mini:medium PASS 100% 155 in → 120 out (93 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-mini:medium__json_schema PASS 100% 193 in → 146 out (107 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-mini:none PASS 100% 157 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-mini:none__json_schema PASS 100% 195 in → 43 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-mini:xhigh PASS 100% 158 in → 341 out (314 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 195 in → 555 out (516 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-nano:high PASS 100% 156 in → 108 out (81 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-nano:high__json_schema PASS 100% 194 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-nano:low PASS 100% 155 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-nano:low__json_schema PASS 100% 194 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-nano:medium PASS 100% 156 in → 56 out (29 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-nano:medium__json_schema PASS 100% 194 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-nano:none PASS 100% 156 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-nano:none__json_schema PASS 100% 193 in → 38 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 158 in → 147 out (120 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 192 in → 155 out (116 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4:high PASS 100% 156 in → 108 out (81 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4:high__json_schema PASS 100% 192 in → 276 out (237 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4:low PASS 100% 156 in → 51 out (24 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4:low__json_schema PASS 100% 192 in → 65 out (26 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4:medium PASS 100% 157 in → 59 out (32 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4:medium__json_schema PASS 100% 192 in → 101 out (62 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4:none PASS 100% 156 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4:none__json_schema PASS 100% 194 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4:xhigh PASS 100% 157 in → 85 out (58 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4:xhigh__json_schema PASS 100% 193 in → 148 out (109 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.5:high PASS 100% 155 in → 105 out (78 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.5:high__json_schema PASS 100% 194 in → 178 out (139 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.5:low PASS 100% 158 in → 52 out (25 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.5:low__json_schema PASS 100% 193 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.5:medium PASS 100% 157 in → 57 out (30 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.5:medium__json_schema PASS 100% 192 in → 75 out (36 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.5:none PASS 100% 156 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.5:none__json_schema PASS 100% 193 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.5:xhigh PASS 100% 157 in → 136 out (109 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.5:xhigh__json_schema PASS 100% 192 in → 217 out (178 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:high PASS 100% 156 in → 53 out (26 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:high__json_schema PASS 100% 191 in → 289 out (250 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:low PASS 100% 156 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:low__json_schema PASS 100% 194 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:max PASS 100% 156 in → 68 out (41 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:max__json_schema PASS 100% 192 in → 555 out (516 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:medium PASS 100% 158 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:medium__json_schema PASS 100% 193 in → 117 out (78 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:none PASS 100% 159 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:none__json_schema PASS 100% 192 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:xhigh PASS 100% 157 in → 58 out (31 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 193 in → 75 out (36 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:high PASS 100% 157 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:high__json_schema PASS 100% 193 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:low PASS 100% 158 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:low__json_schema PASS 100% 191 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:max PASS 100% 157 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:max__json_schema PASS 100% 193 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:medium PASS 100% 158 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:medium__json_schema PASS 100% 193 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:none PASS 100% 158 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:none__json_schema PASS 100% 193 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:xhigh PASS 100% 157 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 194 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:high PASS 100% 157 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:high__json_schema PASS 100% 193 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:low PASS 100% 156 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:low__json_schema PASS 100% 194 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:max PASS 100% 157 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:max__json_schema PASS 100% 195 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:medium PASS 100% 157 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:medium__json_schema PASS 100% 194 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:none PASS 100% 156 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:none__json_schema PASS 100% 193 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:xhigh PASS 100% 157 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 192 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-oss-120b:high PASS 100% 219 in → 205 out (184 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-120b:low PASS 100% 228 in → 76 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-120b:medium PASS 100% 218 in → 144 out (94 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-120b:xhigh PASS 100% 219 in → 433 out (377 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-20b:high PASS 100% 217 in → 565 out (551 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-20b:low PASS 100% 216 in → 79 out (46 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-20b:medium PASS 100% 215 in → 158 out (112 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-20b:xhigh PASS 100% 204 in → 410 out (387 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:high PASS 100% 171 in → 517 out (491 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:low PASS 100% 172 in → 374 out (348 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:low__json_schema PASS 100% 174 in → 363 out (327 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
poolside/laguna-m.1:medium PASS 100% 174 in → 416 out (390 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:medium__json_schema PASS 100% 173 in → 333 out (298 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
poolside/laguna-m.1:none PASS 100% 173 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:xhigh PASS 100% 173 in → 458 out (432 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:xhigh__json_schema PASS 100% 172 in → 379 out (348 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
poolside/laguna-xs-2.1:high PASS 100% 174 in → 474 out (450 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs-2.1:low PASS 100% 172 in → 487 out (463 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs-2.1:medium PASS 100% 174 in → 998 out (974 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs-2.1:none PASS 100% 171 in → 23 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs-2.1:xhigh PASS 100% 172 in → 417 out (393 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs.2:high PASS 100% 211 in → 248 out (222 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs.2:high__json_schema PASS 100% 211 in → 253 out (221 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
poolside/laguna-xs.2:low PASS 100% 209 in → 392 out (366 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs.2:low__json_schema PASS 100% 211 in → 359 out (324 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
poolside/laguna-xs.2:medium PASS 100% 210 in → 323 out (292 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
poolside/laguna-xs.2:medium__json_schema PASS 100% 211 in → 228 out (193 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
poolside/laguna-xs.2:none PASS 100% 210 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs.2:xhigh PASS 100% 211 in → 247 out (221 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 210 in → 326 out (291 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-27b:high PASS 100% 172 in → 1,273 out (1,087 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-27b:low PASS 100% 172 in → 1,362 out (1,179 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-27b:medium PASS 100% 171 in → 1,010 out (938 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-27b:medium__json_schema PASS 100% 171 in → 577 out (454 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-27b:none PASS 100% 174 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-27b:none__json_schema PASS 100% 170 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-27b:xhigh PASS 100% 171 in → 1,106 out (985 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:high PASS 100% 171 in → 848 out (783 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 172 in → 1,145 out (1,013 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-35b-a3b:low PASS 100% 172 in → 573 out (469 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 169 in → 213 out (152 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 172 in → 1,020 out (991 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 171 in → 950 out (848 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-35b-a3b:none PASS 100% 171 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 173 in → 36 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 172 in → 1,000 out (899 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 170 in → 1,361 out (1,246 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-flash:high PASS 100% 172 in → 934 out (905 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-flash:low PASS 100% 171 in → 933 out (904 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-flash:medium PASS 100% 171 in → 917 out (888 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-flash:none PASS 100% 172 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-flash:xhigh PASS 100% 169 in → 395 out (366 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:high PASS 100% 170 in → 656 out (627 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:low PASS 100% 172 in → 7,229 out (7,195 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
qwen/qwen3.7-flash:max PASS 100% 170 in → 1,090 out (1,061 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:medium PASS 100% 171 in → 1,301 out (1,272 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:none PASS 100% 172 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:xhigh PASS 100% 170 in → 639 out (610 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-max:high PASS 100% 172 in → 342 out (308 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
qwen/qwen3.7-max:low PASS 100% 170 in → 272 out (238 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
qwen/qwen3.7-max:medium PASS 100% 168 in → 313 out (279 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
qwen/qwen3.7-max:none PASS 100% 171 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-max:xhigh PASS 100% 172 in → 259 out (225 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
qwen/qwen3.7-plus:high PASS 100% 171 in → 477 out (448 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-plus:low PASS 100% 171 in → 682 out (653 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-plus:medium PASS 100% 171 in → 202 out (168 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
qwen/qwen3.7-plus:none PASS 100% 174 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-plus:xhigh PASS 100% 172 in → 819 out (790 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:high PASS 100% 167 in → 650 out (626 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:high__json_schema PASS 100% 169 in → 460 out (408 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
tencent/hy3:low PASS 100% 167 in → 655 out (631 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:low__json_schema PASS 100% 167 in → 511 out (428 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
tencent/hy3:medium PASS 100% 168 in → 671 out (647 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:medium__json_schema PASS 100% 169 in → 554 out (471 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
tencent/hy3:none PASS 100% 171 in → 23 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:xhigh PASS 100% 168 in → 465 out (441 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:xhigh__json_schema PASS 100% 167 in → 1,569 out (1,458 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
thinkingmachines/inkling-small:high PASS 100% 167 in → 160 out (133 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling-small:low PASS 100% 164 in → 54 out (27 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling-small:low__json_schema PASS 100% 164 in → 118 out (78 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
thinkingmachines/inkling-small:max PASS 100% 166 in → 219 out (182 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling-small:medium PASS 100% 168 in → 129 out (99 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling-small:medium__json_schema PASS 100% 165 in → 171 out (131 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
thinkingmachines/inkling-small:none PASS 100% 163 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling-small:xhigh PASS 100% 166 in → 168 out (130 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 166 in → 208 out (168 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
thinkingmachines/inkling:high PASS 100% 166 in → 248 out (221 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:low PASS 100% 167 in → 50 out (22 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:low__json_schema PASS 100% 164 in → 106 out (66 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
thinkingmachines/inkling:max PASS 100% 165 in → 519 out (492 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:medium PASS 100% 167 in → 144 out (117 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:xhigh PASS 100% 167 in → 177 out (144 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:xhigh__json_schema PASS 100% 167 in → 302 out (263 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.3:high PASS 100% 336 in → 666 out (645 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:high__json_schema PASS 100% 392 in → 971 out (939 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.3:low PASS 100% 341 in → 450 out (429 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:low__json_schema PASS 100% 398 in → 462 out (430 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.3:max PASS 100% 337 in → 423 out (402 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:max__json_schema PASS 100% 391 in → 819 out (790 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
x-ai/grok-4.3:medium PASS 100% 340 in → 462 out (441 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:medium__json_schema PASS 100% 398 in → 843 out (811 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.3:none PASS 100% 335 in → 21 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:none__json_schema PASS 100% 389 in → 32 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.3:xhigh PASS 100% 336 in → 473 out (452 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:xhigh__json_schema PASS 100% 393 in → 876 out (844 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.5:high PASS 100% 358 in → 481 out (459 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.5:high__json_schema PASS 100% 429 in → 368 out (336 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
x-ai/grok-4.5:low PASS 100% 358 in → 172 out (150 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.5:low__json_schema PASS 100% 429 in → 201 out (166 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.5:max PASS 100% 359 in → 307 out (285 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.5:max__json_schema PASS 100% 428 in → 553 out (518 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.5:medium PASS 100% 361 in → 309 out (287 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.5:medium__json_schema PASS 100% 429 in → 441 out (406 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.5:xhigh PASS 100% 360 in → 181 out (159 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.5:xhigh__json_schema PASS 100% 427 in → 506 out (471 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
z-ai/glm-4.7-flash:high PASS 100% 156 in → 2,586 out (2,575 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
z-ai/glm-4.7-flash:low PASS 100% 157 in → 638 out (572 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
z-ai/glm-4.7-flash:medium PASS 100% 158 in → 749 out (634 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
z-ai/glm-4.7-flash:none PASS 100% 155 in → 25 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
z-ai/glm-4.7-flash:none__json_schema PASS 100% 156 in → 33 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
z-ai/glm-4.7-flash:xhigh PASS 100% 154 in → 543 out (455 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:high PASS 100% 163 in → 157 out (115 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:high__json_schema PASS 100% 162 in → 290 out (249 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
JSON Schema Structured Output
(raw) { "resolved_content": "```json\n{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n```" }
z-ai/glm-5.2:low PASS 100% 164 in → 88 out (65 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:low__json_schema PASS 100% 338 in → 254 out (220 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
z-ai/glm-5.2:medium PASS 100% 164 in → 443 out (371 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:none PASS 100% 156 in → 22 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:none__json_schema PASS 100% 333 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
z-ai/glm-5.2:xhigh PASS 100% 163 in → 122 out (100 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:xhigh__json_schema PASS 100% 162 in → 31 out (30 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-fable-5:high FAIL 0% 228 in → 149 out (17 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` **Rationale:** Each branch changed a different field relative to the base (`name: app, version: 1.0.0`). Main renamed `name` to `MyApp`; the cherry-picked commit bumped `version` to `2.0.0`. The correct resolution merges both intentional changes.
Failure: Content mismatch: ['settings.json']
anthropic/claude-haiku-4.5:xhigh__json_schema FAIL 0% 377 in → 385 out (294 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
anthropic/claude-opus-4.6:max FAIL 0% 180 in → 483 out (154 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` ## Reasoning The resolution applies **both non-conflicting changes** relative to the base: | Field | Base | HEAD (main) | Incoming (v2) | Resolution | |-----------|---------|-------------|---------------|------------| | `name` | `app` | `MyApp` ✏️ | `app` | **`MyApp`** (main's rename) | | `version` | `1.0.0` | `1.0.0` | `2.0.0` ✏️ | **`2.0.0`** (v2's bump) | - **`name`**: Only `main` changed it (`app` → `MyApp`), so we keep `MyApp`. - **`version`**: Only the incoming commit changed it (`1.0.0` → `2.0.0`), so we keep `2.0.0`. Each field was modified by only **one side**, so the correct merge takes both changes.
Failure: Content mismatch: ['settings.json']
anthropic/claude-opus-4.8:medium FAIL 0% 227 in → 145 out (38 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` The resolution combines both intentional changes: the rename to `"MyApp"` from the current branch (main) and the version bump to `"2.0.0"` from the incoming v2 branch.
Failure: Content mismatch: ['settings.json']
anthropic/claude-opus-4.8:none FAIL 0% 228 in → 89 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` The resolution combines both intentional changes: `main`'s rename of `name` to `"MyApp"` and the v2 branch's version bump to `"2.0.0"`.
Failure: Content mismatch: ['settings.json']
anthropic/claude-sonnet-4.6:high FAIL 0% 181 in → 310 out (79 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` **Resolution logic:** | Field | Base | Main (HEAD) | v2 Branch | Resolved | |-------|------|-------------|-----------|----------| | `name` | `"app"` | `"MyApp"` ✏️ | `"app"` | `"MyApp"` | | `version` | `"1.0.0"` | `"1.0.0"` | `"2.0.0"` ✏️ | `"2.0.0"` | Each branch made an **independent, non-overlapping change** — main renamed `name`, v2 bumped `version` — so both changes are applied in the resolved output.
Failure: Content mismatch: ['settings.json']
anthropic/claude-sonnet-4.6:max FAIL 0% 180 in → 500 out (181 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` **Resolution logic:** | Field | Base | HEAD (main) | Incoming (v2) | Resolution | |-------|------|-------------|---------------|------------| | `name` | `"app"` | `"MyApp"` ✏️ | `"app"` (unchanged) | `"MyApp"` — main made an intentional rename; incoming didn't touch it | | `version` | `"1.0.0"` | `"1.0.0"` (unchanged) | `"2.0.0"` ✏️ | `"2.0.0"` — this is exactly the change being cherry-picked | Both changes are independent (different keys), so both are accepted: the rename from main and the version bump from the cherry-picked commit.
Failure: Content mismatch: ['settings.json']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high__json_schema FAIL 0% 159 in → 444 out (426 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
Failure: Content mismatch: ['settings.json']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 159 in → 506 out (506 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
arcee-ai/trinity-mini:xhigh FAIL 0% 160 in → 382 out (363 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 158 in → 469 out (467 reasoning)
MyApp
JSON Schema Structured Output
(raw) { "resolved_content": "MyApp" }
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
deepseek/deepseek-v4-flash:xhigh FAIL 0% 240 in → 404 out (390 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
google/gemini-3-flash-preview:high__json_schema FAIL 0% 165 in → 1,120 out (1,097 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content":"{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3-flash-preview:low__json_schema FAIL 0% 224 in → 1,477 out (1,454 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content":"{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3-flash-preview:medium__json_schema FAIL 0% 163 in → 596 out (573 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content":"{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3-flash-preview:xhigh__json_schema FAIL 0% 223 in → 982 out (959 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content":"{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 165 in → 637 out (609 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-flash-lite-preview:xhigh__json_schema FAIL 0% 165 in → 1,211 out (1,187 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-pro-preview:high__json_schema FAIL 0% 225 in → 704 out (684 reasoning)
{"name":"MyApp","version":"2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content":"{\"name\":\"MyApp\",\"version\":\"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 225 in → 400 out (376 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-pro-preview:medium__json_schema FAIL 0% 223 in → 1,040 out (1,016 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-pro-preview:xhigh__json_schema FAIL 0% 225 in → 872 out (848 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.5-flash-lite:high__json_schema FAIL 0% 225 in → 1,007 out (983 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.5-flash-lite:max__json_schema FAIL 0% 225 in → 1,736 out (1,712 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.5-flash-lite:medium__json_schema FAIL 0% 224 in → 478 out (450 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
google/gemini-3.5-flash-lite:xhigh__json_schema FAIL 0% 225 in → 1,507 out (1,483 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemma-4-26b-a4b-it:high__json_schema FAIL 0% 180 in → 442 out (362 reasoning)
not needed, providing JSON directly
JSON Schema Structured Output
(raw) { "resolved_content": "not needed, providing JSON directly" }
Failure: Content mismatch: ['settings.json']
google/gemma-4-26b-a4b-it:low__json_schema FAIL 0% 180 in → 469 out (342 reasoning)
to be replaced by actual JSON below
JSON Schema Structured Output
(raw) { "resolved_content": "to be replaced by actual JSON below" }
Failure: Content mismatch: ['settings.json']
google/gemma-4-26b-a4b-it:medium__json_schema FAIL 0% 178 in → 588 out (459 reasoning)
not applicable, providing JSON
JSON Schema Structured Output
(raw) { "resolved_content": "not applicable, providing JSON" }
Failure: Content mismatch: ['settings.json']
google/gemma-4-26b-a4b-it:none__json_schema FAIL 0% 175 in → 29 out (0 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
google/gemma-4-26b-a4b-it:xhigh__json_schema FAIL 0% 180 in → 476 out (350 reasoning)
...
JSON Schema Structured Output
(raw) { "resolved_content": "..." }
Failure: Content mismatch: ['settings.json']
google/gemma-4-31b-it:high__json_schema FAIL 0% 181 in → 241 out (165 reasoning)
invalid_key_ignore_just_provide_json
JSON Schema Structured Output
(raw) { "resolved_content": "invalid_key_ignore_just_provide_json" }
Failure: Content mismatch: ['settings.json']
google/gemma-4-31b-it:low__json_schema FAIL 0% 184 in → 225 out (217 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"resolved_content":""}
Failure: Content mismatch: ['settings.json']
google/gemma-4-31b-it:medium__json_schema FAIL 0% 180 in → 225 out (154 reasoning)
Not applicable, providing JSON directly below
JSON Schema Structured Output
(raw) { "resolved_content": "Not applicable, providing JSON directly below" }
Failure: Content mismatch: ['settings.json']
google/gemma-4-31b-it:none__json_schema FAIL 0% 176 in → 16 out (0 reasoning)
not_applicable
JSON Schema Structured Output
(raw) { "resolved_content": "not_applicable" }
Failure: Content mismatch: ['settings.json']
"name": "MyApp", "version": "2.0.0"
JSON Schema Structured Output
(raw) { "resolved_content": " \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n" }
Failure: Content mismatch: ['settings.json']
liquid/lfm-2-24b-a2b FAIL 0% 172 in → 19 out
{"name": "MyApp", "version": "2.0.0"}
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: {"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
minimax/minimax-m2.5:high__json_schema FAIL 0% 192 in → 284 out (270 reasoning)
MyApp
JSON Schema Structured Output
(raw) { "resolved_content": "MyApp" }
Failure: Content mismatch: ['settings.json']
minimax/minimax-m2.5:low__json_schema FAIL 0% 208 in → 317 out (301 reasoning)
settings.json
JSON Schema Structured Output
(raw) { "resolved_content": "settings.json" }
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
minimax/minimax-m2.7:low FAIL 0% 193 in → 222 out (198 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high FAIL 0% 278 in → 207 out (191 reasoning)
{"name":"MyApp","version":"2.0.0"}
Failure: Content mismatch: ['settings.json']
minimax/minimax-m3:medium FAIL 0% 330 in → 236 out (204 reasoning)
{"name":"MyApp","version":"2.0.0"}
Failure: Content mismatch: ['settings.json']
moonshotai/kimi-k2.6:high FAIL 0% 160 in → 374 out (351 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
moonshotai/kimi-k2.6:low FAIL 0% 162 in → 602 out (582 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
moonshotai/kimi-k2.7-code:medium FAIL 0% 160 in → 221 out (209 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
moonshotai/kimi-k2.7-code:xhigh FAIL 0% 160 in → 290 out (283 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
(empty output)
Failure: OpenAI-compatible response contained provider error: Provider returned error
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema FAIL 0% 178 in → 523 out (457 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 176 in → 30 out (0 reasoning)
{"name": "MyApp","version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\",\"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
nvidia/nemotron-3-super-120b-a12b:xhigh FAIL 0% 177 in → 256 out (213 reasoning)
{"name":"MyApp","version":"2.0.0"}
Failure: Content mismatch: ['settings.json']
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema FAIL 0% 177 in → 197 out (159 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }" }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-120b:high__json_schema FAIL 0% 218 in → 419 out (371 reasoning)
Conflicts manually resolved. The name follows the main branch change, and the version follows the v2 branch change.
JSON Schema Structured Output
(raw) { "resolved_content": "Conflicts manually resolved. The name follows the main branch change, and the version follows the v2 branch change." }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-120b:low__json_schema FAIL 0% 206 in → 110 out (68 reasoning)
settings.json has been merged successfully.
JSON Schema Structured Output
(raw) { "resolved_content": "settings.json has been merged successfully." }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-120b:medium__json_schema FAIL 0% 220 in → 224 out (183 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-120b:xhigh__json_schema FAIL 0% 217 in → 514 out (467 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": " {\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-20b:high__json_schema FAIL 0% 215 in → 657 out (641 reasoning)
name: MyApp, version: 2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "name: MyApp, version: 2.0.0" }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-20b:low__json_schema FAIL 0% 205 in → 100 out (64 reasoning)
MyApp
JSON Schema Structured Output
(raw) { "resolved_content": "MyApp" }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-20b:medium__json_schema FAIL 0% 217 in → 169 out (121 reasoning)
The conflicts resolved by adopting the name change from the current branch and the version bump from the incoming branch.
JSON Schema Structured Output
(raw) { "resolved_content": "The conflicts resolved by adopting the name change from the current branch and the version bump from the incoming branch." }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-20b:xhigh__json_schema FAIL 0% 216 in → 602 out (592 reasoning)
Your resolved settings.json content here
JSON Schema Structured Output
(raw) { "resolved_content": "Your resolved settings.json content here" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-m.1:high__json_schema FAIL 0% 174 in → 359 out (333 reasoning)
{"name":"MyApp","version":"2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\":\"MyApp\",\"version\":\"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-m.1:none__json_schema FAIL 0% 172 in → 23 out (0 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:high__json_schema FAIL 0% 171 in → 651 out (623 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:low__json_schema FAIL 0% 174 in → 528 out (498 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:medium__json_schema FAIL 0% 173 in → 610 out (582 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 174 in → 27 out (0 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:xhigh__json_schema FAIL 0% 173 in → 354 out (326 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs.2:none__json_schema FAIL 0% 210 in → 24 out (0 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
qwen/qwen3.6-27b:low__json_schema FAIL 0% 169 in → 1,195 out (1,071 reasoning)
Providing ONLY the resolved settings.json content as valid JSON.
JSON Schema Structured Output
(raw) { "resolved_content": "Providing ONLY the resolved settings.json content as valid JSON." }
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: {"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: {"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: {"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
(empty output)
JSON Schema Structured Output
Failure: Model 'tencent/hy3:none' violated the reasoning_level=none invariant: provider reported 654 reasoning token(s)
thinkingmachines/inkling-small:high__json_schema FAIL 0% 166 in → 226 out (149 reasoning)
Wait, they ask provide ONLY resolved settings.json content as valid. So just the object, not wrapped. Let's output exactly: {"name":"MyApp","version":"2.0.0"} Or pretty printed? Usually valid can be pretty. Provide content.
JSON Schema Structured Output
(raw) { "resolved_content": "Wait, they ask provide ONLY resolved settings.json content as valid. So just the object, not wrapped.\n\nLet's output exactly:\n{\"name\":\"MyApp\",\"version\":\"2.0.0\"}\nOr pretty printed? Usually valid can be pretty. Provide content.\n" }
Failure: Content mismatch: ['settings.json']
thinkingmachines/inkling-small:max__json_schema FAIL 0% 167 in → 155 out (132 reasoning)
OH wait need just file content
JSON Schema Structured Output
(raw) { "resolved_content": "OH wait need just file content" }
Failure: Content mismatch: ['settings.json']
Invalid JSON. Output: { "name": "MyApp", "version": "2.0.0" }{ "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 4 column 2 (char 43)
Failure: Failed to parse structured JSON response: Extra data: line 4 column 2 (char 43)
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
thinkingmachines/inkling:medium__json_schema FAIL 0% 165 in → 156 out (123 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
thinkingmachines/inkling:none FAIL 0% 166 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }<|end_message|>
Failure: Content mismatch: ['settings.json']
thinkingmachines/inkling:none__json_schema FAIL 0% 165 in → 14 out (0 reasoning)
{
JSON Schema Structured Output
(raw) { "resolved_content": "{" }
Failure: Content mismatch: ['settings.json']
z-ai/glm-4.7-flash:high__json_schema FAIL 0% 156 in → 1,257 out (1,120 reasoning)
MyApp and 2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "MyApp and 2.0.0" }
Failure: Content mismatch: ['settings.json']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0% 156 in → 1,303 out (1,240 reasoning)
This is a placeholder, I need to output the JSON directly.
JSON Schema Structured Output
(raw) { "resolved_content": "This is a placeholder, I need to output the JSON directly." }
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'