Multi-file: conflict in settings.json only
Tests ability to resolve a JSON config merge conflict preserving valid JSON syntax. Evaluates structured-data conflict resolution.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf '{ "name": "app", "version": "1.0.0" }' > settings.json
  5. 05 printf 'DB_HOST=localhost DB_PORT=5432' > database.ini
  6. 06 git add settings.json database.ini
  7. 07 git commit -m 'Initial config'
  8. 08 git checkout -b v2
  9. 09 printf '{ "name": "app", "version": "2.0.0" }' > settings.json
  10. 10 git add settings.json
  11. 11 git commit -m 'Bump version to 2.0.0'
  12. 12 git checkout main
  13. 13 printf '{ "name": "MyApp", "version": "1.0.0" }' > settings.json
  14. 14 git add settings.json
  15. 15 git commit -m 'Rename app to MyApp'
  16. 16 git merge v2
Prompt
Resolve the merge conflict in settings.json. The base had name: app, version: 1.0.0. Branch A (v2) kept name: app but bumped version to 2.0.0. Branch B (main) renamed name to MyApp but kept version 1.0.0. Provide ONLY the resolved settings.json content as valid JSON.
Expected
{ "name": "MyApp", "version": "2.0.0" }

Scoped model quality, cost, API time, and token usage for merge_conflicts/f011.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high__json_schema PASS 100% 422 in → 75 out (17 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-fable-5:low PASS 100% 202 in → 28 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-fable-5:low__json_schema PASS 100% 422 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-fable-5:max PASS 100% 202 in → 303 out (87 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-fable-5:max__json_schema PASS 100% 422 in → 379 out (58 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-fable-5:medium__json_schema PASS 100% 422 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-fable-5:xhigh PASS 100% 202 in → 75 out (29 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 422 in → 77 out (22 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-haiku-4.5:low PASS 100% 188 in → 316 out (244 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 356 in → 665 out (485 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-haiku-4.5:none PASS 100% 159 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-haiku-4.5:xhigh PASS 100% 188 in → 375 out (295 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 356 in → 431 out (328 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:high__json_schema PASS 100% 327 in → 81 out (24 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:low PASS 100% 159 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 327 in → 38 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 327 in → 229 out (140 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:medium PASS 100% 159 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 327 in → 38 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:none PASS 100% 159 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 327 in → 38 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 327 in → 78 out (19 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.7:high PASS 100% 207 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:high__json_schema PASS 100% 427 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.7:low PASS 100% 207 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:low__json_schema PASS 100% 427 in → 54 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-opus-4.7:max PASS 100% 207 in → 277 out (65 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:max__json_schema PASS 100% 427 in → 266 out (72 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-opus-4.7:medium PASS 100% 207 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 427 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.7:none PASS 100% 207 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:none__json_schema PASS 100% 427 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.7:xhigh PASS 100% 207 in → 30 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 427 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.8:high PASS 100% 202 in → 81 out (41 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 422 in → 99 out (44 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.8:low PASS 100% 202 in → 73 out (35 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 422 in → 50 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.8:max PASS 100% 202 in → 399 out (129 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
anthropic/claude-opus-4.8:max__json_schema PASS 100% 422 in → 299 out (55 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.8:medium PASS 100% 202 in → 98 out (55 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 422 in → 50 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.8:none PASS 100% 202 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 422 in → 49 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-opus-4.8:xhigh PASS 100% 202 in → 283 out (60 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 422 in → 143 out (57 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 327 in → 129 out (64 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-4.6:low PASS 100% 159 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 327 in → 39 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 327 in → 241 out (132 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-4.6:medium PASS 100% 159 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 327 in → 39 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-4.6:none PASS 100% 159 in → 32 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 327 in → 39 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 327 in → 129 out (64 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-5:high PASS 100% 202 in → 96 out (59 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 422 in → 83 out (28 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
anthropic/claude-sonnet-5:low PASS 100% 202 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 422 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-sonnet-5:medium PASS 100% 202 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 422 in → 52 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-sonnet-5:none PASS 100% 202 in → 33 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 422 in → 51 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 202 in → 125 out (57 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 422 in → 88 out (31 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 142 in → 655 out (632 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-large-thinking:low PASS 100% 142 in → 567 out (544 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-large-thinking:medium PASS 100% 142 in → 654 out (630 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-large-thinking:xhigh PASS 100% 142 in → 949 out (927 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-mini:high__json_schema PASS 100% 142 in → 621 out (600 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
arcee-ai/trinity-mini:low PASS 100% 142 in → 586 out (571 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-mini:xhigh PASS 100% 142 in → 956 out (928 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 142 in → 1,152 out (1,086 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
deepseek/deepseek-v4-flash-0731:high PASS 100% 221 in → 309 out (268 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 142 in → 296 out (246 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 142 in → 217 out (160 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 142 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 234 in → 294 out (272 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 221 in → 163 out (115 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 142 in → 236 out (214 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 142 in → 283 out (219 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 148 in → 22 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 148 in → 36 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 221 in → 170 out (148 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash:high PASS 100% 142 in → 132 out (93 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash:low PASS 100% 142 in → 185 out (162 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash:medium PASS 100% 142 in → 183 out (134 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-flash:none PASS 100% 142 in → 26 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 144 in → 38 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 221 in → 345 out (322 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-pro:high PASS 100% 142 in → 411 out (388 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 144 in → 238 out (199 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-pro:low PASS 100% 142 in → 310 out (287 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-pro:medium PASS 100% 142 in → 332 out (309 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 334 in → 520 out (485 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
deepseek/deepseek-v4-pro:none PASS 100% 142 in → 26 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 142 in → 36 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 221 in → 601 out (578 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 223 in → 328 out (290 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemini-3-flash-preview:high PASS 100% 143 in → 359 out (336 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3-flash-preview:high__json_schema PASS 100% 144 in → 463 out (430 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemini-3-flash-preview:low PASS 100% 143 in → 280 out (257 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3-flash-preview:medium PASS 100% 143 in → 294 out (271 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3-flash-preview:xhigh PASS 100% 143 in → 293 out (270 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-flash-lite-preview:high PASS 100% 144 in → 470 out (442 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemini-3.1-flash-lite-preview:low PASS 100% 144 in → 169 out (146 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 144 in → 214 out (179 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 143 in → 225 out (202 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 143 in → 267 out (244 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-pro-preview:high PASS 100% 143 in → 641 out (618 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-pro-preview:low PASS 100% 143 in → 322 out (299 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-pro-preview:medium PASS 100% 143 in → 317 out (294 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.1-pro-preview:xhigh PASS 100% 143 in → 424 out (401 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash-lite:high PASS 100% 144 in → 722 out (699 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 144 in → 1,521 out (1,491 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.5-flash-lite:low PASS 100% 143 in → 23 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 203 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemini-3.5-flash-lite:max PASS 100% 144 in → 379 out (356 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash-lite:medium PASS 100% 143 in → 353 out (330 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 144 in → 875 out (852 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash:high PASS 100% 143 in → 716 out (693 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash:high__json_schema PASS 100% 203 in → 742 out (712 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.5-flash:low PASS 100% 143 in → 198 out (175 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash:low__json_schema PASS 100% 203 in → 243 out (208 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemini-3.5-flash:medium PASS 100% 143 in → 521 out (498 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash:medium__json_schema PASS 100% 203 in → 883 out (848 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemini-3.5-flash:xhigh PASS 100% 143 in → 505 out (482 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 203 in → 1,065 out (1,034 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.6-flash:high PASS 100% 143 in → 613 out (590 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.6-flash:high__json_schema PASS 100% 203 in → 891 out (860 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.6-flash:low PASS 100% 144 in → 342 out (319 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.6-flash:low__json_schema PASS 100% 203 in → 555 out (525 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.6-flash:max PASS 100% 143 in → 417 out (394 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.6-flash:max__json_schema PASS 100% 144 in → 1,867 out (1,837 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemini-3.6-flash:medium PASS 100% 144 in → 509 out (486 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.6-flash:medium__json_schema PASS 100% 203 in → 765 out (730 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemini-3.6-flash:xhigh PASS 100% 143 in → 500 out (477 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 144 in → 1,099 out (1,069 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemma-4-26b-a4b-it:high PASS 100% 159 in → 771 out (599 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-26b-a4b-it:low PASS 100% 159 in → 345 out (262 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-26b-a4b-it:medium PASS 100% 159 in → 512 out (458 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-26b-a4b-it:none PASS 100% 156 in → 29 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-26b-a4b-it:xhigh PASS 100% 159 in → 284 out (255 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 158 in → 312 out (212 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
google/gemma-4-31b-it:high PASS 100% 159 in → 226 out (148 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-31b-it:low PASS 100% 158 in → 273 out (166 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-31b-it:medium PASS 100% 159 in → 225 out (147 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-31b-it:none PASS 100% 156 in → 29 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
google/gemma-4-31b-it:none__json_schema PASS 100% 156 in → 32 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
google/gemma-4-31b-it:xhigh PASS 100% 159 in → 255 out (180 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
ibm-granite/granite-4.1-8b PASS 100% 151 in → 22 out
{ "name": "MyApp", "version": "2.0.0" }
ibm-granite/granite-4.1-8b__json_schema PASS 100% 151 in → 36 out
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}\n" }
inclusionai/ling-2.6-flash PASS 100% 163 in → 25 out
{ "name": "MyApp", "version": "2.0.0" }
inclusionai/ling-2.6-flash__json_schema PASS 100% 163 in → 37 out
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m2.5:high__json_schema PASS 100% 173 in → 396 out (332 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m2.5:low PASS 100% 173 in → 549 out (500 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.5:medium PASS 100% 173 in → 395 out (347 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.5:medium__json_schema PASS 100% 186 in → 429 out (339 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m2.5:xhigh PASS 100% 173 in → 637 out (532 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 186 in → 506 out (470 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m2.7:high PASS 100% 176 in → 380 out (358 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.7:low PASS 100% 176 in → 326 out (304 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.7:medium PASS 100% 176 in → 510 out (488 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m2.7:xhigh PASS 100% 173 in → 309 out (303 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m3:high PASS 100% 311 in → 234 out (189 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
minimax/minimax-m3:high__json_schema PASS 100% 298 in → 32 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
minimax/minimax-m3:low PASS 100% 311 in → 206 out (145 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
minimax/minimax-m3:low__json_schema PASS 100% 298 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m3:medium PASS 100% 311 in → 275 out (223 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
minimax/minimax-m3:medium__json_schema PASS 100% 298 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
minimax/minimax-m3:xhigh PASS 100% 308 in → 234 out (182 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
minimax/minimax-m3:xhigh__json_schema PASS 100% 298 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
mistralai/devstral-2512 PASS 100% 143 in → 26 out
```json { "name": "MyApp", "version": "2.0.0" } ```
mistralai/devstral-2512__json_schema PASS 100% 143 in → 33 out
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
mistralai/mistral-medium-3-5:high PASS 100% 155 in → 492 out (449 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 143 in → 451 out (404 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
mistralai/mistral-medium-3-5:low PASS 100% 155 in → 516 out (485 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 143 in → 510 out (422 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
mistralai/mistral-medium-3-5:medium PASS 100% 155 in → 338 out (306 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 143 in → 347 out (300 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
mistralai/mistral-medium-3-5:none PASS 100% 155 in → 26 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 143 in → 33 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 155 in → 352 out (301 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 143 in → 283 out (214 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
moonshotai/kimi-k2.6:high PASS 100% 142 in → 236 out (209 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k2.6:high__json_schema PASS 100% 141 in → 306 out (272 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
moonshotai/kimi-k2.6:low__json_schema PASS 100% 142 in → 33 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 142 in → 654 out (620 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
moonshotai/kimi-k2.6:xhigh PASS 100% 142 in → 555 out (532 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 142 in → 33 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
moonshotai/kimi-k2.7-code:high PASS 100% 142 in → 655 out (628 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 142 in → 224 out (178 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
moonshotai/kimi-k2.7-code:low PASS 100% 142 in → 195 out (172 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
moonshotai/kimi-k2.7-code:medium PASS 100% 142 in → 283 out (256 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 142 in → 166 out (130 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 142 in → 251 out (224 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:high PASS 100% 219 in → 223 out (183 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:high__json_schema PASS 100% 219 in → 204 out (164 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:low PASS 100% 219 in → 207 out (167 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:low__json_schema PASS 100% 219 in → 85 out (49 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
moonshotai/kimi-k3:max PASS 100% 219 in → 221 out (185 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
moonshotai/kimi-k3:max__json_schema PASS 100% 219 in → 200 out (160 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:medium PASS 100% 219 in → 262 out (222 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:medium__json_schema PASS 100% 219 in → 73 out (37 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
moonshotai/kimi-k3:xhigh PASS 100% 219 in → 161 out (121 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 219 in → 288 out (252 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 156 in → 811 out (727 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 156 in → 954 out (797 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 156 in → 461 out (413 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 156 in → 648 out (531 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 156 in → 608 out (545 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 156 in → 549 out (597 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 156 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 156 in → 575 out (542 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 156 in → 166 out (121 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 156 in → 315 out (276 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 156 in → 262 out (222 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 156 in → 217 out (178 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 156 in → 215 out (175 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 156 in → 22 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 156 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 156 in → 209 out (154 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 156 in → 312 out (271 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
openai/gpt-5.4-mini:high PASS 100% 138 in → 276 out (249 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-mini:high__json_schema PASS 100% 174 in → 241 out (202 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-mini:low PASS 100% 138 in → 104 out (77 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-mini:low__json_schema PASS 100% 174 in → 57 out (18 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-mini:medium PASS 100% 138 in → 129 out (102 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-mini:medium__json_schema PASS 100% 174 in → 125 out (86 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-mini:none PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-mini:none__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-mini:xhigh PASS 100% 138 in → 403 out (376 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 174 in → 438 out (399 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-nano:high PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-nano:high__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-nano:low PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-nano:low__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-nano:medium PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-nano:medium__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4-nano:none PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-nano:xhigh PASS 100% 138 in → 375 out (348 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 174 in → 1,943 out (1,904 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4:high PASS 100% 138 in → 113 out (86 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4:high__json_schema PASS 100% 174 in → 451 out (412 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4:low PASS 100% 138 in → 51 out (24 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4:low__json_schema PASS 100% 174 in → 65 out (26 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4:medium PASS 100% 138 in → 52 out (25 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4:medium__json_schema PASS 100% 174 in → 121 out (82 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4:none PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4:none__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.4:xhigh PASS 100% 138 in → 133 out (106 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.4:xhigh__json_schema PASS 100% 174 in → 163 out (124 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.5:high PASS 100% 138 in → 63 out (36 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.5:high__json_schema PASS 100% 174 in → 259 out (220 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.5:low PASS 100% 138 in → 52 out (25 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.5:low__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.5:medium PASS 100% 138 in → 53 out (26 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.5:medium__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.5:none PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.5:none__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.5:xhigh PASS 100% 138 in → 106 out (79 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.5:xhigh__json_schema PASS 100% 174 in → 132 out (93 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:high PASS 100% 138 in → 57 out (30 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:high__json_schema PASS 100% 174 in → 68 out (29 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:low PASS 100% 138 in → 53 out (26 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:low__json_schema PASS 100% 174 in → 65 out (26 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:max PASS 100% 138 in → 119 out (92 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:max__json_schema PASS 100% 174 in → 112 out (73 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:medium PASS 100% 138 in → 57 out (30 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:medium__json_schema PASS 100% 174 in → 65 out (26 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:none PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:none__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-luna:xhigh PASS 100% 138 in → 90 out (63 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 174 in → 75 out (36 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:high PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:high__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:low PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:low__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:max PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:max__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:medium PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:medium__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:none PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:none__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-sol:xhigh PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:high PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:high__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:low PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:low__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:max PASS 100% 138 in → 64 out (37 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:max__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:medium PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:medium__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:none PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:none__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-5.6-terra:xhigh PASS 100% 138 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 174 in → 37 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
openai/gpt-oss-120b:high PASS 100% 186 in → 566 out (567 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-120b:high__json_schema PASS 100% 197 in → 427 out (405 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
openai/gpt-oss-120b:low PASS 100% 199 in → 68 out (37 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-120b:medium PASS 100% 199 in → 217 out (173 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-120b:xhigh PASS 100% 186 in → 532 out (495 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-20b:high PASS 100% 199 in → 312 out (291 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-20b:low PASS 100% 204 in → 91 out (53 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
openai/gpt-oss-20b:medium PASS 100% 204 in → 339 out (275 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
openai/gpt-oss-20b:xhigh PASS 100% 199 in → 413 out (367 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:high PASS 100% 153 in → 843 out (817 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:low PASS 100% 153 in → 511 out (485 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:medium PASS 100% 153 in → 551 out (526 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:medium__json_schema PASS 100% 153 in → 1,746 out (1,711 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
poolside/laguna-m.1:none PASS 100% 153 in → 25 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:xhigh PASS 100% 153 in → 1,065 out (1,039 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-m.1:xhigh__json_schema PASS 100% 153 in → 341 out (306 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
poolside/laguna-xs-2.1:high PASS 100% 153 in → 351 out (327 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs-2.1:low PASS 100% 153 in → 411 out (387 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs-2.1:medium PASS 100% 153 in → 559 out (535 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs-2.1:xhigh PASS 100% 153 in → 571 out (547 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs.2:high PASS 100% 190 in → 306 out (280 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs.2:high__json_schema PASS 100% 190 in → 287 out (256 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
poolside/laguna-xs.2:low PASS 100% 190 in → 257 out (231 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs.2:medium PASS 100% 190 in → 264 out (238 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
poolside/laguna-xs.2:none PASS 100% 190 in → 30 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
poolside/laguna-xs.2:xhigh PASS 100% 190 in → 319 out (288 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 190 in → 309 out (278 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
qwen/qwen3.6-27b:high PASS 100% 149 in → 907 out (839 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-27b:high__json_schema PASS 100% 149 in → 1,103 out (994 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-27b:low PASS 100% 149 in → 351 out (253 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-27b:medium PASS 100% 149 in → 2,850 out (2,616 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-27b:none PASS 100% 151 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-27b:xhigh PASS 100% 149 in → 1,086 out (979 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 149 in → 873 out (746 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-35b-a3b:high PASS 100% 149 in → 211 out (169 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 149 in → 637 out (505 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-35b-a3b:low PASS 100% 149 in → 470 out (371 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 149 in → 1,187 out (1,041 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 149 in → 1,161 out (1,132 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 149 in → 955 out (821 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-35b-a3b:none PASS 100% 151 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 149 in → 868 out (839 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 149 in → 220 out (141 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
qwen/qwen3.6-flash:high PASS 100% 149 in → 1,070 out (1,041 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-flash:low PASS 100% 149 in → 1,332 out (1,303 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-flash:medium PASS 100% 149 in → 1,224 out (1,195 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-flash:none PASS 100% 151 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.6-flash:xhigh PASS 100% 149 in → 367 out (338 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:high PASS 100% 149 in → 1,164 out (1,135 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:low PASS 100% 149 in → 463 out (434 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:max PASS 100% 149 in → 1,022 out (993 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:medium PASS 100% 149 in → 1,122 out (1,093 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:none PASS 100% 151 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-flash:xhigh PASS 100% 149 in → 784 out (755 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-max:high PASS 100% 149 in → 213 out (184 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-max:low PASS 100% 149 in → 446 out (417 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-max:medium PASS 100% 149 in → 188 out (159 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-max:none PASS 100% 151 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-max:xhigh PASS 100% 149 in → 185 out (156 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-plus:high PASS 100% 149 in → 699 out (670 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-plus:low PASS 100% 149 in → 237 out (203 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
qwen/qwen3.7-plus:medium PASS 100% 149 in → 578 out (549 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-plus:none PASS 100% 151 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
qwen/qwen3.7-plus:xhigh PASS 100% 149 in → 838 out (809 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:high PASS 100% 149 in → 294 out (270 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:low PASS 100% 149 in → 335 out (311 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:medium PASS 100% 149 in → 558 out (535 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:none PASS 100% 152 in → 23 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:none__json_schema PASS 100% 152 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
tencent/hy3:xhigh PASS 100% 149 in → 469 out (445 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
tencent/hy3:xhigh__json_schema PASS 100% 149 in → 410 out (322 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
thinkingmachines/inkling-small:high PASS 100% 147 in → 305 out (235 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling-small:high__json_schema PASS 100% 147 in → 257 out (216 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
thinkingmachines/inkling-small:low PASS 100% 147 in → 126 out (99 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling-small:max PASS 100% 147 in → 258 out (231 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling-small:max__json_schema PASS 100% 147 in → 242 out (203 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
thinkingmachines/inkling-small:medium PASS 100% 147 in → 155 out (116 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling-small:medium__json_schema PASS 100% 147 in → 180 out (141 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
thinkingmachines/inkling-small:xhigh PASS 100% 147 in → 255 out (214 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:high PASS 100% 147 in → 259 out (231 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:low PASS 100% 147 in → 50 out (22 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:max PASS 100% 147 in → 489 out (461 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:medium PASS 100% 147 in → 145 out (102 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:none PASS 100% 145 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:none__json_schema PASS 100% 145 in → 35 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
thinkingmachines/inkling:xhigh PASS 100% 147 in → 254 out (226 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
thinkingmachines/inkling:xhigh__json_schema PASS 100% 147 in → 237 out (198 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.3:high PASS 100% 317 in → 556 out (535 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:high__json_schema PASS 100% 374 in → 1,145 out (1,113 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.3:low PASS 100% 323 in → 277 out (256 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:low__json_schema PASS 100% 380 in → 521 out (489 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.3:max PASS 100% 317 in → 640 out (619 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:max__json_schema PASS 100% 374 in → 1,098 out (1,069 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
x-ai/grok-4.3:medium PASS 100% 323 in → 484 out (463 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:medium__json_schema PASS 100% 380 in → 794 out (762 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.3:none PASS 100% 315 in → 25 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
x-ai/grok-4.3:none__json_schema PASS 100% 372 in → 32 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.3:xhigh PASS 100% 317 in → 503 out (482 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.3:xhigh__json_schema PASS 100% 374 in → 957 out (925 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.5:high PASS 100% 341 in → 299 out (277 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.5:high__json_schema PASS 100% 410 in → 526 out (491 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.5:low PASS 100% 341 in → 173 out (151 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
x-ai/grok-4.5:low__json_schema PASS 100% 410 in → 311 out (276 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.5:max PASS 100% 341 in → 292 out (266 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
x-ai/grok-4.5:max__json_schema PASS 100% 410 in → 500 out (465 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.5:medium PASS 100% 341 in → 230 out (204 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
x-ai/grok-4.5:medium__json_schema PASS 100% 410 in → 527 out (492 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
x-ai/grok-4.5:xhigh PASS 100% 341 in → 252 out (226 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 410 in → 432 out (397 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
z-ai/glm-4.7-flash:high PASS 100% 136 in → 532 out (431 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
z-ai/glm-4.7-flash:low PASS 100% 136 in → 547 out (448 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
z-ai/glm-4.7-flash:xhigh PASS 100% 136 in → 2,242 out (2,047 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ```
z-ai/glm-5.2:high PASS 100% 143 in → 159 out (136 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:high__json_schema PASS 100% 145 in → 169 out (132 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
z-ai/glm-5.2:low PASS 100% 143 in → 181 out (143 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:medium PASS 100% 143 in → 486 out (463 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:medium__json_schema PASS 100% 136 in → 431 out (333 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
z-ai/glm-5.2:none PASS 100% 137 in → 22 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:none__json_schema PASS 100% 137 in → 36 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
z-ai/glm-5.2:xhigh PASS 100% 143 in → 206 out (183 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
z-ai/glm-5.2:xhigh__json_schema PASS 100% 136 in → 452 out (362 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
anthropic/claude-fable-5:high FAIL 0% 202 in → 117 out (17 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` This resolution combines both intentional changes: Branch B's rename to `MyApp` and Branch A's version bump to `2.0.0`, since each branch modified a different field relative to the base.
Failure: Content mismatch: ['settings.json']
anthropic/claude-fable-5:medium FAIL 0% 202 in → 79 out (0 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` Each branch made a distinct change relative to the base (main renamed the app, v2 bumped the version), so the correct resolution combines both changes.
Failure: Content mismatch: ['settings.json']
anthropic/claude-haiku-4.5:high FAIL 0% 188 in → 486 out (315 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` **Resolution rationale:** - Keep `name: "MyApp"` from main (intentional rename) - Keep `version: "2.0.0"` from v2 (version bump) - These are independent, non-conflicting changes that should both be applied
Failure: Content mismatch: ['settings.json']
anthropic/claude-haiku-4.5:high__json_schema FAIL 0% 356 in → 893 out (510 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
anthropic/claude-haiku-4.5:low__json_schema FAIL 0% 356 in → 437 out (337 reasoning)
{"name":"MyApp","version":"2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content":"{\"name\":\"MyApp\",\"version\":\"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
anthropic/claude-haiku-4.5:medium FAIL 0% 188 in → 568 out (429 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` This resolution combines both branches' changes: the name update from main and the version bump from v2, since they modify different fields and represent complementary updates.
Failure: Content mismatch: ['settings.json']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 326 in → 31 out (0 reasoning)
{"name": "MyApp","version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\",\"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
anthropic/claude-opus-4.6:high FAIL 0% 159 in → 190 out (31 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` This takes **both** non-conflicting changes: - **`"name": "MyApp"`** — from Branch B (main), which intentionally renamed `app` → `MyApp`. - **`"version": "2.0.0"`** — from Branch A (v2), which intentionally bumped `1.0.0` → `2.0.0`. Each branch modified a *different* field relative to the base, so the correct resolution is to accept both updates.
Failure: Content mismatch: ['settings.json']
anthropic/claude-opus-4.6:max FAIL 0% 159 in → 358 out (126 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` **Reasoning:** Each field's change is taken from the branch that actually modified it relative to the base: - **`name`**: Branch B (HEAD/main) changed `"app"` → `"MyApp"`. Branch A (v2) left it as `"app"`. → Take **`"MyApp"`**. - **`version`**: Branch A (v2) bumped `"1.0.0"` → `"2.0.0"`. Branch B (HEAD/main) left it as `"1.0.0"`. → Take **`"2.0.0"`**. This preserves both intentional changes without data loss.
Failure: Content mismatch: ['settings.json']
anthropic/claude-opus-4.6:xhigh FAIL 0% 159 in → 198 out (31 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` **Reasoning:** Each branch made a distinct, non-overlapping change from the base (`name: app`, `version: 1.0.0`): - **Branch B (main/HEAD):** Changed `name` from `"app"` → `"MyApp"` (keep this) - **Branch A (v2):** Changed `version` from `"1.0.0"` → `"2.0.0"` (keep this) The resolved file incorporates **both** intentional changes.
Failure: Content mismatch: ['settings.json']
anthropic/claude-sonnet-4.6:high FAIL 0% 159 in → 357 out (106 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` **Resolution logic:** | Field | Base | Branch A (v2) | Branch B (main) | Resolution | |-------|------|--------------|----------------|------------| | `name` | `"app"` | `"app"` *(unchanged)* | `"MyApp"` *(modified)* | `"MyApp"` | | `version` | `"1.0.0"` | `"2.0.0"` *(modified)* | `"1.0.0"` *(unchanged)* | `"2.0.0"` | Each branch made a **distinct, non-overlapping change** relative to the base, so both changes are accepted — `name` from main and `version` from v2.
Failure: Content mismatch: ['settings.json']
anthropic/claude-sonnet-4.6:max FAIL 0% 159 in → 455 out (171 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` **Resolution logic:** | Field | Base | Main (HEAD) | v2 | Resolution | |-------|------|-------------|-----|------------| | `name` | `"app"` | `"MyApp"` ✏️ | `"app"` — | Take `"MyApp"` | | `version` | `"1.0.0"` | `"1.0.0"` — | `"2.0.0"` ✏️ | Take `"2.0.0"` | Each branch modified a **different field** independently, so both changes are non-conflicting and can be safely combined — `name` from **main** and `version` from **v2**.
Failure: Content mismatch: ['settings.json']
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 159 in → 338 out (75 reasoning)
```json { "name": "MyApp", "version": "2.0.0" } ``` **Resolution logic:** | Field | Base | Branch A (v2) | Branch B (main) | Resolution | |-------|------|---------------|-----------------|------------| | `name` | `"app"` | `"app"` *(unchanged)* | `"MyApp"` *(changed)* | `"MyApp"` | | `version` | `"1.0.0"` | `"2.0.0"` *(changed)* | `"1.0.0"` *(unchanged)* | `"2.0.0"` | Both changes are independent and non-conflicting when compared against the base — each branch modified a **different field** — so the resolved file accepts **both** changes: the rename from main and the version bump from v2.
Failure: Content mismatch: ['settings.json']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 142 in → 471 out (434 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 142 in → 2,046 out (1,949 reasoning)
Theresolved JSON content after merging the branches. The main branch (HEAD) has name: MyApp and version: 1.0.0, while branch v2 has name: app and version: 2.0.0. Since the user didn't specify which branch to take, the resolved content is based on the main branch's changes to resolve the conflict.
JSON Schema Structured Output
(raw) { "resolved_content": "Theresolved JSON content after merging the branches. The main branch (HEAD) has name: MyApp and version: 1.0.0, while branch v2 has name: app and version: 2.0.0. Since the user didn't specify which branch to take, the resolved content is based on the main branch's changes to resolve the conflict." }
Failure: Content mismatch: ['settings.json']
arcee-ai/trinity-mini:medium FAIL 0% 142 in → 599 out (574 reasoning)
{"name": "MyApp", "version": "2.0.0"}
Failure: Content mismatch: ['settings.json']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 142 in → 471 out (426 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content":"{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 221 in → 182 out (153 reasoning)
settings.json
JSON Schema Structured Output
(raw) { "resolved_content": "settings.json" }
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
google/gemini-3-flash-preview:low__json_schema FAIL 0% 203 in → 990 out (967 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content":"{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3-flash-preview:medium__json_schema FAIL 0% 203 in → 385 out (357 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
google/gemini-3-flash-preview:xhigh__json_schema FAIL 0% 203 in → 1,955 out (1,932 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content":"{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-flash-lite-preview:high__json_schema FAIL 0% 203 in → 1,002 out (978 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 203 in → 383 out (355 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-flash-lite-preview:xhigh__json_schema FAIL 0% 144 in → 949 out (925 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-pro-preview:high__json_schema FAIL 0% 203 in → 534 out (511 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 203 in → 308 out (284 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-pro-preview:medium__json_schema FAIL 0% 203 in → 505 out (481 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content": "{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.1-pro-preview:xhigh__json_schema FAIL 0% 203 in → 809 out (785 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.5-flash-lite:max__json_schema FAIL 0% 144 in → 1,233 out (1,213 reasoning)
{"name":"MyApp","version":"2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content":"{\"name\":\"MyApp\",\"version\":\"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemini-3.5-flash-lite:medium__json_schema FAIL 0% 203 in → 707 out (679 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
google/gemini-3.5-flash-lite:xhigh__json_schema FAIL 0% 203 in → 1,228 out (1,204 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
google/gemma-4-26b-a4b-it:high__json_schema FAIL 0% 159 in → 360 out (258 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
google/gemma-4-26b-a4b-it:low__json_schema FAIL 0% 159 in → 597 out (501 reasoning)
error in thought process, let me re-evaluate
JSON Schema Structured Output
(raw) { "resolved_content": "error in thought process, let me re-evaluate" }
Failure: Content mismatch: ['settings.json']
google/gemma-4-26b-a4b-it:medium__json_schema FAIL 0% 158 in → 565 out (447 reasoning)
...
JSON Schema Structured Output
(raw) { "resolved_content": "..." }
Failure: Content mismatch: ['settings.json']
google/gemma-4-26b-a4b-it:none__json_schema FAIL 0% 156 in → 75 out (0 reasoning)
error: No clear business rule was provided for whether the name change from Branch B or the version bump from Branch A should take precedence. However, following standard semantic merging where feature updates (version) and renaming (branding) are both kept, the common resolution is to apply the new identity and the new version.
JSON Schema Structured Output
(raw) { "resolved_content": "error: No clear business rule was provided for whether the name change from Branch B or the version bump from Branch A should take precedence. However, following standard semantic merging where feature updates (version) and renaming (branding) are both kept, the common resolution is to apply the new identity and the new version." }
Failure: Content mismatch: ['settings.json']
google/gemma-4-31b-it:high__json_schema FAIL 0% 159 in → 249 out (182 reasoning)
No, just the content
JSON Schema Structured Output
(raw) { "resolved_content": "No, just the content" }
Failure: Content mismatch: ['settings.json']
google/gemma-4-31b-it:low__json_schema FAIL 0% 158 in → 232 out (137 reasoning)
Invalid request: Provide ONLY the resolved settings.json content as valid JSON.
JSON Schema Structured Output
(raw) { "resolved_content": "Invalid request: Provide ONLY the resolved settings.json content as valid JSON." }
Failure: Content mismatch: ['settings.json']
google/gemma-4-31b-it:medium__json_schema FAIL 0% 158 in → 287 out (195 reasoning)
Not requested, just the JSON content
JSON Schema Structured Output
(raw) { "resolved_content": "Not requested, just the JSON content" }
Failure: Content mismatch: ['settings.json']
google/gemma-4-31b-it:xhigh__json_schema FAIL 0% 158 in → 279 out (200 reasoning)
settings.json
JSON Schema Structured Output
(raw) { "resolved_content": "settings.json" }
Failure: Content mismatch: ['settings.json']
liquid/lfm-2-24b-a2b FAIL 0% 153 in → 15 out
{"name":"app","version":"2.0.0"}
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: {"name":"app","version":"2.0.0"}
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
minimax/minimax-m2.5:high FAIL 0% 178 in → 519 out (477 reasoning)
"name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
minimax/minimax-m2.5:low__json_schema FAIL 0% 173 in → 349 out (294 reasoning)
MyApp
JSON Schema Structured Output
(raw) { "resolved_content": "MyApp" }
Failure: Content mismatch: ['settings.json']
minimax/minimax-m2.7:high__json_schema FAIL 0% 172 in → 497 out (424 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": "{ \"name\": \"MyApp\", \"version\": \"2.0.0\" }" }
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: {"name":"MyApp","version":"2.0.0"}
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.6:low FAIL 0% 142 in → 997 out (982 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
moonshotai/kimi-k2.6:medium FAIL 0% 142 in → 377 out (312 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
Invalid JSON. Output: ```json { "resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" } ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 156 in → 21 out (0 reasoning)
{ "name": "app", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 0% 156 in → 497 out (441 reasoning)
"name": "MyApp", "version": "2.0.0"
JSON Schema Structured Output
(raw) {"resolved_content":" \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n"}
Failure: Content mismatch: ['settings.json']
nvidia/nemotron-3-super-120b-a12b:low FAIL 0% 156 in → 154 out (136 reasoning)
{"name":"MyApp","version":"2.0.0"}
Failure: Content mismatch: ['settings.json']
openai/gpt-5.4-nano:none__json_schema FAIL 0% 174 in → 36 out (0 reasoning)
{ "name": "app", "version": "2.0.0" }
JSON Schema Structured Output
(raw) {"resolved_content":"{\n \"name\": \"app\",\n \"version\": \"2.0.0\"\n}"}
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-120b:low__json_schema FAIL 0% 186 in → 73 out (32 reasoning)
conflict resolved: name: MyApp, version: 2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "conflict resolved: name: MyApp, version: 2.0.0" }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-120b:medium__json_schema FAIL 0% 186 in → 218 out (171 reasoning)
merged name: MyApp, version: 2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "merged name: MyApp, version: 2.0.0" }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-120b:xhigh__json_schema FAIL 0% 197 in → 423 out (392 reasoning)
{ "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
(raw) { "resolved_content": " {\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}" }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-20b:high__json_schema FAIL 0% 201 in → 577 out (540 reasoning)
This file merges the renaming from main and the version bump from v2, resulting in the name "MyApp" and the updated version "2.0.0".
JSON Schema Structured Output
(raw) { "resolved_content": "This file merges the renaming from main and the version bump from v2, resulting in the name \"MyApp\" and the updated version \"2.0.0\"." }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-20b:low__json_schema FAIL 0% 199 in → 35 out (0 reasoning)
"-"}??????????????????...??? ???……??…………Alright.......…
JSON Schema Structured Output
(raw) { "resolved_content": "\"-\"}??????????????????...??? ???……??…………Alright.......…" }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-20b:medium__json_schema FAIL 0% 186 in → 406 out (355 reasoning)
name: MyApp, version: 2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "name: MyApp, version: 2.0.0" }
Failure: Content mismatch: ['settings.json']
openai/gpt-oss-20b:xhigh__json_schema FAIL 0% 186 in → 444 out (400 reasoning)
The final merged settings.json file retains the updated name from Branch B (MyApp) and the bumped version from Branch A (2.0.0).
JSON Schema Structured Output
(raw) { "resolved_content": "The final merged settings.json file retains the updated name from Branch B (MyApp) and the bumped version from Branch A (2.0.0)." }
Failure: Content mismatch: ['settings.json']
poolside/laguna-m.1:high__json_schema FAIL 0% 153 in → 417 out (393 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
poolside/laguna-m.1:low__json_schema FAIL 0% 153 in → 584 out (555 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-m.1:none__json_schema FAIL 0% 153 in → 24 out (0 reasoning)
{"name": "app", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"app\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:high__json_schema FAIL 0% 153 in → 514 out (486 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:low__json_schema FAIL 0% 153 in → 285 out (256 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:medium__json_schema FAIL 0% 153 in → 527 out (499 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:none FAIL 0% 153 in → 34 out (0 reasoning)
{"name": "app", "version": "2.0.0"} </think>{"name": "app", "version": "2.0.0"}
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:none__json_schema FAIL 0% 153 in → 27 out (0 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs-2.1:xhigh__json_schema FAIL 0% 153 in → 581 out (553 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs.2:low__json_schema FAIL 0% 190 in → 269 out (238 reasoning)
{ "name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) { "resolved_content": "{ \"name\": \"MyApp\", \"version\": \"2.0.0\"}" }
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs.2:medium__json_schema FAIL 0% 190 in → 265 out (241 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
poolside/laguna-xs.2:none__json_schema FAIL 0% 190 in → 24 out (0 reasoning)
{"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
(raw) {"resolved_content": "{\"name\": \"MyApp\", \"version\": \"2.0.0\"}"}
Failure: Content mismatch: ['settings.json']
qwen/qwen3.6-27b:low__json_schema FAIL 0% 151 in → 14 out (0 reasoning)
npm install
JSON Schema Structured Output
(raw) { "resolved_content": "npm install" }
Failure: Content mismatch: ['settings.json']
qwen/qwen3.6-27b:medium__json_schema FAIL 0% 149 in → 1,210 out (1,039 reasoning)
Provide ONLY the resolved settings.json content as valid JSON.
JSON Schema Structured Output
(raw) { "resolved_content": "Provide ONLY the resolved settings.json content as valid JSON." }
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 151 in → 16 out (0 reasoning)
invalid_json_syntax
JSON Schema Structured Output
(raw) { "resolved_content": "invalid_json_syntax" }
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: {"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: {"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: {"name": "MyApp", "version": "2.0.0"}
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
tencent/hy3:high__json_schema FAIL 0% 149 in → 1,934 out (1,832 reasoning)
placeholder
JSON Schema Structured Output
(raw) { "resolved_content": "placeholder" }
Failure: Content mismatch: ['settings.json']
tencent/hy3:low__json_schema FAIL 0% 149 in → 736 out (573 reasoning)
not needed
JSON Schema Structured Output
(raw) { "resolved_content": "not needed" }
Failure: Content mismatch: ['settings.json']
tencent/hy3:medium__json_schema FAIL 0% 149 in → 277 out (197 reasoning)
not needed
JSON Schema Structured Output
(raw) { "resolved_content": "not needed" }
Failure: Content mismatch: ['settings.json']
thinkingmachines/inkling-small:low__json_schema FAIL 0% 147 in → 71 out (27 reasoning)
import java.util.*; public class Main { public static void main(String[] args) { System.out.println("Hello"); } }
JSON Schema Structured Output
(raw) { "resolved_content": "import java.util.*; public class Main { public static void main(String[] args) { System.out.println(\"Hello\"); } }" }
Failure: Content mismatch: ['settings.json']
thinkingmachines/inkling-small:none FAIL 0% 147 in → 24 out (0 reasoning)
{ "name": "MyApp", "version": "2.0.0" }<|end_message|>
Failure: Content mismatch: ['settings.json']
Invalid JSON. Output: { "name": "MyApp", "version": "2.0.0" }{"resolved_content": "{\n \"name\": \"MyApp\",\n \"version\": \"2.0.0\"\n}"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 4 column 2 (char 43)
Failure: Failed to parse structured JSON response: Extra data: line 4 column 2 (char 43)
thinkingmachines/inkling-small:xhigh__json_schema FAIL 0% 147 in → 257 out (217 reasoning)
Keep both: name from B (MyApp), version from A (2.0.0).
JSON Schema Structured Output
(raw) { "resolved_content": "Keep both: name from B (MyApp), version from A (2.0.0)." }
Failure: Content mismatch: ['settings.json']
thinkingmachines/inkling:high__json_schema FAIL 0% 147 in → 159 out (138 reasoning)
Invalid, need correct.
JSON Schema Structured Output
(raw) { "resolved_content": "Invalid, need correct." }
Failure: Content mismatch: ['settings.json']
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid structured output. Output: { "name": "MyApp", "version": "2.0.0" }
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ is missing required property 'resolved_content'
Failure: Structured output schema validation failed: $ is missing required property 'resolved_content'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium FAIL 0% 136 in → 5,658 out (5,146 reasoning)
```json { "name": "app", "version": "2.0.0" } ```
Failure: Content mismatch: ['settings.json']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 136 in → 20 out (0 reasoning)
{ "name": "app", "version": "2.0.0" }
Failure: Content mismatch: ['settings.json']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 136 in → 369 out (0 reasoning)
The user wants the resolved JSON content. Based on the prompt, the user likely wants the version from v2 (2.0.0) and the name from main (MyApp), or vice versa. However, usually, in a merge, you pick one. Let's look at the prompt again: 'Resolve the merge conflict'. It doesn't specify which branch to keep. But wait, the prompt says 'Provide ONLY the resolved settings.json content'. It implies I should make a choice. Let's look at the 'Conflicted file content' block. It shows HEAD (main) and v2. The prompt says 'Branch A (v2)... Branch B (main)...'. The conflict block shows HEAD (main) and v2. So HEAD is main. The prompt asks to resolve it. I will provide the content that combines the name from main and the version from v2, as that is a common 'best of both' resolution, or I will just pick one. Actually, looking at the prompt again, it's a bit ambiguous. But usually, if I have to provide *the* resolved content, I should probably pick the most logical one. Let's assume the user wants the version from v2 and the name from main. Or maybe just the v2 content? Or just the main content? Let's look at the prompt again. 'Resolve the merge conflict'. It doesn't say 'merge'. It says 'resolve'. This usually means pick one. I will provide the content that keeps the name 'MyApp' and the version '2.0.0'. This seems like a reasonable resolution. Or maybe I should just output the JSON block. I will output the JSON block with 'MyApp' and '2.0.0'.
JSON Schema Structured Output
(raw) { "resolved_content": "The user wants the resolved JSON content. Based on the prompt, the user likely wants the version from v2 (2.0.0) and the name from main (MyApp), or vice versa. However, usually, in a merge, you pick one. Let's look at the prompt again: 'Resolve the merge conflict'. It doesn't specify which branch to keep. But wait, the prompt says 'Provide ONLY the resolved settings.json content'. It implies I should make a choice. Let's look at the 'Conflicted file content' block. It shows HEAD (main) and v2. The prompt says 'Branch A (v2)... Branch B (main)...'. The conflict block shows HEAD (main) and v2. So HEAD is main. The prompt asks to resolve it. I will provide the content that combines the name from main and the version from v2, as that is a common 'best of both' resolution, or I will just pick one. Actually, looking at the prompt again, it's a bit ambiguous. But usually, if I have to provide *the* resolved content, I should probably pick the most logical one. Let's assume the user wants the version from v2 and the name from main. Or maybe just the v2 content? Or just the main content? Let's look at the prompt again. 'Resolve the merge conflict'. It doesn't say 'merge'. It says 'resolve'. This usually means pick one. I will provide the content that keeps the name 'MyApp' and the version '2.0.0'. This seems like a reasonable resolution. Or maybe I should just output the JSON block. I will output the JSON block with 'MyApp' and '2.0.0'." }
Failure: Content mismatch: ['settings.json']
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0% 136 in → 2,746 out (2,493 reasoning)
MyApp 2.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "MyApp 2.0.0" }
Failure: Content mismatch: ['settings.json']
z-ai/glm-5.2:low__json_schema FAIL 0% 143 in → 606 out (592 reasoning)
json
JSON Schema Structured Output
(raw) { "resolved_content": "json" }
Failure: Content mismatch: ['settings.json']