Multi-file: conflicts in both main.py and utils.py
Tests ability to resolve simultaneous rebase conflicts in two files with different strategies. Evaluates multi-file rebase juggling.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'def main(): print("Running")' > main.py
  5. 05 printf 'DEBUG = True PORT = 8000' > utils.py
  6. 06 git add main.py utils.py
  7. 07 git commit -m 'Initial code'
  8. 08 git checkout -b feature
  9. 09 printf 'def main(): print("Running v2")' > main.py
  10. 10 printf 'DEBUG = False PORT = 8000' > utils.py
  11. 11 git add main.py utils.py
  12. 12 git commit -m 'Feature: update main and disable debug'
  13. 13 git checkout main
  14. 14 printf 'def main(): print("Running enterprise")' > main.py
  15. 15 printf 'DEBUG = True PORT = 9000' > utils.py
  16. 16 git add main.py utils.py
  17. 17 git commit -m 'Enterprise: update main and port'
  18. 18 git rebase feature
Prompt
Resolve the rebase conflicts in main.py and utils.py. For main.py: branch being rebased says 'Running v2', main says 'Running enterprise'. For utils.py: branch being rebased set DEBUG=False with PORT=8000, main kept DEBUG=True with PORT=9000. Resolve both conflicts preferring the more descriptive version for main.py and the safer DEBUG=True for utils.py. Provide ONLY the two resolved file contents, separated by a blank line, with each file prefixed by its filename.
Expected
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000

Scoped model quality, cost, API time, and token usage for rebase/f010.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 341 in → 151 out (62 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-fable-5:low PASS 100% 343 in → 88 out (28 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-fable-5:max PASS 100% 337 in → 2,074 out (277 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-fable-5:medium PASS 100% 341 in → 162 out (68 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-fable-5:xhigh PASS 100% 341 in → 454 out (179 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:high PASS 100% 259 in → 280 out (224 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:low PASS 100% 263 in → 226 out (176 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:medium PASS 100% 261 in → 325 out (277 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-haiku-4.5:xhigh PASS 100% 261 in → 666 out (576 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.6:high PASS 100% 230 in → 320 out (162 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.6:max PASS 100% 228 in → 1,001 out (311 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.6:xhigh PASS 100% 230 in → 335 out (165 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.7:max PASS 100% 346 in → 1,527 out (133 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:none PASS 100% 344 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:xhigh PASS 100% 344 in → 887 out (187 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:high PASS 100% 343 in → 444 out (121 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:low PASS 100% 345 in → 377 out (149 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:max PASS 100% 345 in → 2,335 out (186 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:medium PASS 100% 343 in → 493 out (146 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:xhigh PASS 100% 341 in → 1,303 out (330 reasoning)
main.py: ```python def main(): print("Running enterprise") ``` utils.py: ```python DEBUG = True PORT = 9000 ```
anthropic/claude-sonnet-5:high PASS 100% 343 in → 801 out (306 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-sonnet-5:low PASS 100% 345 in → 77 out (33 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-sonnet-5:medium PASS 100% 339 in → 138 out (75 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
anthropic/claude-sonnet-5:none PASS 100% 343 in → 51 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-sonnet-5:xhigh PASS 100% 341 in → 1,474 out (292 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:high PASS 100% 199 in → 526 out (503 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:low PASS 100% 195 in → 1,660 out (1,642 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:medium PASS 100% 195 in → 613 out (590 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:xhigh PASS 100% 199 in → 992 out (970 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-mini:high PASS 100% 199 in → 275 out (294 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-mini:high__json_schema PASS 100% 195 in → 370 out (371 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
arcee-ai/trinity-mini:low PASS 100% 199 in → 406 out (429 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-mini:low__json_schema PASS 100% 193 in → 284 out (284 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
arcee-ai/trinity-mini:medium__json_schema PASS 100% 197 in → 330 out (332 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
arcee-ai/trinity-mini:xhigh PASS 100% 199 in → 323 out (345 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 199 in → 482 out (510 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n\n", "utils.py": "DEBUG = True\nPORT = 9000\n\n"}}
deepseek/deepseek-v4-flash-0731:high PASS 100% 205 in → 360 out (364 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 280 in → 488 out (497 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
deepseek/deepseek-v4-flash-0731:low PASS 100% 203 in → 520 out (495 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 203 in → 225 out (183 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"} }
deepseek/deepseek-v4-flash-0731:max PASS 100% 282 in → 1,519 out (1,598 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 278 in → 1,523 out (1,593 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n\nutils.py:\nDEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 203 in → 1,297 out (1,408 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 205 in → 344 out (346 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash-0731:none PASS 100% 203 in → 25 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 205 in → 48 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 284 in → 378 out (353 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 284 in → 688 out (709 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash:high PASS 100% 203 in → 419 out (393 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 201 in → 183 out (146 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash:low PASS 100% 201 in → 553 out (528 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 205 in → 424 out (372 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash:medium PASS 100% 201 in → 642 out (618 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 281 in → 1,402 out (1,592 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
deepseek/deepseek-v4-flash:xhigh PASS 100% 282 in → 1,952 out (1,926 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:high PASS 100% 201 in → 534 out (569 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:medium PASS 100% 201 in → 1,053 out (1,027 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 201 in → 49 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 284 in → 3,052 out (3,322 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files":{ "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
google/gemini-3-flash-preview:high PASS 100% 219 in → 642 out (612 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:high__json_schema PASS 100% 305 in → 1,340 out (1,303 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3-flash-preview:low PASS 100% 217 in → 567 out (537 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:low__json_schema PASS 100% 305 in → 901 out (864 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3-flash-preview:medium PASS 100% 217 in → 932 out (902 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:medium__json_schema PASS 100% 303 in → 1,954 out (1,913 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3-flash-preview:xhigh PASS 100% 217 in → 1,311 out (1,281 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 305 in → 1,975 out (1,938 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.1-flash-lite-preview:high PASS 100% 220 in → 710 out (680 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 297 in → 2,179 out (2,142 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 219 in → 554 out (524 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 220 in → 1,453 out (1,423 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 220 in → 2,368 out (2,318 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemini-3.1-pro-preview:high PASS 100% 219 in → 1,298 out (1,268 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 305 in → 3,054 out (3,013 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.1-pro-preview:low PASS 100% 217 in → 593 out (563 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 299 in → 350 out (296 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemini-3.1-pro-preview:medium PASS 100% 223 in → 619 out (589 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 305 in → 1,149 out (1,108 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.1-pro-preview:xhigh PASS 100% 223 in → 1,280 out (1,250 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 301 in → 2,279 out (2,238 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.5-flash-lite:high PASS 100% 223 in → 1,942 out (1,912 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:max PASS 100% 224 in → 1,669 out (1,639 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 301 in → 1,594 out (1,557 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.5-flash-lite:medium PASS 100% 219 in → 1,375 out (1,345 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 222 in → 972 out (914 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
google/gemini-3.5-flash-lite:xhigh PASS 100% 224 in → 1,827 out (1,797 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 222 in → 1,758 out (1,717 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.5-flash:high PASS 100% 221 in → 1,271 out (1,241 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:high__json_schema PASS 100% 305 in → 1,367 out (1,322 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.5-flash:low PASS 100% 219 in → 622 out (592 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:low__json_schema PASS 100% 301 in → 573 out (532 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3.5-flash:medium PASS 100% 221 in → 1,020 out (990 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:medium__json_schema PASS 100% 305 in → 1,306 out (1,248 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 303 in → 1,453 out (1,408 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.6-flash:high PASS 100% 218 in → 2,895 out (2,865 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:high__json_schema PASS 100% 224 in → 1,826 out (1,785 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.6-flash:low PASS 100% 222 in → 463 out (433 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:max PASS 100% 224 in → 1,915 out (1,885 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:max__json_schema PASS 100% 220 in → 1,406 out (1,365 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.6-flash:medium PASS 100% 218 in → 1,344 out (1,314 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:medium__json_schema PASS 100% 224 in → 1,394 out (1,353 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.6-flash:xhigh PASS 100% 221 in → 2,656 out (2,626 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 301 in → 1,648 out (1,603 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
google/gemma-4-26b-a4b-it:high PASS 100% 235 in → 3,046 out (2,716 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 239 in → 4,589 out (4,095 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:low PASS 100% 237 in → 2,489 out (2,275 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 234 in → 2,526 out (2,299 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:medium PASS 100% 235 in → 2,469 out (2,497 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 234 in → 1,898 out (1,714 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:none PASS 100% 236 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 236 in → 51 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-31b-it:high PASS 100% 238 in → 340 out (281 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:high__json_schema PASS 100% 239 in → 504 out (1 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-31b-it:low PASS 100% 235 in → 366 out (286 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:low__json_schema PASS 100% 235 in → 80 out (27 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
google/gemma-4-31b-it:medium PASS 100% 234 in → 469 out (397 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:medium__json_schema PASS 100% 236 in → 451 out (338 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-31b-it:none PASS 100% 234 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:none__json_schema PASS 100% 230 in → 50 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" }}
google/gemma-4-31b-it:xhigh PASS 100% 233 in → 567 out (453 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 232 in → 375 out (302 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
ibm-granite/granite-4.1-8b PASS 100% 214 in → 26 out
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
inclusionai/ling-2.6-flash PASS 100% 234 in → 30 out
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
liquid/lfm-2-24b-a2b PASS 100% 228 in → 26 out
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:high PASS 100% 230 in → 897 out (1,021 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:high__json_schema PASS 100% 251 in → 604 out (675 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" }}
minimax/minimax-m2.5:low PASS 100% 253 in → 1,204 out (1,178 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
minimax/minimax-m2.5:low__json_schema PASS 100% 228 in → 759 out (848 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
minimax/minimax-m2.5:medium PASS 100% 230 in → 491 out (561 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:medium__json_schema PASS 100% 228 in → 641 out (699 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.5:xhigh PASS 100% 230 in → 857 out (831 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 255 in → 947 out (900 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
minimax/minimax-m2.7:high PASS 100% 228 in → 424 out (398 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:high__json_schema PASS 100% 225 in → 925 out (1,029 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.7:medium PASS 100% 232 in → 544 out (517 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:medium__json_schema PASS 100% 375 in → 934 out (884 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.7:xhigh PASS 100% 229 in → 828 out (932 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 379 in → 927 out (877 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m3:high PASS 100% 314 in → 157 out (129 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:high__json_schema PASS 100% 314 in → 50 out (48 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m3:low PASS 100% 318 in → 321 out (295 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:medium PASS 100% 368 in → 466 out (470 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:medium__json_schema PASS 100% 357 in → 46 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m3:xhigh PASS 100% 368 in → 174 out (159 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:xhigh__json_schema PASS 100% 318 in → 43 out (41 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
mistralai/devstral-2512__json_schema PASS 100% 216 in → 40 out
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
mistralai/mistral-medium-3-5:high PASS 100% 230 in → 2,629 out (2,804 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 216 in → 2,367 out (2,499 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
mistralai/mistral-medium-3-5:low PASS 100% 228 in → 2,434 out (2,544 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 220 in → 390 out (379 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
mistralai/mistral-medium-3-5:medium PASS 100% 232 in → 809 out (798 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 216 in → 2,151 out (2,228 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
mistralai/mistral-medium-3-5:none PASS 100% 232 in → 27 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:xhigh PASS 100% 230 in → 1,056 out (1,029 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 216 in → 1,717 out (1,856 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
moonshotai/kimi-k2.6:high PASS 100% 199 in → 2,380 out (2,701 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.6:low PASS 100% 201 in → 1,956 out (1,932 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.6:medium PASS 100% 197 in → 1,547 out (1,519 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 196 in → 1,336 out (1,325 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
moonshotai/kimi-k2.6:xhigh PASS 100% 195 in → 3,047 out (3,021 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 200 in → 2,199 out (2,003 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
moonshotai/kimi-k2.7-code:high PASS 100% 199 in → 1,001 out (975 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 196 in → 732 out (555 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
moonshotai/kimi-k2.7-code:low PASS 100% 199 in → 483 out (457 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:medium PASS 100% 199 in → 346 out (320 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 295 in → 706 out (667 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 200 in → 502 out (476 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 197 in → 486 out (503 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
moonshotai/kimi-k3:high PASS 100% 276 in → 1,486 out (1,447 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:high__json_schema PASS 100% 278 in → 792 out (751 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:low PASS 100% 278 in → 1,030 out (991 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:low__json_schema PASS 100% 278 in → 522 out (481 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:max PASS 100% 274 in → 2,085 out (2,044 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:max__json_schema PASS 100% 274 in → 309 out (270 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:medium PASS 100% 274 in → 598 out (557 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:medium__json_schema PASS 100% 274 in → 415 out (374 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:xhigh PASS 100% 278 in → 520 out (481 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 274 in → 448 out (409 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 231 in → 488 out (516 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 233 in → 344 out (347 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 233 in → 225 out (208 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 229 in → 562 out (544 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 229 in → 206 out (178 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 233 in → 29 out (0 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 229 in → 52 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 233 in → 827 out (903 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 229 in → 483 out (496 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 225 in → 199 out (161 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 233 in → 261 out (200 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 227 in → 197 out (162 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 231 in → 259 out (258 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 233 in → 352 out (284 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 233 in → 27 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 229 in → 48 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 231 in → 259 out (273 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 231 in → 267 out (207 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
openai/gpt-5.4-mini:high PASS 100% 192 in → 155 out (125 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:low PASS 100% 194 in → 189 out (159 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:medium PASS 100% 198 in → 187 out (157 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:none PASS 100% 196 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:xhigh PASS 100% 194 in → 956 out (926 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-nano:high PASS 100% 196 in → 156 out (116 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
openai/gpt-5.4-nano:low PASS 100% 192 in → 38 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
openai/gpt-5.4-nano:medium PASS 100% 190 in → 38 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
openai/gpt-5.4-nano:none PASS 100% 192 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-nano:xhigh PASS 100% 194 in → 253 out (223 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:high PASS 100% 194 in → 943 out (913 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:low PASS 100% 194 in → 50 out (20 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:medium PASS 100% 192 in → 382 out (352 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:none PASS 100% 194 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:xhigh PASS 100% 192 in → 961 out (931 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:high PASS 100% 194 in → 382 out (352 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:medium PASS 100% 192 in → 181 out (151 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:xhigh PASS 100% 192 in → 546 out (516 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:high PASS 100% 194 in → 174 out (144 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:low PASS 100% 192 in → 127 out (97 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:max PASS 100% 194 in → 337 out (307 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:medium PASS 100% 196 in → 152 out (122 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:none PASS 100% 192 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:xhigh PASS 100% 196 in → 198 out (168 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:high PASS 100% 194 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:low PASS 100% 196 in → 78 out (48 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:max PASS 100% 196 in → 262 out (232 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:medium PASS 100% 194 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:none PASS 100% 194 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:xhigh PASS 100% 192 in → 138 out (108 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:high PASS 100% 196 in → 118 out (88 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:low PASS 100% 194 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:max PASS 100% 196 in → 311 out (281 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:medium PASS 100% 194 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:none PASS 100% 196 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:xhigh PASS 100% 192 in → 141 out (111 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:high PASS 100% 255 in → 1,468 out (1,693 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:high__json_schema PASS 100% 242 in → 559 out (602 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
openai/gpt-oss-120b:low PASS 100% 255 in → 192 out (156 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:low__json_schema PASS 100% 253 in → 111 out (62 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
openai/gpt-oss-120b:medium PASS 100% 255 in → 193 out (183 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:medium__json_schema PASS 100% 255 in → 321 out (270 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
openai/gpt-oss-120b:xhigh PASS 100% 242 in → 517 out (577 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 257 in → 684 out (786 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
openai/gpt-oss-20b:high PASS 100% 255 in → 855 out (1,000 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:high__json_schema PASS 100% 255 in → 468 out (505 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"} }
openai/gpt-oss-20b:low PASS 100% 257 in → 65 out (33 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:medium PASS 100% 261 in → 240 out (207 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:xhigh PASS 100% 259 in → 1,023 out (1,202 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 255 in → 689 out (736 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py":"def main():\n print(\"Running enterprise\")", "utils.py":"DEBUG = True\nPORT = 9000"} }
poolside/laguna-m.1:high PASS 100% 219 in → 287 out (256 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:high__json_schema PASS 100% 219 in → 1,642 out (1,603 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-m.1:low PASS 100% 221 in → 429 out (398 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:low__json_schema PASS 100% 221 in → 1,096 out (1,045 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-m.1:medium PASS 100% 221 in → 398 out (368 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:medium__json_schema PASS 100% 223 in → 398 out (356 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
poolside/laguna-m.1:none PASS 100% 219 in → 30 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:xhigh PASS 100% 223 in → 406 out (375 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:xhigh__json_schema PASS 100% 223 in → 392 out (339 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
poolside/laguna-xs-2.1:high__json_schema PASS 100% 223 in → 1,419 out (1,372 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-xs-2.1:low PASS 100% 221 in → 543 out (512 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:low__json_schema PASS 100% 219 in → 2,339 out (2,289 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
poolside/laguna-xs-2.1:medium PASS 100% 223 in → 1,107 out (1,076 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 223 in → 1,116 out (1,068 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-xs-2.1:none PASS 100% 217 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:none__json_schema PASS 100% 221 in → 51 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-xs-2.1:xhigh PASS 100% 223 in → 1,438 out (1,407 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 221 in → 1,370 out (1,321 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-xs.2:high__json_schema PASS 100% 260 in → 280 out (239 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs.2:low PASS 100% 256 in → 288 out (257 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs.2:low__json_schema PASS 100% 258 in → 258 out (217 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs.2:medium PASS 100% 256 in → 227 out (188 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 9000 ```
poolside/laguna-xs.2:medium__json_schema PASS 100% 260 in → 552 out (513 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs.2:none PASS 100% 258 in → 30 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs.2:none__json_schema PASS 100% 258 in → 41 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
poolside/laguna-xs.2:xhigh PASS 100% 258 in → 226 out (195 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 260 in → 201 out (162 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
qwen/qwen3.6-27b:high PASS 100% 218 in → 1,437 out (1 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:high__json_schema PASS 100% 216 in → 5,990 out (1 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-27b:low PASS 100% 214 in → 1,790 out (1,766 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:low__json_schema PASS 100% 214 in → 1,839 out (1,826 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-27b:medium PASS 100% 212 in → 1,920 out (1,887 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:none PASS 100% 220 in → 29 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:xhigh PASS 100% 212 in → 1,509 out (1,498 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 212 in → 2,189 out (1 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-35b-a3b:high PASS 100% 218 in → 1,692 out (1,644 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 216 in → 2,193 out (2,176 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
qwen/qwen3.6-35b-a3b:low PASS 100% 218 in → 2,148 out (2,145 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 216 in → 2,132 out (2,093 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-35b-a3b:medium PASS 100% 218 in → 2,173 out (2,140 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 216 in → 1,724 out (1,673 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
qwen/qwen3.6-35b-a3b:none PASS 100% 220 in → 29 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 218 in → 1,706 out (1,683 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 216 in → 2,156 out (2,114 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
qwen/qwen3.6-flash:high PASS 100% 212 in → 2,192 out (2,159 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:low PASS 100% 218 in → 1,856 out (1,823 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:medium PASS 100% 216 in → 2,014 out (1,981 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:xhigh PASS 100% 216 in → 1,782 out (1,749 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:high PASS 100% 218 in → 1,692 out (1,657 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:low PASS 100% 212 in → 3,327 out (3,294 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:max PASS 100% 216 in → 1,677 out (1,644 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:medium PASS 100% 218 in → 2,193 out (2,160 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:none PASS 100% 216 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:xhigh PASS 100% 214 in → 1,834 out (1,799 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:high PASS 100% 218 in → 822 out (789 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:low PASS 100% 214 in → 682 out (649 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:medium PASS 100% 218 in → 594 out (561 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:none PASS 100% 216 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:xhigh PASS 100% 218 in → 702 out (669 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:high PASS 100% 218 in → 767 out (734 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:low PASS 100% 218 in → 820 out (779 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 9000 ```
qwen/qwen3.7-plus:medium PASS 100% 214 in → 1,043 out (1,008 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:none PASS 100% 216 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:xhigh PASS 100% 218 in → 1,736 out (1,703 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:high PASS 100% 202 in → 2,147 out (2,121 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:low PASS 100% 204 in → 1,599 out (1,573 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:low__json_schema PASS 100% 208 in → 2,158 out (2,183 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:medium PASS 100% 206 in → 1,257 out (1,262 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:medium__json_schema PASS 100% 204 in → 1,116 out (1,090 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:none PASS 100% 213 in → 25 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:none__json_schema PASS 100% 209 in → 55 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:xhigh PASS 100% 206 in → 11,391 out (11,364 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:xhigh__json_schema PASS 100% 206 in → 1,648 out (1,647 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
thinkingmachines/inkling-small:high PASS 100% 207 in → 497 out (467 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:high__json_schema PASS 100% 205 in → 1,386 out (1,337 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
thinkingmachines/inkling-small:low PASS 100% 203 in → 180 out (150 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:low__json_schema PASS 100% 201 in → 219 out (171 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
thinkingmachines/inkling-small:max PASS 100% 201 in → 2,117 out (2,087 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:max__json_schema PASS 100% 205 in → 1,013 out (961 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
thinkingmachines/inkling-small:medium PASS 100% 199 in → 378 out (348 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:medium__json_schema PASS 100% 201 in → 559 out (507 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
thinkingmachines/inkling-small:none PASS 100% 203 in → 27 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:xhigh PASS 100% 199 in → 1,185 out (1,155 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 203 in → 2,569 out (2,524 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
thinkingmachines/inkling:high PASS 100% 199 in → 306 out (303 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:low PASS 100% 205 in → 72 out (46 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:max PASS 100% 203 in → 602 out (572 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:max__json_schema PASS 100% 207 in → 3,647 out (3,606 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
thinkingmachines/inkling:medium PASS 100% 203 in → 334 out (304 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:none PASS 100% 197 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:xhigh PASS 100% 201 in → 421 out (390 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:xhigh__json_schema PASS 100% 203 in → 607 out (563 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
x-ai/grok-4.3:high PASS 100% 373 in → 591 out (567 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:high__json_schema PASS 100% 437 in → 1,762 out (1,732 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
x-ai/grok-4.3:low PASS 100% 375 in → 336 out (312 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:low__json_schema PASS 100% 441 in → 405 out (361 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.3:max PASS 100% 373 in → 463 out (439 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:max__json_schema PASS 100% 433 in → 846 out (804 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.3:medium PASS 100% 377 in → 534 out (510 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:medium__json_schema PASS 100% 441 in → 571 out (529 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.3:none PASS 100% 369 in → 24 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:none__json_schema PASS 100% 437 in → 30 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
x-ai/grok-4.3:xhigh PASS 100% 369 in → 755 out (731 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:xhigh__json_schema PASS 100% 437 in → 1,206 out (1,164 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.5:high PASS 100% 397 in → 1,009 out (983 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:high__json_schema PASS 100% 475 in → 1,064 out (1,015 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:low PASS 100% 403 in → 462 out (436 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:low__json_schema PASS 100% 477 in → 502 out (453 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:max PASS 100% 399 in → 252 out (226 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:max__json_schema PASS 100% 473 in → 800 out (751 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:medium PASS 100% 397 in → 646 out (620 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:medium__json_schema PASS 100% 473 in → 1,306 out (1,257 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:xhigh PASS 100% 401 in → 324 out (298 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:xhigh__json_schema PASS 100% 477 in → 1,587 out (1,538 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
z-ai/glm-4.7-flash:high PASS 100% 191 in → 4,199 out (4,828 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:low PASS 100% 193 in → 582 out (597 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:low__json_schema PASS 100% 195 in → 1,079 out (1,088 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
z-ai/glm-4.7-flash:medium PASS 100% 195 in → 1,168 out (1,254 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 193 in → 553 out (518 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
z-ai/glm-4.7-flash:none PASS 100% 191 in → 25 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:none__json_schema PASS 100% 191 in → 45 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
z-ai/glm-5.2:high PASS 100% 202 in → 691 out (670 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:low PASS 100% 200 in → 1,013 out (1,040 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:low__json_schema PASS 100% 200 in → 1,096 out (1,149 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
z-ai/glm-5.2:medium PASS 100% 200 in → 301 out (275 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:medium__json_schema PASS 100% 198 in → 879 out (883 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
z-ai/glm-5.2:none PASS 100% 194 in → 27 out (0 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:xhigh PASS 100% 200 in → 2,818 out (3,048 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:xhigh__json_schema PASS 100% 202 in → 1,431 out (1,391 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
anthropic/claude-haiku-4.5:none FAIL 50% 234 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:low FAIL 50% 228 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:medium FAIL 50% 230 in → 39 out (0 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:none FAIL 50% 234 in → 39 out (0 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.7:high FAIL 50% 348 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.7:low FAIL 50% 344 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.7:medium FAIL 50% 348 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:high FAIL 50% 234 in → 523 out (196 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution rationale:** - **main.py** — `"Running enterprise"` is chosen over `"Running v2"` as it is the more descriptive/specific version string, conveying the deployment tier rather than just a generic version number. - **utils.py (DEBUG)** — `DEBUG = True` is kept as the safer choice; leaving debug mode enabled avoids accidentally suppressing error output/logging in an environment that may not yet be fully validated. - **utils.py (PORT)** — `PORT = 9000` is kept alongside `DEBUG = True` since both values originate from the same side of the conflict (main), keeping the configuration internally consistent as it was authored.
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:low FAIL 50% 230 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:max FAIL 50% 230 in → 2,429 out (543 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution notes:** - **main.py** — `"Running enterprise"` is kept over `"Running v2"` as it is the more descriptive/meaningful version of the two strings. - **utils.py** — `DEBUG = True` is kept because it is the safer setting (debug mode on is safer during development/testing, avoiding accidental suppression of error output). `PORT = 9000` is taken from the same incoming block as `DEBUG = True`, keeping the two values internally consistent from their originating commit (`cab2fb2`).
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:medium FAIL 50% 228 in → 493 out (197 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution rationale:** - **main.py** — `"Running enterprise"` is chosen as the more descriptive version over `"Running v2"`, per the stated preference. - **utils.py** — `DEBUG = True` is chosen as the safer option (explicit debug visibility). `PORT = 9000` is kept alongside it since it originates from the same side of the conflict (main) that provided `DEBUG = True`, keeping the configuration internally consistent. The instruction only specified a preference for `DEBUG`; no split-pick was indicated for `PORT`.
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:none FAIL 50% 230 in → 41 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:xhigh FAIL 50% 230 in → 861 out (355 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution rationale:** - **main.py** — `"Running enterprise"` is chosen over `"Running v2"` as it is the more descriptive/specific version, per the stated preference. - **utils.py** — `DEBUG = True` is chosen over `DEBUG = False` as it is the safer setting (e.g., enables error output during development, avoids silently swallowing issues). `PORT = 9000` is kept alongside it, as it belongs to the same "main" block that preserved the safer debug behavior.
Failure: Content mismatch: ['utils.py']
deepseek/deepseek-v4-flash:none FAIL 50% 199 in → 25 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
deepseek/deepseek-v4-pro:high__json_schema FAIL 50% 203 in → 486 out (437 reasoning)
main.py: def main(): print("Running enterprise v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise v2\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
Failure: Content mismatch: ['main.py']
deepseek/deepseek-v4-pro:low FAIL 50% 201 in → 2,021 out (2,242 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = TruePORT = 9000
Failure: Content mismatch: ['utils.py']
deepseek/deepseek-v4-pro:none FAIL 50% 205 in → 24 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
google/gemini-3.1-flash-lite-preview:low FAIL 50% 221 in → 141 out (111 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 50% 224 in → 150 out (96 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 8000" } }
Failure: Content mismatch: ['utils.py']
google/gemini-3.5-flash-lite:high__json_schema FAIL 50% 224 in → 1,704 out (1,660 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000000"}}
Failure: Content mismatch: ['utils.py']
google/gemini-3.5-flash-lite:low FAIL 50% 221 in → 30 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
google/gemini-3.5-flash:xhigh FAIL 50% 223 in → 2,135 out (2,103 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000_
Failure: Content mismatch: ['utils.py']
google/gemini-3.6-flash:low__json_schema FAIL 50% 224 in → 58 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 8000\n" } }
Failure: Content mismatch: ['utils.py']
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9_000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9_000\n" } }
Failure: Content mismatch: ['utils.py']
main.py: def main(): print("Running enterprise - updated with enterprise configurations and settings") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise - updated with enterprise configurations and settings\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
Failure: Content mismatch: ['main.py']
minimax/minimax-m2.7:low FAIL 50% 226 in → 1,203 out (1,349 reasoning)
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
nvidia/nemotron-3-nano-30b-a3b:high FAIL 50% 233 in → 222 out (232 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
openai/gpt-5.5:low FAIL 50% 194 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
openai/gpt-5.5:none FAIL 50% 192 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
openai/gpt-oss-20b:low__json_schema FAIL 50% 240 in → 87 out (32 reasoning)
main.py: def main(): print("Running enterprise"), utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\"),\n\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
Failure: Content mismatch: ['main.py']
poolside/laguna-xs-2.1:high FAIL 50% 223 in → 1,415 out (1,384 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
poolside/laguna-xs.2:high FAIL 50% 260 in → 240 out (205 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
qwen/qwen3.6-flash:none FAIL 50% 218 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = False PORT = 8000
Failure: Content mismatch: ['utils.py']
z-ai/glm-4.7-flash:xhigh FAIL 50% 191 in → 2,638 out (2,862 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-fable-5:high__json_schema FAIL 0% 572 in → 217 out (98 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:low__json_schema FAIL 0% 576 in → 43 out (28 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-fable-5:max__json_schema FAIL 0% 578 in → 909 out (148 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:medium__json_schema FAIL 0% 572 in → 84 out (35 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:xhigh__json_schema FAIL 0% 576 in → 368 out (166 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:high__json_schema FAIL 0% 438 in → 1,413 out (807 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:low__json_schema FAIL 0% 436 in → 1,566 out (777 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:medium__json_schema FAIL 0% 436 in → 4,763 out (1,736 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 410 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:xhigh__json_schema FAIL 0% 440 in → 3,027 out (1,339 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 407 in → 407 out (191 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:low__json_schema FAIL 0% 409 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:max__json_schema FAIL 0% 407 in → 420 out (192 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:medium__json_schema FAIL 0% 407 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:none__json_schema FAIL 0% 405 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 411 in → 317 out (185 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:high__json_schema FAIL 0% 575 in → 29 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:low__json_schema FAIL 0% 581 in → 25 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:max__json_schema FAIL 0% 579 in → 2,363 out (167 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:medium__json_schema FAIL 0% 581 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:none__json_schema FAIL 0% 579 in → 146 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:xhigh__json_schema FAIL 0% 577 in → 16 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:high__json_schema FAIL 0% 576 in → 383 out (124 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:low__json_schema FAIL 0% 574 in → 54 out (51 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 572 in → 1,834 out (258 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-opus-4.8:medium__json_schema FAIL 0% 574 in → 388 out (208 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:none FAIL 0% 347 in → 286 out (0 reasoning)
Here are the resolved files: main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` A note on the conflict markers: during a rebase, `HEAD` refers to the branch you're rebasing *onto* (main), and the labeled side (`8e9dbd3`) is the commit being replayed from the branch being rebased. So in this case: - **main.py**: `HEAD` = `"Running v2"` (from main?) — but your description says main has `"Running enterprise"`. The more descriptive version is `"Running enterprise"`, so I used that. - **utils.py**: The safer `DEBUG = True` with `PORT = 9000` was kept. If the mapping of HEAD-vs-branch is different in your actual repo than described, double-check which side is which before committing.
Failure: Expected file heading before file content
anthropic/claude-opus-4.8:none__json_schema FAIL 0% 572 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:xhigh__json_schema FAIL 0% 572 in → 360 out (70 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:high__json_schema FAIL 0% 407 in → 590 out (216 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:low__json_schema FAIL 0% 407 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:max__json_schema FAIL 0% 407 in → 1,379 out (346 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:medium__json_schema FAIL 0% 409 in → 303 out (259 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:none__json_schema FAIL 0% 407 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:xhigh__json_schema FAIL 0% 407 in → 630 out (448 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:high__json_schema FAIL 0% 572 in → 63 out (37 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 580 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 578 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 572 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 574 in → 189 out (58 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:medium FAIL 0% 195 in → 484 out (508 reasoning)
main.pydef main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
Failure: Expected file heading before file content
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 203 in → 45 out (0 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running v2\")\n", "utils.py": "DEBUG = True\nPORT = 8000\n" } }
Failure: Content mismatch: ['main.py', 'utils.py']
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 209 in → 355 out (342 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "files": {} }
Failure: No file blocks found
(empty output)
JSON Schema Structured Output
Failure: Connection error.
deepseek/deepseek-v4-pro:xhigh FAIL 0% 203 in → 32 out (0 reasoning)
```python --- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000 ```
Failure: Expected file heading before file content
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 222 in → 1,299 out (1,245 reasoning)
main.py: def main():\n print("Running enterprise") utils.py: DEBUG = True\nPORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\\nPORT = 9000" } }
Failure: Content mismatch: ['main.py', 'utils.py']
google/gemini-3.5-flash-lite:low__json_schema FAIL 0% 220 in → 55 out (0 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running v2\")", "utils.py": "DEBUG = True\nPORT = 8000" } }
Failure: Content mismatch: ['main.py', 'utils.py']
google/gemma-4-26b-a4b-it:xhigh FAIL 0% 237 in → 525 out (494 reasoning)
(empty output)
Failure: No file blocks found
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
mistralai/devstral-2512 FAIL 0% 220 in → 38 out
```python --- main.py def main(): print("Running enterprise") ``` ```python --- utils.py DEBUG = True PORT = 9000 ```
Failure: Expected file heading before file content
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
nvidia/nemotron-3-super-120b-a12b:low__json_schema FAIL 0% 233 in → 449 out (375 reasoning)
main.py: main.py def main(): print("Running enterprise") utils.py: utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "main.py\ndef main():\n print(\"Running enterprise\")", "utils.py": "utils.py\nDEBUG = True\nPORT = 9000" } }
Failure: Duplicate file block: main.py
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
openai/gpt-oss-20b:medium__json_schema FAIL 0% 259 in → 16 out (0 reasoning)
analysis: ... final: ...
JSON Schema Structured Output
(raw) {"files": {"analysis": "...", "final": "..."} }
Failure: Expected file heading before file content
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ebd58c6f-9fa0-9d62-8e1e-faceebf90cbb","request_id":"ebd58c6f-9fa0-9d62-8e1e-faceebf90cbb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c281b11f-a2f7-9e3b-9b4f-f5b4ba352fad","request_id":"c281b11f-a2f7-9e3b-9b4f-f5b4ba352fad"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-75598cf4-455a-9e07-bdb5-036b0e1ad2bd","request_id":"75598cf4-455a-9e07-bdb5-036b0e1ad2bd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a3117fc9-f723-9fb2-9d0d-ff9f1e3b216e","request_id":"a3117fc9-f723-9fb2-9d0d-ff9f1e3b216e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cab7eb26-8379-9c69-8f91-3f5b5005e3a5","request_id":"cab7eb26-8379-9c69-8f91-3f5b5005e3a5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6ff29f62-604f-9c35-b053-93f5b8c54102"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7a79c528-8d5e-918d-ac4d-6f750674c712"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-eaf1e527-7182-96c4-82f6-0531760be9b6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c830985e-5c10-9672-acc3-a7d2f09011bb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-700a494b-f203-9acc-afa7-d4384b749145"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ffc699ba-213e-996f-8265-05ddc3c762ea"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-53b0eb5f-5dda-944a-8001-4ee2ae18b2f5","request_id":"53b0eb5f-5dda-944a-8001-4ee2ae18b2f5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e8633280-8b80-9e3e-83df-b90e8b42680d","request_id":"e8633280-8b80-9e3e-83df-b90e8b42680d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a68a0932-f5a4-90ad-ac61-a2a4ea9f11c4","request_id":"a68a0932-f5a4-90ad-ac61-a2a4ea9f11c4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f2497c13-b486-9d38-b7ec-620a8e32ea14","request_id":"f2497c13-b486-9d38-b7ec-620a8e32ea14"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-63d10762-f208-96ef-8bbf-16071f71e00e","request_id":"63d10762-f208-96ef-8bbf-16071f71e00e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6f43b3b2-a942-9c8b-ab83-bfac40e1afa9","request_id":"6f43b3b2-a942-9c8b-ab83-bfac40e1afa9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-da8f1676-0f32-9f93-a227-1d8b51a6eb6a","request_id":"da8f1676-0f32-9f93-a227-1d8b51a6eb6a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-05faefa2-e7b0-99d3-b361-d1e7bacfdaa9","request_id":"05faefa2-e7b0-99d3-b361-d1e7bacfdaa9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:high__json_schema FAIL 0% 204 in → 5,064 out (5,350 reasoning)
main.py: main.py def main(): print("Running enterprise") utils.py: utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "main.py\ndef main():\n print(\"Running enterprise\")\n", "utils.py": "utils.py\nDEBUG = True\nPORT = 9000\n" } }
Failure: Duplicate file block: main.py
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000{"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000{ "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: --- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.