f007 — git_show

← git_show
Show commit author date
Tests ability to extract commit author date from git show. Evaluates date-field extraction from commit metadata.
easy git-show author-date metadata
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'dev@org.io'
  3. 03 git config user.name 'Developer'
  4. 04 git config commit.gpgsign false
  5. 05 echo 'data' > config.yaml
  6. 06 git add config.yaml
  7. 07 GIT_AUTHOR_DATE='2025-06-15T10:30:00' GIT_COMMITTER_DATE='2025-06-15T10:30:00' git commit -m 'Mid-year update'
Prompt
Using git show -s --format=%ae, what is the author email of the latest commit? Output ONLY the email, nothing else.
Expected
dev@org.io
Fixture Metrics

Scoped model quality, cost, API time, and token usage for git_show/f007.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 232 in → 28 out (19 reasoning)
dev@org.io
anthropic/claude-fable-5:high__json_schema PASS 100% 436 in → 35 out (10 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
anthropic/claude-fable-5:low PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-fable-5:low__json_schema PASS 100% 436 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
anthropic/claude-fable-5:max PASS 100% 232 in → 114 out (64 reasoning)
dev@org.io
anthropic/claude-fable-5:medium PASS 100% 232 in → 25 out (24 reasoning)
dev@org.io
anthropic/claude-fable-5:medium__json_schema PASS 100% 436 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-fable-5:xhigh PASS 100% 232 in → 31 out (15 reasoning)
dev@org.io
anthropic/claude-haiku-4.5:high PASS 100% 220 in → 115 out (90 reasoning)
dev@org.io
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 380 in → 260 out (222 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-haiku-4.5:low PASS 100% 220 in → 171 out (130 reasoning)
dev@org.io
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 380 in → 349 out (338 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-haiku-4.5:medium PASS 100% 220 in → 189 out (152 reasoning)
dev@org.io
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 380 in → 239 out (202 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
anthropic/claude-haiku-4.5:none PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 350 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 220 in → 188 out (150 reasoning)
dev@org.io
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 380 in → 250 out (213 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.6:high PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.6:high__json_schema PASS 100% 351 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.6:low PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.6:low__json_schema PASS 100% 351 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.6:max PASS 100% 191 in → 108 out (67 reasoning)
dev@org.io
anthropic/claude-opus-4.6:max__json_schema PASS 100% 351 in → 85 out (56 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.6:medium PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 351 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.6:none PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.6:none__json_schema PASS 100% 351 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.6:xhigh PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 351 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.7:high PASS 100% 237 in → 10 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.7:high__json_schema PASS 100% 441 in → 17 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.7:low PASS 100% 237 in → 10 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.7:low__json_schema PASS 100% 441 in → 17 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.7:max PASS 100% 237 in → 10 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.7:max__json_schema PASS 100% 441 in → 18 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.7:medium PASS 100% 237 in → 10 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 441 in → 17 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.7:none PASS 100% 237 in → 10 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.7:none__json_schema PASS 100% 441 in → 17 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.7:xhigh PASS 100% 237 in → 10 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 441 in → 17 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.8:high PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.8:high__json_schema PASS 100% 436 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.8:low PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.8:low__json_schema PASS 100% 436 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.8:max PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.8:max__json_schema PASS 100% 436 in → 67 out (14 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
anthropic/claude-opus-4.8:medium PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 436 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.8:none PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.8:none__json_schema PASS 100% 436 in → 14 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-opus-4.8:xhigh PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 436 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-sonnet-4.6:high PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 351 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-sonnet-4.6:low PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 351 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-sonnet-4.6:max PASS 100% 191 in → 106 out (67 reasoning)
dev@org.io
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 351 in → 65 out (41 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-sonnet-4.6:medium PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 351 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-sonnet-4.6:none PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 351 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 191 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 351 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
anthropic/claude-sonnet-5:high PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-sonnet-5:high__json_schema PASS 100% 436 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
anthropic/claude-sonnet-5:low PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-sonnet-5:low__json_schema PASS 100% 436 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
anthropic/claude-sonnet-5:medium PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 436 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
anthropic/claude-sonnet-5:none PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-sonnet-5:none__json_schema PASS 100% 436 in → 14 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
anthropic/claude-sonnet-5:xhigh PASS 100% 232 in → 8 out (0 reasoning)
dev@org.io
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 436 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
arcee-ai/trinity-large-thinking:high PASS 100% 166 in → 502 out (497 reasoning)
dev@org.io
arcee-ai/trinity-large-thinking:low PASS 100% 166 in → 331 out (326 reasoning)
dev@org.io
arcee-ai/trinity-large-thinking:medium PASS 100% 166 in → 322 out (315 reasoning)
dev@org.io
arcee-ai/trinity-large-thinking:xhigh PASS 100% 166 in → 321 out (315 reasoning)
dev@org.io
arcee-ai/trinity-mini:high PASS 100% 166 in → 258 out (277 reasoning)
dev@org.io
arcee-ai/trinity-mini:high__json_schema PASS 100% 166 in → 248 out (278 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
arcee-ai/trinity-mini:low PASS 100% 166 in → 325 out (374 reasoning)
dev@org.io
arcee-ai/trinity-mini:low__json_schema PASS 100% 166 in → 250 out (272 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
arcee-ai/trinity-mini:medium PASS 100% 166 in → 254 out (294 reasoning)
dev@org.io
arcee-ai/trinity-mini:medium__json_schema PASS 100% 166 in → 314 out (353 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
arcee-ai/trinity-mini:xhigh PASS 100% 166 in → 251 out (274 reasoning)
dev@org.io
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 166 in → 346 out (378 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 162 in → 77 out (60 reasoning)
dev@org.io
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 241 in → 86 out (82 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 162 in → 89 out (67 reasoning)
dev@org.io
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 162 in → 45 out (33 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 241 in → 37 out (36 reasoning)
dev@org.io
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 254 in → 68 out (58 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 162 in → 80 out (75 reasoning)
dev@org.io
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 164 in → 85 out (75 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
deepseek/deepseek-v4-flash-0731:none PASS 100% 162 in → 4 out (0 reasoning)
dev@org.io
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 162 in → 9 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 247 in → 193 out (191 reasoning)
dev@org.io
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 247 in → 63 out (53 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
deepseek/deepseek-v4-flash:high PASS 100% 162 in → 81 out (75 reasoning)
dev@org.io
deepseek/deepseek-v4-flash:low PASS 100% 162 in → 72 out (66 reasoning)
dev@org.io
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 164 in → 107 out (91 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
deepseek/deepseek-v4-flash:medium PASS 100% 162 in → 71 out (66 reasoning)
dev@org.io
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 164 in → 70 out (57 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
deepseek/deepseek-v4-flash:none PASS 100% 162 in → 5 out (0 reasoning)
dev@org.io
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 164 in → 11 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 241 in → 110 out (104 reasoning)
dev@org.io
deepseek/deepseek-v4-pro:high PASS 100% 162 in → 82 out (76 reasoning)
dev@org.io
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 164 in → 104 out (91 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
deepseek/deepseek-v4-pro:low PASS 100% 162 in → 97 out (91 reasoning)
dev@org.io
deepseek/deepseek-v4-pro:medium PASS 100% 162 in → 94 out (88 reasoning)
dev@org.io
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 164 in → 128 out (114 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
deepseek/deepseek-v4-pro:none PASS 100% 162 in → 5 out (0 reasoning)
dev@org.io
deepseek/deepseek-v4-pro:xhigh PASS 100% 241 in → 182 out (176 reasoning)
dev@org.io
google/gemini-3-flash-preview:high PASS 100% 203 in → 221 out (216 reasoning)
dev@org.io
google/gemini-3-flash-preview:high__json_schema PASS 100% 258 in → 349 out (340 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3-flash-preview:low PASS 100% 203 in → 114 out (109 reasoning)
dev@org.io
google/gemini-3-flash-preview:low__json_schema PASS 100% 204 in → 9 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3-flash-preview:medium PASS 100% 203 in → 237 out (232 reasoning)
dev@org.io
google/gemini-3-flash-preview:medium__json_schema PASS 100% 204 in → 295 out (286 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3-flash-preview:xhigh PASS 100% 203 in → 198 out (193 reasoning)
dev@org.io
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 258 in → 273 out (264 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 204 in → 280 out (275 reasoning)
dev@org.io
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 204 in → 201 out (192 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 203 in → 154 out (149 reasoning)
dev@org.io
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 204 in → 147 out (137 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 204 in → 197 out (192 reasoning)
dev@org.io
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 204 in → 316 out (307 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 204 in → 324 out (319 reasoning)
dev@org.io
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 204 in → 302 out (292 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
google/gemini-3.1-pro-preview:high PASS 100% 203 in → 319 out (314 reasoning)
dev@org.io
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 258 in → 106 out (97 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.1-pro-preview:low PASS 100% 203 in → 213 out (208 reasoning)
dev@org.io
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 258 in → 122 out (113 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.1-pro-preview:medium PASS 100% 203 in → 228 out (223 reasoning)
dev@org.io
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 258 in → 243 out (234 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 203 in → 279 out (274 reasoning)
dev@org.io
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 258 in → 130 out (121 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.5-flash-lite:high PASS 100% 204 in → 302 out (297 reasoning)
dev@org.io
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 258 in → 352 out (343 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.5-flash-lite:low PASS 100% 203 in → 5 out (0 reasoning)
dev@org.io
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 258 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
google/gemini-3.5-flash-lite:max PASS 100% 203 in → 301 out (296 reasoning)
dev@org.io
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 258 in → 429 out (419 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
google/gemini-3.5-flash-lite:medium PASS 100% 204 in → 221 out (216 reasoning)
dev@org.io
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 258 in → 270 out (261 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 203 in → 294 out (289 reasoning)
dev@org.io
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 204 in → 450 out (441 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.5-flash:high PASS 100% 203 in → 253 out (248 reasoning)
dev@org.io
google/gemini-3.5-flash:high__json_schema PASS 100% 258 in → 260 out (251 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.5-flash:low PASS 100% 203 in → 68 out (63 reasoning)
dev@org.io
google/gemini-3.5-flash:low__json_schema PASS 100% 258 in → 119 out (110 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.5-flash:medium PASS 100% 203 in → 310 out (305 reasoning)
dev@org.io
google/gemini-3.5-flash:medium__json_schema PASS 100% 258 in → 210 out (201 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.5-flash:xhigh PASS 100% 203 in → 338 out (333 reasoning)
dev@org.io
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 258 in → 202 out (193 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.6-flash:high PASS 100% 203 in → 286 out (281 reasoning)
dev@org.io
google/gemini-3.6-flash:high__json_schema PASS 100% 204 in → 238 out (229 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.6-flash:low PASS 100% 204 in → 73 out (68 reasoning)
dev@org.io
google/gemini-3.6-flash:low__json_schema PASS 100% 204 in → 9 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.6-flash:max PASS 100% 204 in → 303 out (298 reasoning)
dev@org.io
google/gemini-3.6-flash:max__json_schema PASS 100% 204 in → 271 out (262 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.6-flash:medium PASS 100% 204 in → 286 out (281 reasoning)
dev@org.io
google/gemini-3.6-flash:medium__json_schema PASS 100% 258 in → 237 out (228 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemini-3.6-flash:xhigh PASS 100% 203 in → 282 out (277 reasoning)
dev@org.io
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 258 in → 248 out (239 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
google/gemma-4-26b-a4b-it:high PASS 100% 219 in → 291 out (285 reasoning)
dev@org.io
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 218 in → 297 out (212 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
google/gemma-4-26b-a4b-it:low PASS 100% 218 in → 313 out (212 reasoning)
dev@org.io
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 219 in → 315 out (220 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
google/gemma-4-26b-a4b-it:medium PASS 100% 219 in → 273 out (267 reasoning)
dev@org.io
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 219 in → 292 out (205 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
google/gemma-4-26b-a4b-it:none PASS 100% 216 in → 6 out (0 reasoning)
dev@org.io
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 216 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 218 in → 269 out (191 reasoning)
dev@org.io
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 219 in → 275 out (188 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
google/gemma-4-31b-it:high PASS 100% 219 in → 227 out (153 reasoning)
dev@org.io
google/gemma-4-31b-it:high__json_schema PASS 100% 219 in → 267 out (177 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
google/gemma-4-31b-it:low PASS 100% 219 in → 215 out (160 reasoning)
dev@org.io
google/gemma-4-31b-it:low__json_schema PASS 100% 218 in → 231 out (144 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
google/gemma-4-31b-it:medium PASS 100% 219 in → 271 out (200 reasoning)
dev@org.io
google/gemma-4-31b-it:medium__json_schema PASS 100% 219 in → 271 out (183 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
google/gemma-4-31b-it:none__json_schema PASS 100% 216 in → 13 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
google/gemma-4-31b-it:xhigh PASS 100% 222 in → 250 out (244 reasoning)
dev@org.io
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 219 in → 247 out (160 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
ibm-granite/granite-4.1-8b PASS 100% 183 in → 5 out
dev@org.io
ibm-granite/granite-4.1-8b__json_schema PASS 100% 183 in → 12 out
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
inclusionai/ling-2.6-flash PASS 100% 216 in → 5 out
dev@org.io
inclusionai/ling-2.6-flash__json_schema PASS 100% 216 in → 15 out
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
liquid/lfm-2-24b-a2b PASS 100% 181 in → 5 out
dev@org.io
minimax/minimax-m2.5:high PASS 100% 195 in → 165 out (175 reasoning)
dev@org.io
minimax/minimax-m2.5:high__json_schema PASS 100% 195 in → 163 out (150 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
minimax/minimax-m2.5:low PASS 100% 195 in → 122 out (107 reasoning)
dev@org.io
minimax/minimax-m2.5:low__json_schema PASS 100% 195 in → 101 out (89 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
minimax/minimax-m2.5:medium PASS 100% 195 in → 160 out (155 reasoning)
dev@org.io
minimax/minimax-m2.5:medium__json_schema PASS 100% 213 in → 179 out (166 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
minimax/minimax-m2.5:xhigh PASS 100% 195 in → 163 out (155 reasoning)
dev@org.io
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 213 in → 150 out (137 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
minimax/minimax-m2.7:high PASS 100% 197 in → 137 out (132 reasoning)
dev@org.io
minimax/minimax-m2.7:high__json_schema PASS 100% 324 in → 314 out (304 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
minimax/minimax-m2.7:low PASS 100% 195 in → 102 out (104 reasoning)
dev@org.io
minimax/minimax-m2.7:low__json_schema PASS 100% 324 in → 321 out (311 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
minimax/minimax-m2.7:medium PASS 100% 195 in → 157 out (162 reasoning)
dev@org.io
minimax/minimax-m2.7:xhigh PASS 100% 195 in → 156 out (168 reasoning)
dev@org.io
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 324 in → 327 out (317 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
minimax/minimax-m3:high PASS 100% 333 in → 35 out (35 reasoning)
dev@org.io
minimax/minimax-m3:high__json_schema PASS 100% 320 in → 10 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
minimax/minimax-m3:low PASS 100% 333 in → 59 out (55 reasoning)
dev@org.io
minimax/minimax-m3:low__json_schema PASS 100% 320 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
minimax/minimax-m3:medium PASS 100% 333 in → 106 out (86 reasoning)
dev@org.io
minimax/minimax-m3:medium__json_schema PASS 100% 281 in → 12 out (11 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
minimax/minimax-m3:xhigh PASS 100% 333 in → 54 out (54 reasoning)
dev@org.io
minimax/minimax-m3:xhigh__json_schema PASS 100% 320 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
mistralai/devstral-2512 PASS 100% 192 in → 5 out
dev@org.io
mistralai/devstral-2512__json_schema PASS 100% 192 in → 10 out
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
mistralai/mistral-medium-3-5:high PASS 100% 204 in → 179 out (177 reasoning)
dev@org.io
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 192 in → 289 out (245 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
mistralai/mistral-medium-3-5:low PASS 100% 204 in → 151 out (122 reasoning)
dev@org.io
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 192 in → 234 out (199 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
mistralai/mistral-medium-3-5:medium PASS 100% 204 in → 345 out (333 reasoning)
dev@org.io
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 192 in → 129 out (125 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
mistralai/mistral-medium-3-5:none PASS 100% 204 in → 5 out (0 reasoning)
dev@org.io
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 192 in → 10 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 204 in → 228 out (201 reasoning)
dev@org.io
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 192 in → 166 out (139 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
moonshotai/kimi-k2.6:high PASS 100% 165 in → 151 out (132 reasoning)
dev@org.io
moonshotai/kimi-k2.6:high__json_schema PASS 100% 164 in → 865 out (102 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
moonshotai/kimi-k2.6:low PASS 100% 164 in → 121 out (109 reasoning)
dev@org.io
moonshotai/kimi-k2.6:low__json_schema PASS 100% 164 in → 230 out (158 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
moonshotai/kimi-k2.6:medium PASS 100% 164 in → 139 out (133 reasoning)
dev@org.io
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 164 in → 413 out (142 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
moonshotai/kimi-k2.6:xhigh PASS 100% 165 in → 128 out (124 reasoning)
dev@org.io
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 164 in → 299 out (180 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
moonshotai/kimi-k2.7-code:high PASS 100% 165 in → 101 out (95 reasoning)
dev@org.io
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 165 in → 153 out (139 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
moonshotai/kimi-k2.7-code:low PASS 100% 164 in → 77 out (71 reasoning)
dev@org.io
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 245 in → 88 out (78 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
moonshotai/kimi-k2.7-code:medium PASS 100% 165 in → 116 out (110 reasoning)
dev@org.io
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 245 in → 114 out (104 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 165 in → 96 out (90 reasoning)
dev@org.io
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 165 in → 133 out (120 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
moonshotai/kimi-k3:high PASS 100% 242 in → 114 out (95 reasoning)
dev@org.io
moonshotai/kimi-k3:high__json_schema PASS 100% 242 in → 136 out (117 reasoning)
dev@org.io
moonshotai/kimi-k3:low PASS 100% 242 in → 121 out (102 reasoning)
dev@org.io
moonshotai/kimi-k3:low__json_schema PASS 100% 335 in → 172 out (149 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
moonshotai/kimi-k3:max PASS 100% 242 in → 104 out (85 reasoning)
dev@org.io
moonshotai/kimi-k3:max__json_schema PASS 100% 242 in → 159 out (140 reasoning)
dev@org.io
moonshotai/kimi-k3:medium PASS 100% 242 in → 127 out (108 reasoning)
dev@org.io
moonshotai/kimi-k3:medium__json_schema PASS 100% 242 in → 99 out (80 reasoning)
dev@org.io
moonshotai/kimi-k3:xhigh PASS 100% 242 in → 122 out (103 reasoning)
dev@org.io
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 335 in → 107 out (84 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 205 in → 268 out (204 reasoning)
dev@org.io
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 205 in → 66 out (54 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 205 in → 76 out (71 reasoning)
dev@org.io
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 205 in → 122 out (107 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 205 in → 101 out (96 reasoning)
dev@org.io
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 205 in → 75 out (67 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 205 in → 5 out (0 reasoning)
dev@org.io
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 205 in → 13 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email" : "dev@org.io" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 205 in → 519 out (552 reasoning)
dev@org.io
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 205 in → 160 out (164 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 205 in → 87 out (68 reasoning)
dev@org.io
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 205 in → 38 out (25 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 205 in → 107 out (81 reasoning)
dev@org.io
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 205 in → 108 out (82 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 205 in → 116 out (89 reasoning)
dev@org.io
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 205 in → 63 out (49 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 205 in → 5 out (0 reasoning)
dev@org.io
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 205 in → 13 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 205 in → 100 out (81 reasoning)
dev@org.io
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 205 in → 100 out (75 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
openai/gpt-5.4-mini:high PASS 100% 163 in → 47 out (37 reasoning)
dev@org.io
openai/gpt-5.4-mini:high__json_schema PASS 100% 194 in → 249 out (232 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4-mini:low PASS 100% 163 in → 28 out (18 reasoning)
dev@org.io
openai/gpt-5.4-mini:low__json_schema PASS 100% 194 in → 80 out (63 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4-mini:medium PASS 100% 163 in → 43 out (33 reasoning)
dev@org.io
openai/gpt-5.4-mini:medium__json_schema PASS 100% 194 in → 110 out (93 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4-mini:none PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.4-mini:none__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4-mini:xhigh PASS 100% 163 in → 123 out (113 reasoning)
dev@org.io
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 194 in → 118 out (101 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4-nano:high PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.4-nano:high__json_schema PASS 100% 194 in → 53 out (36 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4-nano:low PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.4-nano:low__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4-nano:medium PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.4-nano:medium__json_schema PASS 100% 194 in → 48 out (31 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4-nano:none PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.4-nano:none__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4-nano:xhigh PASS 100% 163 in → 68 out (58 reasoning)
dev@org.io
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 194 in → 72 out (55 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4:high PASS 100% 163 in → 48 out (38 reasoning)
dev@org.io
openai/gpt-5.4:high__json_schema PASS 100% 194 in → 58 out (41 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4:low PASS 100% 163 in → 24 out (14 reasoning)
dev@org.io
openai/gpt-5.4:low__json_schema PASS 100% 194 in → 33 out (16 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4:medium PASS 100% 163 in → 47 out (37 reasoning)
dev@org.io
openai/gpt-5.4:medium__json_schema PASS 100% 194 in → 247 out (230 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4:none PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.4:none__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.4:xhigh PASS 100% 163 in → 74 out (64 reasoning)
dev@org.io
openai/gpt-5.4:xhigh__json_schema PASS 100% 194 in → 1,215 out (1,198 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.5:high PASS 100% 163 in → 63 out (53 reasoning)
dev@org.io
openai/gpt-5.5:high__json_schema PASS 100% 194 in → 198 out (181 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.5:low PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.5:low__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.5:medium PASS 100% 163 in → 36 out (26 reasoning)
dev@org.io
openai/gpt-5.5:medium__json_schema PASS 100% 194 in → 105 out (88 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.5:none PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.5:none__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.5:xhigh PASS 100% 163 in → 89 out (79 reasoning)
dev@org.io
openai/gpt-5.5:xhigh__json_schema PASS 100% 194 in → 189 out (172 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-luna:high PASS 100% 163 in → 34 out (24 reasoning)
dev@org.io
openai/gpt-5.6-luna:high__json_schema PASS 100% 194 in → 88 out (71 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-luna:low PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-luna:low__json_schema PASS 100% 194 in → 81 out (64 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-luna:max PASS 100% 163 in → 36 out (26 reasoning)
dev@org.io
openai/gpt-5.6-luna:max__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-luna:medium PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-luna:medium__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-luna:none PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-luna:none__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-luna:xhigh PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 194 in → 302 out (285 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-sol:high PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-sol:high__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-sol:low PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-sol:low__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-sol:max PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-sol:max__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-sol:medium PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-sol:medium__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-sol:none PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-sol:none__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-sol:xhigh PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-terra:high PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-terra:high__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-terra:low PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-terra:low__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-terra:max PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-terra:max__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-terra:medium PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-terra:medium__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-terra:none PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-terra:none__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-5.6-terra:xhigh PASS 100% 163 in → 8 out (0 reasoning)
dev@org.io
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 194 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-oss-120b:high PASS 100% 235 in → 203 out (0 reasoning)
dev@org.io
openai/gpt-oss-120b:high__json_schema PASS 100% 211 in → 154 out (120 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-oss-120b:low PASS 100% 226 in → 90 out (81 reasoning)
dev@org.io
openai/gpt-oss-120b:low__json_schema PASS 100% 222 in → 35 out (17 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-oss-120b:medium PASS 100% 224 in → 68 out (64 reasoning)
dev@org.io
openai/gpt-oss-120b:medium__json_schema PASS 100% 224 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
openai/gpt-oss-120b:xhigh PASS 100% 211 in → 204 out (187 reasoning)
dev@org.io
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 211 in → 157 out (124 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
openai/gpt-oss-20b:high PASS 100% 211 in → 282 out (255 reasoning)
dev@org.io
openai/gpt-oss-20b:high__json_schema PASS 100% 226 in → 253 out (237 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
openai/gpt-oss-20b:low PASS 100% 229 in → 30 out (13 reasoning)
dev@org.io
openai/gpt-oss-20b:low__json_schema PASS 100% 224 in → 22 out (10 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
openai/gpt-oss-20b:medium PASS 100% 211 in → 76 out (59 reasoning)
dev@org.io
openai/gpt-oss-20b:medium__json_schema PASS 100% 224 in → 93 out (72 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
openai/gpt-oss-20b:xhigh PASS 100% 224 in → 115 out (101 reasoning)
dev@org.io
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 222 in → 253 out (237 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
poolside/laguna-m.1:high PASS 100% 208 in → 377 out (371 reasoning)
dev@org.io
poolside/laguna-m.1:high__json_schema PASS 100% 208 in → 349 out (333 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
poolside/laguna-m.1:low PASS 100% 208 in → 410 out (405 reasoning)
dev@org.io
poolside/laguna-m.1:low__json_schema PASS 100% 208 in → 345 out (332 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
poolside/laguna-m.1:medium PASS 100% 208 in → 332 out (325 reasoning)
dev@org.io
poolside/laguna-m.1:medium__json_schema PASS 100% 208 in → 278 out (268 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
poolside/laguna-m.1:none PASS 100% 208 in → 6 out (0 reasoning)
dev@org.io
poolside/laguna-m.1:none__json_schema PASS 100% 208 in → 10 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
poolside/laguna-m.1:xhigh PASS 100% 208 in → 215 out (209 reasoning)
dev@org.io
poolside/laguna-m.1:xhigh__json_schema PASS 100% 208 in → 253 out (238 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
poolside/laguna-xs-2.1:high PASS 100% 208 in → 211 out (207 reasoning)
dev@org.io
poolside/laguna-xs-2.1:high__json_schema PASS 100% 208 in → 277 out (262 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
poolside/laguna-xs-2.1:low PASS 100% 208 in → 248 out (244 reasoning)
dev@org.io
poolside/laguna-xs-2.1:low__json_schema PASS 100% 208 in → 275 out (265 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
poolside/laguna-xs-2.1:medium PASS 100% 208 in → 319 out (315 reasoning)
dev@org.io
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 208 in → 233 out (223 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
poolside/laguna-xs-2.1:none__json_schema PASS 100% 208 in → 10 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
poolside/laguna-xs-2.1:xhigh PASS 100% 208 in → 253 out (249 reasoning)
dev@org.io
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 208 in → 254 out (239 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
poolside/laguna-xs.2:high PASS 100% 245 in → 157 out (151 reasoning)
dev@org.io
poolside/laguna-xs.2:high__json_schema PASS 100% 245 in → 184 out (174 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
poolside/laguna-xs.2:low PASS 100% 245 in → 149 out (143 reasoning)
dev@org.io
poolside/laguna-xs.2:low__json_schema PASS 100% 245 in → 154 out (143 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
poolside/laguna-xs.2:medium PASS 100% 245 in → 133 out (127 reasoning)
dev@org.io
poolside/laguna-xs.2:medium__json_schema PASS 100% 245 in → 151 out (142 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
poolside/laguna-xs.2:none PASS 100% 245 in → 6 out (0 reasoning)
dev@org.io
poolside/laguna-xs.2:none__json_schema PASS 100% 245 in → 10 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
poolside/laguna-xs.2:xhigh PASS 100% 245 in → 172 out (166 reasoning)
dev@org.io
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 245 in → 294 out (285 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
qwen/qwen3.6-27b:high PASS 100% 204 in → 197 out (153 reasoning)
dev@org.io
qwen/qwen3.6-27b:high__json_schema PASS 100% 204 in → 211 out (160 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
qwen/qwen3.6-27b:low PASS 100% 204 in → 448 out (396 reasoning)
dev@org.io
qwen/qwen3.6-27b:low__json_schema PASS 100% 204 in → 199 out (1 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
qwen/qwen3.6-27b:medium PASS 100% 204 in → 171 out (1 reasoning)
dev@org.io
qwen/qwen3.6-27b:medium__json_schema PASS 100% 204 in → 182 out (168 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
qwen/qwen3.6-27b:none PASS 100% 206 in → 5 out (0 reasoning)
dev@org.io
qwen/qwen3.6-27b:none__json_schema PASS 100% 206 in → 14 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
qwen/qwen3.6-27b:xhigh PASS 100% 204 in → 406 out (324 reasoning)
dev@org.io
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 204 in → 687 out (576 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
qwen/qwen3.6-35b-a3b:high PASS 100% 204 in → 221 out (181 reasoning)
dev@org.io
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 204 in → 205 out (162 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
qwen/qwen3.6-35b-a3b:low PASS 100% 204 in → 179 out (161 reasoning)
dev@org.io
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 204 in → 256 out (195 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 204 in → 121 out (116 reasoning)
dev@org.io
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 204 in → 173 out (161 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
qwen/qwen3.6-35b-a3b:none PASS 100% 206 in → 4 out (0 reasoning)
dev@org.io
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 206 in → 15 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 204 in → 238 out (194 reasoning)
dev@org.io
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 204 in → 111 out (93 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
qwen/qwen3.6-flash:high PASS 100% 204 in → 195 out (186 reasoning)
dev@org.io
qwen/qwen3.6-flash:low PASS 100% 204 in → 367 out (358 reasoning)
dev@org.io
qwen/qwen3.6-flash:medium PASS 100% 204 in → 132 out (123 reasoning)
dev@org.io
qwen/qwen3.6-flash:none PASS 100% 206 in → 4 out (0 reasoning)
dev@org.io
qwen/qwen3.6-flash:xhigh PASS 100% 204 in → 233 out (224 reasoning)
dev@org.io
qwen/qwen3.7-flash:high PASS 100% 204 in → 109 out (100 reasoning)
dev@org.io
qwen/qwen3.7-flash:low PASS 100% 204 in → 157 out (148 reasoning)
dev@org.io
qwen/qwen3.7-flash:max PASS 100% 204 in → 185 out (176 reasoning)
dev@org.io
qwen/qwen3.7-flash:medium PASS 100% 204 in → 116 out (107 reasoning)
dev@org.io
qwen/qwen3.7-flash:none PASS 100% 206 in → 4 out (0 reasoning)
dev@org.io
qwen/qwen3.7-flash:xhigh PASS 100% 204 in → 236 out (227 reasoning)
dev@org.io
qwen/qwen3.7-max:high PASS 100% 204 in → 133 out (124 reasoning)
dev@org.io
qwen/qwen3.7-max:low PASS 100% 204 in → 114 out (105 reasoning)
dev@org.io
qwen/qwen3.7-max:medium PASS 100% 204 in → 136 out (127 reasoning)
dev@org.io
qwen/qwen3.7-max:none PASS 100% 206 in → 4 out (0 reasoning)
dev@org.io
qwen/qwen3.7-max:xhigh PASS 100% 204 in → 340 out (331 reasoning)
dev@org.io
qwen/qwen3.7-plus:high PASS 100% 204 in → 134 out (125 reasoning)
dev@org.io
qwen/qwen3.7-plus:low PASS 100% 204 in → 433 out (424 reasoning)
dev@org.io
qwen/qwen3.7-plus:medium PASS 100% 204 in → 158 out (149 reasoning)
dev@org.io
qwen/qwen3.7-plus:none PASS 100% 206 in → 4 out (0 reasoning)
dev@org.io
qwen/qwen3.7-plus:xhigh PASS 100% 204 in → 121 out (112 reasoning)
dev@org.io
tencent/hy3:high PASS 100% 169 in → 208 out (202 reasoning)
dev@org.io
tencent/hy3:high__json_schema PASS 100% 169 in → 140 out (105 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
tencent/hy3:low PASS 100% 169 in → 209 out (203 reasoning)
dev@org.io
tencent/hy3:low__json_schema PASS 100% 169 in → 181 out (146 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
tencent/hy3:medium PASS 100% 169 in → 190 out (158 reasoning)
dev@org.io
tencent/hy3:medium__json_schema PASS 100% 169 in → 192 out (152 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
tencent/hy3:none PASS 100% 172 in → 5 out (0 reasoning)
dev@org.io
tencent/hy3:none__json_schema PASS 100% 172 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
tencent/hy3:xhigh PASS 100% 169 in → 216 out (197 reasoning)
dev@org.io
tencent/hy3:xhigh__json_schema PASS 100% 169 in → 154 out (122 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
thinkingmachines/inkling-small:high PASS 100% 172 in → 67 out (58 reasoning)
dev@org.io
thinkingmachines/inkling-small:high__json_schema PASS 100% 172 in → 25 out (8 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
thinkingmachines/inkling-small:low PASS 100% 172 in → 18 out (8 reasoning)
dev@org.io
thinkingmachines/inkling-small:low__json_schema PASS 100% 172 in → 24 out (8 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
thinkingmachines/inkling-small:max PASS 100% 172 in → 70 out (58 reasoning)
dev@org.io
thinkingmachines/inkling-small:max__json_schema PASS 100% 172 in → 67 out (54 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
thinkingmachines/inkling-small:medium PASS 100% 172 in → 61 out (51 reasoning)
dev@org.io
thinkingmachines/inkling-small:medium__json_schema PASS 100% 172 in → 21 out (8 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
thinkingmachines/inkling-small:xhigh PASS 100% 172 in → 60 out (47 reasoning)
dev@org.io
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 172 in → 188 out (174 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
thinkingmachines/inkling:high PASS 100% 172 in → 42 out (32 reasoning)
dev@org.io
thinkingmachines/inkling:low PASS 100% 172 in → 20 out (10 reasoning)
dev@org.io
thinkingmachines/inkling:max PASS 100% 172 in → 87 out (79 reasoning)
dev@org.io
thinkingmachines/inkling:max__json_schema PASS 100% 172 in → 95 out (78 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
thinkingmachines/inkling:medium PASS 100% 172 in → 49 out (37 reasoning)
dev@org.io
thinkingmachines/inkling:medium__json_schema PASS 100% 172 in → 53 out (37 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
thinkingmachines/inkling:none PASS 100% 170 in → 7 out (0 reasoning)
dev@org.io
thinkingmachines/inkling:none__json_schema PASS 100% 170 in → 13 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
thinkingmachines/inkling:xhigh PASS 100% 172 in → 80 out (73 reasoning)
dev@org.io
thinkingmachines/inkling:xhigh__json_schema PASS 100% 172 in → 80 out (63 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
x-ai/grok-4.3:high PASS 100% 341 in → 162 out (158 reasoning)
dev@org.io
x-ai/grok-4.3:high__json_schema PASS 100% 394 in → 294 out (286 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
x-ai/grok-4.3:low PASS 100% 347 in → 268 out (264 reasoning)
dev@org.io
x-ai/grok-4.3:low__json_schema PASS 100% 400 in → 340 out (328 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
x-ai/grok-4.3:max PASS 100% 341 in → 142 out (138 reasoning)
dev@org.io
x-ai/grok-4.3:max__json_schema PASS 100% 394 in → 415 out (407 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
x-ai/grok-4.3:medium PASS 100% 347 in → 173 out (169 reasoning)
dev@org.io
x-ai/grok-4.3:medium__json_schema PASS 100% 400 in → 443 out (435 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
x-ai/grok-4.3:none PASS 100% 339 in → 4 out (0 reasoning)
dev@org.io
x-ai/grok-4.3:none__json_schema PASS 100% 392 in → 8 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
x-ai/grok-4.3:xhigh PASS 100% 341 in → 179 out (175 reasoning)
dev@org.io
x-ai/grok-4.3:xhigh__json_schema PASS 100% 394 in → 255 out (247 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
x-ai/grok-4.5:high PASS 100% 370 in → 263 out (259 reasoning)
dev@org.io
x-ai/grok-4.5:high__json_schema PASS 100% 433 in → 307 out (298 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
x-ai/grok-4.5:low PASS 100% 370 in → 123 out (119 reasoning)
dev@org.io
x-ai/grok-4.5:low__json_schema PASS 100% 433 in → 262 out (253 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
x-ai/grok-4.5:max PASS 100% 370 in → 256 out (252 reasoning)
dev@org.io
x-ai/grok-4.5:max__json_schema PASS 100% 433 in → 212 out (203 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
x-ai/grok-4.5:medium PASS 100% 370 in → 229 out (225 reasoning)
dev@org.io
x-ai/grok-4.5:medium__json_schema PASS 100% 433 in → 266 out (257 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
x-ai/grok-4.5:xhigh PASS 100% 370 in → 177 out (173 reasoning)
dev@org.io
x-ai/grok-4.5:xhigh__json_schema PASS 100% 433 in → 317 out (308 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
z-ai/glm-4.7-flash:high PASS 100% 168 in → 602 out (516 reasoning)
dev@org.io
z-ai/glm-4.7-flash:high__json_schema PASS 100% 168 in → 381 out (337 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
z-ai/glm-4.7-flash:low PASS 100% 168 in → 340 out (334 reasoning)
dev@org.io
z-ai/glm-4.7-flash:medium PASS 100% 168 in → 469 out (439 reasoning)
dev@org.io
z-ai/glm-4.7-flash:none PASS 100% 168 in → 4 out (0 reasoning)
dev@org.io
z-ai/glm-4.7-flash:none__json_schema PASS 100% 168 in → 10 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email": "dev@org.io"}
z-ai/glm-4.7-flash:xhigh PASS 100% 168 in → 384 out (336 reasoning)
dev@org.io
z-ai/glm-5.2:high PASS 100% 175 in → 246 out (240 reasoning)
dev@org.io
z-ai/glm-5.2:low PASS 100% 175 in → 18 out (12 reasoning)
dev@org.io
z-ai/glm-5.2:low__json_schema PASS 100% 346 in → 24 out (14 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) {"email":"dev@org.io"}
z-ai/glm-5.2:medium PASS 100% 175 in → 118 out (112 reasoning)
dev@org.io
z-ai/glm-5.2:medium__json_schema PASS 100% 175 in → 68 out (60 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
z-ai/glm-5.2:none PASS 100% 169 in → 4 out (0 reasoning)
dev@org.io
z-ai/glm-5.2:none__json_schema PASS 100% 169 in → 12 out (0 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
z-ai/glm-5.2:xhigh PASS 100% 175 in → 242 out (236 reasoning)
dev@org.io
z-ai/glm-5.2:xhigh__json_schema PASS 100% 175 in → 260 out (207 reasoning)
dev@org.io
JSON Schema Structured Output
(raw) { "email": "dev@org.io" }
anthropic/claude-fable-5:max__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
anthropic/claude-fable-5:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:high__json_schema FAIL 0%
Invalid JSON. Output: dev@org.io
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output: dev@org.io
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:low__json_schema FAIL 0%
Invalid JSON. Output: dev@org.io
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:none__json_schema FAIL 0%
Invalid JSON. Output: dev@org.io
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0%
Invalid JSON. Output: dev@org.io
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
google/gemma-4-31b-it:none FAIL 0%
(empty output)
Failure: Model 'google/gemma-4-31b-it:none' violated the reasoning_level=none invariant: provider reported 202 reasoning token(s)
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: dev@org.io
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
poolside/laguna-xs-2.1:none FAIL 0% 208 in → 10 out (0 reasoning)
dev@org.io </think>dev@org.io
Failure: Expected 'dev@org.io', got 'dev@org.io </think>dev@org.io'
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5ca02362-eb00-9596-a8af-5160fe81ace9","request_id":"5ca02362-eb00-9596-a8af-5160fe81ace9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-870e9fbd-d268-9f2f-9c18-f9bf5b1264ab","request_id":"870e9fbd-d268-9f2f-9c18-f9bf5b1264ab"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-382c896c-4b31-9cef-836a-de1b206169ca","request_id":"382c896c-4b31-9cef-836a-de1b206169ca"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-13c3e085-f4d7-9099-8397-57948aa4a63c","request_id":"13c3e085-f4d7-9099-8397-57948aa4a63c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-41ba7b06-dcc6-9465-a811-8ca79b411e8c","request_id":"41ba7b06-dcc6-9465-a811-8ca79b411e8c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-48e6bc02-c92c-9680-a29e-898d97e4a6ea"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7345f9f6-0648-9b14-8422-275673286584"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-08146008-0678-91e8-bfb8-6030cdea434c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a03ef5b2-d13b-9dc9-8252-74b77f851ef9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8b45b3a1-8efd-9b51-8f4c-217094bf70e6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c8f9b1c5-b28d-972d-897b-e4755101a7b6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-74b82be3-56d6-906c-a984-712f4d68f8a3","request_id":"74b82be3-56d6-906c-a984-712f4d68f8a3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cb63ce40-5bf0-9d86-830d-9d47019acfe8","request_id":"cb63ce40-5bf0-9d86-830d-9d47019acfe8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cb1896f5-3ef1-98b8-9f2d-6b6ca2c91b75","request_id":"cb1896f5-3ef1-98b8-9f2d-6b6ca2c91b75"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3245c795-b365-928b-9e4c-cf74bf6d4a70","request_id":"3245c795-b365-928b-9e4c-cf74bf6d4a70"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bbe62792-e8dd-9165-8108-b257cd1ca88e","request_id":"bbe62792-e8dd-9165-8108-b257cd1ca88e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a3392a34-7370-904c-a92f-68351ba4718c","request_id":"a3392a34-7370-904c-a92f-68351ba4718c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d9691278-ea8e-948e-8745-2e037093a30f","request_id":"d9691278-ea8e-948e-8745-2e037093a30f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-65319f1d-ca36-96db-8b5b-d0198d67e639","request_id":"65319f1d-ca36-96db-8b5b-d0198d67e639"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-36391466-25bd-9bdd-b050-cb6ab9c90a64","request_id":"36391466-25bd-9bdd-b050-cb6ab9c90a64"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-790109d7-6e9f-9f50-9707-234986040eef","request_id":"790109d7-6e9f-9f50-9707-234986040eef"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 172 in → 7 out (0 reasoning)
dev@org.io<|end_message|>
Failure: Expected 'dev@org.io', got 'dev@org.io<|end_message|>'
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: dev@org.io{ "email": "dev@org.io" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: dev@org.io
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: dev@org.io
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)