Deepseek / deepseek-v4-pro

medium

none low medium high xhigh

93.6% 191 / 204 fixtures · 1 run(s)

34,710 input / 76,546 total output / 72,687 reasoning within output tokens $0.22833374

Reliability by Benchmark (Text)

Loading reliability summary…

Text vs JSON Schema Comparison

Pass Rate Delta

-11.3% Text: 93.6% → JSON: 82.4%

+9

Gained

JSON pass / text fail

−32

Lost

Text pass / JSON fail

159

Unchanged Pass

Both pass

4

Unchanged Fail

Both fail

Fixture Reliability Delta

Fixture	Text	JSON	Delta

Benchmark Deltas

Benchmark	Text	JSON	Delta
blame_forensics	91.7%	50%	-41.7%
commit_squash	91.7%	50%	-41.7%
worktree_usage	58.3%	100%	+ 41.7%
branch_cleanup	100%	66.7%	-33.3%
git_show	100%	66.7%	-33.3%
git_grep	100%	75%	-25%
git_clean	100%	83.3%	-16.7%
tag_management	100%	83.3%	-16.7%
merge_conflicts	83.3%	66.7%	-16.7%
cherry_pick	83.3%	91.7%	+ 8.3%
rebase	83.3%	91.7%	+ 8.3%
git_log_format	100%	91.7%	-8.3%
reflog	100%	91.7%	-8.3%
submodule_usage	100%	91.7%	-8.3%
commit_messages	100%	100%	+ 0%
git_bisect	100%	100%	+ 0%
stash_recovery	100%	100%	+ 0%

Changed Fixtures (41)

Fixture Gallery (204)

blame_forensics f001

Reasoning within output

blame_forensics f002

Reasoning within output

blame_forensics f003

Reasoning within output

blame_forensics f004

Reasoning within output

blame_forensics f005

Reasoning within output

blame_forensics f006

Reasoning within output

blame_forensics f007

Reasoning within output

blame_forensics f008

Reasoning within output

blame_forensics f009

Reasoning within output

blame_forensics f010

Reasoning within output

blame_forensics f011

Reasoning within output

blame_forensics f012

Reasoning within output

branch_cleanup f001

Reasoning within output

branch_cleanup f002

Reasoning within output

branch_cleanup f003

Reasoning within output

branch_cleanup f004

Reasoning within output

branch_cleanup f005

Reasoning within output

branch_cleanup f006

Reasoning within output

branch_cleanup f007

Reasoning within output

branch_cleanup f008

Reasoning within output

branch_cleanup f009

Reasoning within output

branch_cleanup f010

Reasoning within output

branch_cleanup f011

Reasoning within output

branch_cleanup f012

Reasoning within output

cherry_pick f001

Reasoning within output

cherry_pick f002

Reasoning within output

cherry_pick f003

Reasoning within output

cherry_pick f004

Reasoning within output

cherry_pick f005

Reasoning within output

cherry_pick f006

Reasoning within output

cherry_pick f007

Reasoning within output

cherry_pick f008

Reasoning within output

cherry_pick f009

Reasoning within output

cherry_pick f010

Reasoning within output

cherry_pick f011

Reasoning within output

cherry_pick f012

Reasoning within output

commit_messages f001

Reasoning within output

commit_messages f002

Reasoning within output

commit_messages f003

Reasoning within output

commit_messages f004

Reasoning within output

commit_messages f005

Reasoning within output

commit_messages f006

Reasoning within output

commit_messages f007

Reasoning within output

commit_messages f008

Reasoning within output

commit_messages f009

Reasoning within output

commit_messages f010

Reasoning within output

commit_messages f011

Reasoning within output

commit_messages f012

Reasoning within output

commit_squash f001

Reasoning within output

commit_squash f002

Reasoning within output

commit_squash f003

Reasoning within output

commit_squash f004

Reasoning within output

commit_squash f005

Reasoning within output

commit_squash f006

Reasoning within output

commit_squash f007

Reasoning within output

commit_squash f008

Reasoning within output

commit_squash f009

Reasoning within output

commit_squash f010

Reasoning within output

commit_squash f011

Reasoning within output

commit_squash f012

Reasoning within output

git_bisect f001

Reasoning within output

git_bisect f002

Reasoning within output

git_bisect f003

Reasoning within output

git_bisect f004

Reasoning within output

git_bisect f005

Reasoning within output

git_bisect f006

Reasoning within output

git_bisect f007

Reasoning within output

git_bisect f008

Reasoning within output

git_bisect f009

Reasoning within output

git_bisect f010

Reasoning within output

git_bisect f011

Reasoning within output

git_bisect f012

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

git_log_format f001

Reasoning within output

git_log_format f002

Reasoning within output

git_log_format f003

Reasoning within output

git_log_format f004

Reasoning within output

git_log_format f005

Reasoning within output

git_log_format f006

Reasoning within output

git_log_format f007

Reasoning within output

git_log_format f008

Reasoning within output

git_log_format f009

Reasoning within output

git_log_format f010

Reasoning within output

git_log_format f011

Reasoning within output

git_log_format f012

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

merge_conflicts f001

Reasoning within output

merge_conflicts f002

Reasoning within output

merge_conflicts f003

Reasoning within output

merge_conflicts f004

Reasoning within output

merge_conflicts f005

Reasoning within output

merge_conflicts f006

Reasoning within output

merge_conflicts f007

Reasoning within output

merge_conflicts f008

Reasoning within output

merge_conflicts f009

Reasoning within output

merge_conflicts f010

Reasoning within output

merge_conflicts f011

Reasoning within output

merge_conflicts f012

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

stash_recovery f001

Reasoning within output

stash_recovery f002

Reasoning within output

stash_recovery f003

Reasoning within output

stash_recovery f004

Reasoning within output

stash_recovery f005

Reasoning within output

stash_recovery f006

Reasoning within output

stash_recovery f007

Reasoning within output

stash_recovery f008

Reasoning within output

stash_recovery f009

Reasoning within output

stash_recovery f010

Reasoning within output

stash_recovery f011

Reasoning within output

stash_recovery f012

Reasoning within output

submodule_usage f001

Reasoning within output

submodule_usage f002

Reasoning within output

submodule_usage f003

Reasoning within output

submodule_usage f004

Reasoning within output

submodule_usage f005

Reasoning within output

submodule_usage f006

Reasoning within output

submodule_usage f007

Reasoning within output

submodule_usage f008

Reasoning within output

submodule_usage f009

Reasoning within output

submodule_usage f010

Reasoning within output

submodule_usage f011

Reasoning within output

submodule_usage f012

Reasoning within output

tag_management f001

Reasoning within output

tag_management f002

Reasoning within output

tag_management f003

Reasoning within output

tag_management f004

Reasoning within output

tag_management f005

Reasoning within output

tag_management f006

Reasoning within output

tag_management f007

Reasoning within output

tag_management f008

Reasoning within output

tag_management f009

Reasoning within output

tag_management f010

Reasoning within output

tag_management f011

Reasoning within output

tag_management f012

Reasoning within output

worktree_usage f001

Reasoning within output

worktree_usage f002

Reasoning within output

worktree_usage f003

Reasoning within output

worktree_usage f004

Reasoning within output

worktree_usage f005

Reasoning within output

worktree_usage f006

Reasoning within output

worktree_usage f007

Reasoning within output

worktree_usage f008

Reasoning within output

worktree_usage f009

Reasoning within output

worktree_usage f010

Reasoning within output

worktree_usage f011

Reasoning within output

worktree_usage f012

Reasoning within output

blame_forensics f001

blame_forensics f002

blame_forensics f003

Reasoning within output

blame_forensics f004

Reasoning within output

blame_forensics f005

Reasoning within output

blame_forensics f006

blame_forensics f007

Reasoning within output

blame_forensics f008

blame_forensics f009

Reasoning within output

blame_forensics f010

blame_forensics f011

Reasoning within output

blame_forensics f012

branch_cleanup f001

Reasoning within output

branch_cleanup f002

Reasoning within output

branch_cleanup f003

Reasoning within output

branch_cleanup f004

Reasoning within output

branch_cleanup f005

Reasoning within output

branch_cleanup f006

Reasoning within output

branch_cleanup f007

Reasoning within output

branch_cleanup f008

Reasoning within output

branch_cleanup f009

Reasoning within output

branch_cleanup f010

Reasoning within output

branch_cleanup f011

Reasoning within output

branch_cleanup f012

Reasoning within output

cherry_pick f001

Reasoning within output

cherry_pick f002

Reasoning within output

cherry_pick f003

Reasoning within output

cherry_pick f004

Reasoning within output

cherry_pick f005

Reasoning within output

cherry_pick f006

Reasoning within output

cherry_pick f007

Reasoning within output

cherry_pick f008

Reasoning within output

cherry_pick f009

Reasoning within output

cherry_pick f010

Reasoning within output

cherry_pick f011

Reasoning within output

cherry_pick f012

Reasoning within output

commit_messages f001

Reasoning within output

commit_messages f002

Reasoning within output

commit_messages f003

Reasoning within output

commit_messages f004

Reasoning within output

commit_messages f005

Reasoning within output

commit_messages f006

Reasoning within output

commit_messages f007

Reasoning within output

commit_messages f008

Reasoning within output

commit_messages f009

Reasoning within output

commit_messages f010

Reasoning within output

commit_messages f011

Reasoning within output

commit_messages f012

Reasoning within output

commit_squash f001

Reasoning within output

commit_squash f002

Reasoning within output

commit_squash f003

commit_squash f004

Reasoning within output

commit_squash f005

Reasoning within output

commit_squash f006

commit_squash f007

commit_squash f008

Reasoning within output

commit_squash f009

Reasoning within output

commit_squash f010

Reasoning within output

commit_squash f011

Reasoning within output

commit_squash f012

Reasoning within output

git_bisect f001

Reasoning within output

git_bisect f002

Reasoning within output

git_bisect f003

Reasoning within output

git_bisect f004

Reasoning within output

git_bisect f005

Reasoning within output

git_bisect f006

Reasoning within output

git_bisect f007

Reasoning within output

git_bisect f008

Reasoning within output

git_bisect f009

Reasoning within output

git_bisect f010

Reasoning within output

git_bisect f011

Reasoning within output

git_bisect f012

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

git_log_format f001

Reasoning within output

git_log_format f002

Reasoning within output

git_log_format f003

Reasoning within output

git_log_format f004

Reasoning within output

git_log_format f005

Reasoning within output

git_log_format f006

Reasoning within output

git_log_format f007

Reasoning within output

git_log_format f008

Reasoning within output

git_log_format f009

git_log_format f010

Reasoning within output

git_log_format f011

Reasoning within output

git_log_format f012

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

merge_conflicts f001

Reasoning within output

merge_conflicts f002

Reasoning within output

merge_conflicts f003

Reasoning within output

merge_conflicts f004

Reasoning within output

merge_conflicts f005

Reasoning within output

merge_conflicts f006

Reasoning within output

merge_conflicts f007

Reasoning within output

merge_conflicts f008

Reasoning within output

merge_conflicts f009

Reasoning within output

merge_conflicts f010

Reasoning within output

merge_conflicts f011

Reasoning within output

merge_conflicts f012

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

Reasoning within output

stash_recovery f001

Reasoning within output

stash_recovery f002

Reasoning within output

stash_recovery f003

Reasoning within output

stash_recovery f004

Reasoning within output

stash_recovery f005

Reasoning within output

stash_recovery f006

Reasoning within output

stash_recovery f007

Reasoning within output

stash_recovery f008

Reasoning within output

stash_recovery f009

Reasoning within output

stash_recovery f010

Reasoning within output

stash_recovery f011

Reasoning within output

stash_recovery f012

Reasoning within output

submodule_usage f001

Reasoning within output

submodule_usage f002

Reasoning within output

submodule_usage f003

Reasoning within output

submodule_usage f004

Reasoning within output

submodule_usage f005

Reasoning within output

submodule_usage f006

Reasoning within output

submodule_usage f007

Reasoning within output

submodule_usage f008

Reasoning within output

submodule_usage f009

Reasoning within output

submodule_usage f010

Reasoning within output

submodule_usage f011

Reasoning within output

submodule_usage f012

Reasoning within output

tag_management f001

Reasoning within output

tag_management f002

Reasoning within output

tag_management f003

Reasoning within output

tag_management f004

Reasoning within output

tag_management f005

Reasoning within output

tag_management f006

Reasoning within output

tag_management f007

Reasoning within output

tag_management f008

tag_management f009

Reasoning within output

tag_management f010

Reasoning within output

tag_management f011

Reasoning within output

tag_management f012

Reasoning within output

worktree_usage f001

Reasoning within output

worktree_usage f002

Reasoning within output

worktree_usage f003

Reasoning within output

worktree_usage f004

Reasoning within output

worktree_usage f005

Reasoning within output

worktree_usage f006

Reasoning within output

worktree_usage f007

Reasoning within output

worktree_usage f008

Reasoning within output

worktree_usage f009

Reasoning within output

worktree_usage f010

Reasoning within output

worktree_usage f011

Reasoning within output

worktree_usage f012

Reasoning within output

blame_forensics f001

blame_forensics f002

blame_forensics f003

blame_forensics f004

blame_forensics f005

blame_forensics f006

blame_forensics f007

blame_forensics f008

blame_forensics f009

blame_forensics f010

blame_forensics f011

blame_forensics f012

branch_cleanup f001

branch_cleanup f002

branch_cleanup f003

branch_cleanup f004

branch_cleanup f005

branch_cleanup f006

branch_cleanup f007

branch_cleanup f008

branch_cleanup f009

branch_cleanup f010

branch_cleanup f011

branch_cleanup f012

cherry_pick f001

cherry_pick f002

cherry_pick f003

cherry_pick f004

cherry_pick f005

cherry_pick f006

cherry_pick f007

cherry_pick f008

cherry_pick f009

cherry_pick f010

cherry_pick f011

cherry_pick f012

commit_messages f001

commit_messages f002

commit_messages f003

commit_messages f004

commit_messages f005

commit_messages f006

commit_messages f007

commit_messages f008

commit_messages f009

commit_messages f010

commit_messages f011

commit_messages f012

commit_squash f001

commit_squash f002

commit_squash f003

commit_squash f004

commit_squash f005

commit_squash f006

commit_squash f007

commit_squash f008

commit_squash f009

commit_squash f010

commit_squash f011

commit_squash f012

git_bisect f001

git_bisect f002

git_bisect f003

git_bisect f004

git_bisect f005

git_bisect f006

git_bisect f007

git_bisect f008

git_bisect f009

git_bisect f010

git_bisect f011

git_bisect f012

git_log_format f001

git_log_format f002

git_log_format f003

git_log_format f004

git_log_format f005

git_log_format f006

git_log_format f007

git_log_format f008

git_log_format f009

git_log_format f010

git_log_format f011

git_log_format f012

merge_conflicts f001

merge_conflicts f002

merge_conflicts f003

merge_conflicts f004

merge_conflicts f005

merge_conflicts f006

merge_conflicts f007

merge_conflicts f008

merge_conflicts f009

merge_conflicts f010

merge_conflicts f011

merge_conflicts f012

stash_recovery f001

stash_recovery f002

stash_recovery f003

stash_recovery f004

stash_recovery f005

stash_recovery f006

stash_recovery f007

stash_recovery f008

stash_recovery f009

stash_recovery f010

stash_recovery f011

stash_recovery f012

submodule_usage f001

submodule_usage f002

submodule_usage f003

submodule_usage f004

submodule_usage f005

submodule_usage f006

submodule_usage f007

submodule_usage f008

submodule_usage f009

submodule_usage f010

submodule_usage f011

submodule_usage f012

tag_management f001

tag_management f002

tag_management f003

tag_management f004

tag_management f005

tag_management f006

tag_management f007

tag_management f008

tag_management f009

tag_management f010

tag_management f011

tag_management f012

worktree_usage f001

worktree_usage f002

worktree_usage f003

worktree_usage f004

worktree_usage f005

worktree_usage f006

worktree_usage f007

worktree_usage f008

worktree_usage f009

worktree_usage f010

worktree_usage f011

worktree_usage f012