What it is
Calibration-oriented eval for decision models including Jev.
What it does: Generates memos with known posteriors and scores model probabilities against them.
Why it belongs in the showcase: Rare public bench that tests probability quality, with an explicit TypeSafe Jev runner path.
Caveats: Synthetic worlds; API cost for hosted models.
Evidence
Public MIT repo (3★ at intake) with uv-based install, pytest, task worlds (paper_review, covert_transfer, financial_default), Hugging Face dataset ishidalab/probabilitybench, blog https://ishida-lab.github.io/blog_probabilitybench.html, and arXiv https://arxiv.org/abs/2610.17016. README names default models including TypeSafe Jev 1.13 and shows examples/decision_models.py --models typesafe/jev-1.13.
Caveats
- Worlds are synthetic mechanisms, not empirical peer-review or credit data
- Running against hosted Jev needs your own API key and spend
- 3★ at intake; results depend on which models you configure
Unofficial case page. Stars / list position observed for heat; do not treat heat as a quality SLA. Review the upstream README before you depend on it.