What it is
A small, dependency-free evaluator for one question every Jev workflow has to answer: at what confidence is it safe to act automatically?
What it does: Sends each labeled row to Jev once, caches the answers, and then reports accuracy, expected calibration error, a reliability table and a risk-coverage table per question and for the row-level gate. thresholds recommends a cut-point for a target accuracy with an out-of-bag check; compare diffs two runs (model-pin drift or two question wordings).
Why it belongs in the showcase: It turns the confidence-gated pattern into a measured number instead of a guess, and re-slicing the cache costs nothing.
Caveats: Results are only as good as your labels; no published benchmark is included.
Evidence
Public MIT README; stdlib-only Python 3.10+; only the `run` command calls Jev (one batched call per labeled row, cached to resumable JSONL); `report`, `thresholds`, `compare` and `calibrate` work offline on the cache; supports Choice, Noul and Score questions and writes `thresholds.json`.
Caveats
- You need your own labeled data; the quality of the cut-point depends on how representative it is
- The example output in the README is illustrative; no published benchmark run ships with the repo
- New repository at intake (0 stars); only `run` spends API credit
Unofficial case page. Stars / list position observed for heat; do not treat heat as a quality SLA. Review the upstream README before you depend on it.