What it is
A fair-test harness for typed classifiers versus LLMs.
What it does: Every classifier gets the same questions in its own native shape. Runs use seeded shuffles, rotate which classifier goes first, freeze the protocol before the first call and seal results with checksums. The analyzer reports match rate, latency percentiles, errors and rate limits, spend, calibration, self-consistency, paired confidence intervals, and a cascade that escalates when the baseline is not confident enough.
Why it belongs in the showcase: A reusable way to decide, with evidence, whether a fast decision model can replace an LLM call in a routing or classification step.
Caveats: No published results; labels are not from independent human experts.
Evidence
Public MIT repo and PyPI package; ships a frozen corpus of 2,321 cases across 31 tasks (434 authored cases, where a blind second annotator agreed on 431, plus public dataset samples); a dry-run mode needs no keys; the author's rough cost for one full pass is Jev $0.02 versus Haiku $2.30 and Sonnet $4.65 (registry-priced estimates); a cascade report shows escalation from a baseline classifier to another one by confidence threshold.
Caveats
- The README publishes no head-to-head results; users run the benchmark themselves
- Authored labels were checked by LLM-assisted annotators, and public test sets may be in LLM training data
- The author discloses that he also builds open-source tools on Jev
Unofficial case page. Stars / list position observed for heat; do not treat heat as a quality SLA. Review the upstream README before you depend on it.