Decision Models for Evals: Tune the Threshold (Hamel Husain)
External write-up by Hamel Husain and Antaripa Saha (2026-10-20): use decision models such as Jev as eval judges, tune each model's threshold instead of trusting 0.50, compare cost and performance, and check calibration before reading scores as confidence.
What this guide points to
Hamel Husain and Antaripa Saha published How To Use Decision Models for Evals on Hamel’s blog (2026-10-20). They used 100 traces from an apartment-leasing support agent and asked each decision model for the probability that a trace contains a substantive failure. They scored 98 traces after removing two internal QA examples. Models tested: Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions. Jev received the same criteria as a structured noul question; the appendix shows that JSON.
The main point: the default 0.50 threshold can leave a lot of performance on the table. They set a minimum of 80% precision and 80% recall, then tuned each threshold with stratified five-fold cross-validation on saved scores, with no new model calls. On this dataset OpenAI Decisions reached F1 0.87 at 3.72 cents per 100 runs, and Perplexity reached F1 0.79 at 0.82 cents. On calibration, Jev and Liquid d1-ultra were already well calibrated (Jev’s Brier score was 0.187 before Platt scaling and 0.188 after). The post includes two copy-ready prompts for a coding agent: one for threshold tuning, one for calibration checks.
This page is an unofficial field manual pointer — verify claims against the article and current TypeSafe docs. The authors say results vary by task and dataset; measure on your own data.
Takeaways
- Set minimum precision and recall first, and rule out models that cannot meet them.
- Tune each model’s threshold on labeled data before you compare models.
- Check calibration before you treat a score as a confidence, and measure whether calibration helps.