JevEval: Jev-as-a-Judge in DeepEval

External DeepEval blog and docs: define Noul, Score and Choice questions yourself; TypeSafe Jev answers them with calibrated probabilities; DeepEval turns the answers into a fixed, inspectable metric score with no evaluation LLM.

What this guide points to

Confident AI’s DeepEval framework shipped JevEval, a custom metric where you write the decision points, TypeSafe Jev answers them, and DeepEval computes the score with fixed math. The product blog Introducing JevEval: Jev-as-a-Judge and the docs page JevEval are the sources.

You choose which test-case fields Jev may see, then attach weighted Noul (yes/no), Score (ordered levels) and Choice (behaviors with credits) questions. Jev returns probability distributions in one decide() call. DeepEval maps each distribution to a 0–1 value and takes a weighted mean. Reasons are assembled from those answers in code, so there is no evaluation LLM and no model argument on the metric.

The blog walks through a Tool Faithfulness example and covers score_breakdown, deterministic reasons, strict_mode, and ConversationalJevEval for multi-turn cases. Install path shown in the docs: pip install -U deepeval typesafe-sdk with TYPESAFE_API_KEY set.

This page is an unofficial field manual pointer — behaviour and pricing follow DeepEval and TypeSafe docs.

Takeaways

1. Put evaluation logic in explicit bounded questions, not in one open-ended “be a judge” prompt.
2. Use score_breakdown when a metric fails: you are debugging a specification, not reverse-engineering a generated number.
3. Turn on strict_mode for hard acceptance gates where partial credit must not mask a failed requirement.

Sources

• Introducing JevEval: Jev-as-a-Judge (DeepEval blog)
• JevEval metric docs
• TypeSafe AI (Jev) integration