Jev 101
Start Guides
Search
中文 EN
Platform heat 1 · Quiet★ 0heat incomplete (no X)

JevVsLLM

A classification benchmark harness that compares TypeSafe Jev with low-cost LLMs and other OpenRouter decision models on five public datasets × 2,000 examples, reporting accuracy, cost and latency.

Repo: danielrapi/JevVsLLM ↗

What it is

Jev versus LLMs and peer decision models on classification.

What it does: Converts five public tasks to a shared multiple-choice shape, runs Jev and baselines through OpenRouter, and publishes accuracy, cost and latency tables plus per-dataset winners.

Why it belongs in the showcase: A clear, negative-capable comparison: Jev wins aggregate LLM matchups and MMLU, while Clef Flash wins the dedicated decision-model aggregate in the author's run.

Caveats: 0★; no license file; public datasets; OpenRouter path.

Evidence

Public repo with harness, converted datasets and published result tables. In the LLM comparison (5×2,000 = 130,000 decisions via OpenRouter), Jev scored 89.0% accuracy at $0.27 and 288 ms p50, ahead of DeepSeek Flash letter (88.0%, $0.55, 395 ms) and other cheap LLMs, with no output/API failures. In the dedicated decision-model run on the same setup, Clef Flash led at 92.0% ($0.11, 372 ms) while Jev scored 89.0% ($0.27, 272 ms) and won MMLU at 93.1%. The author notes public-dataset contamination risk and that invalid outputs count as incorrect.

Caveats

  • 0★ at intake; accepted for the strong published write-up and result tables
  • No license file in the repo at intake
  • Public benchmark datasets; contamination cannot be ruled out
  • All calls go through OpenRouter; provider rate limits affected some LLM arms

Unofficial case page. Stars / list position observed for heat; do not treat heat as a quality SLA. Review the upstream README before you depend on it.