How QuizPilot Uses Jev to Answer Quiz Questions (90-question benchmark)
External post by the QuizPilot developer: how single-choice, true/false and multi-select questions map onto Jev, and a small self-written benchmark against two LLMs, with confidence used to pick answers for re-checking.
What this guide points to
The developer of QuizPilot, an open-source Chrome extension that suggests answers to practice questions on a web page, explains how the extension sends questions to Jev (post dated 2026-10-06, also cross-posted on DEV). Single-choice questions become one Choice question; true/false becomes one Noul; multi-select becomes one Noul per option, ticking every option at 0.5 or above. Fill-in and picture questions are not sent to Jev.
On 90 questions the author wrote (none from a real exam), Jev answered 85–86 correctly across two runs, against 89 for a lightweight LLM and 90 for a reasoning LLM, and answered a 60-question page in 0.17–0.44 s in one request. Every true/false and multi-select answer was right; all nine misses were single-choice questions that need multi-step arithmetic. Seven of the nine misses had confidence below 0.5, while right answers averaged 0.93.
Treat it as a community benchmark, not a TypeSafe product page. The question set is small and self-written, and the author ran Jev 1.13 through OpenRouter.
This page is an unofficial field manual pointer — confirm SDK names, limits, and pricing against current docs.
What to take away
- Map each question format to the matching type: Choice for one answer, Noul for true/false, one Noul per option for multi-select.
- Turn a Noul probability into a confidence with its distance from 0.5, and use the lowest option confidence for multi-select.
- Re-check low-confidence answers with another model: in the author’s numbers, re-checking below 0.6 touched 8% of answers.
Source
Click through: How We Use Jev to Answer Quiz Questions (QuizPilot) · DEV cross-post