What it is
Puts Jev's image questions next to an LLM and a commercial moderation API.
What it does: Scores every image with each model and reports AUROC, F1, recall at fixed false-positive rates, latency and cost.
Why it belongs in the showcase: Every number can be recomputed from the published outputs.
Caveats: Results on your own policy will differ.
Evidence
Public MIT repo, created 2026-10-21. Thresholds are chosen on the train split and scored on the 2,037-image test split. Jev: AUROC 0.866, recall 50.7% at 5% false positives, own-verdict F1 0.710, median 162 ms, 7.7 cents per 1,000 images. The three newer models are statistically tied on AUROC (0.856 to 0.866); Rekognition scores 0.698.
Caveats
- UnsafeBench is gated by a data use agreement; images and labels are not in the repo
- One dataset and one prompt
- Latency includes network time from one workstation
Unofficial case page. Stars / list position observed for heat; do not treat heat as a quality SLA. Review the upstream README before you depend on it.