# Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model

arXiv: https://arxiv.org/abs/2609.23959 · HTML: https://mlsanity.com/paper/2609.23959/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [ShieldGemma: Generative AI Content Moderation Based on Gemma](https://mlsanity.com/paper/2407.21772/) (2024)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models](https://mlsanity.com/paper/2503.02623/) (2025)
- [ConfTuner: Training Large Language Models to Express Their Confidence Verbally](https://mlsanity.com/paper/2508.18847/) (2025)
- [SCORES: Shape Composition with Recursive Substructure Priors](https://mlsanity.com/paper/1809.05398/) (2018)
- ["It Warned Me Just at the Right Moment": Exploring LLM-based Real-time Detection of Phone Scams](https://mlsanity.com/paper/2502.03964/) (2025)
- [TeleAntiFraud-28k: An Audio-Text Slow-Thinking Dataset for Telecom Fraud Detection](https://mlsanity.com/paper/2503.24115/) (2025)
- [Combating Phone Scams with LLM-based Detection: Where Do We Stand?](https://mlsanity.com/paper/2409.11643/) (2024)
- [Voice-Enabled AI Agents can Perform Common Scams](https://mlsanity.com/paper/2410.15650/) (2024)
- [Anatomy of a Scam Call: What 10,000 real scam and spam calls reveal about how phone scammers operate](https://mlsanity.com/paper/2608.24127/) (2026)
- [Vishing-Tactics-Bench: Forecasting Exploitation Trajectories in Voice Phishing Calls](https://mlsanity.com/paper/2609.07151/) (2026)
- [The Machines Are Calling: Measuring Automated and Synthetic Voices in Unwanted Inbound Calls](https://mlsanity.com/paper/2609.11137/) (2026)

## Built on by (most cited first, 8 of 8)

- [Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?](https://mlsanity.com/paper/2609.33209/) (2026)
- [Typed Decision Models: An Early Evidence Audit and Evaluation Checklist](https://mlsanity.com/paper/2609.32160/) (2026)
- [From Text Decisions to Pixels: An Study of Jev-Style Visual Choice Model](https://mlsanity.com/paper/2609.29283/) (2026)
- [Benchmarking System One decision models against trained classifiers and language models for automated decision gates](https://mlsanity.com/paper/2610.00346/) (2026)
- [Koa-decision: Fast and Consistent Structured Decision Making with Generative LLMs](https://mlsanity.com/paper/2609.36115/) (2026)
- [OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System One](https://mlsanity.com/paper/2610.00381/) (2026)
- [Can Jev be Your Q or Policy in Reinforcement Learning?](https://mlsanity.com/paper/2610.11692/) (2026)
- [VideoJev: System One Model for Video Understanding and Interaction](https://mlsanity.com/paper/2610.18615/) (2026)

## Abstract (arXiv)

Screening a phone call for fraud needs a trustworthy probability after every caller turn, in milliseconds. Jev-style typed decisions promise exactly that: declared options go in, one calibrated probability per option comes out of a single forward pass, with no generated text. We test an open implementation of this readout, JevLite, on scam-call screening: Qwen3-4B is LoRA-tuned so that the temperature-scaled softmax over two answer-label logits is P(scam). On 41 held-out CallScreenBench scenarios (577 per-turn decisions) a three-seed ensemble reaches AUROC .974 with calibration error .052, non-inferior to an LLM judge (MiniMax-M3) at a pre-registered .02 margin, with no false alarms on legitimate calls, decisions 1.14 turns earlier under the same hang-up rule, and 64.5 ms per decision on one consumer GPU, 4.9x lower than the same backbone fine-tuned to generate its answer. The gain is in the readout and calibration, not accuracy: a fine-tuned ModernBERT encoder is not significantly worse, the recipe was selected with test-set exposure, and all callers are synthetic. We claim no architectural novelty; the contribution is the application and an evaluation reporting calibration, false alarms and decision timing alongside AUROC.