mlsanityData to 7 Oct 2026

Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model

arXiv 2609.23959 ↗First posted 21 Sep 2026cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Screening a phone call for fraud needs a trustworthy probability after every caller turn, in milliseconds. Jev-style typed decisions promise exactly that: declared options go in, one calibrated probability per option comes out of a single forward pass, with no generated text. We test an open implementation of this readout, JevLite, on scam-call screening: Qwen3-4B is LoRA-tuned so that the temperature-scaled softmax over two answer-label logits is P(scam). On 41 held-out CallScreenBench scenarios (577 per-turn decisions) a three-seed ensemble reaches AUROC .974 with calibration error .052, non-inferior to an LLM judge (MiniMax-M3) at a pre-registered .02 margin, with no false alarms on legitimate calls, decisions 1.14 turns earlier under the same hang-up rule, and 64.5 ms per decision on one consumer GPU, 4.9x lower than the same backbone fine-tuned to generate its answer. The gain is in the readout and calibration, not accuracy: a fine-tuned ModernBERT encoder is not significantly worse, the recipe was selected with test-set exposure, and all callers are synthetic. We claim no architectural novelty; the contribution is the application and an evaluation reporting calibration, false alarms and decision timing alongside AUROC.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

27 of 51 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  6. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  7. 2017Selective Classification for Deep Neural Networkscited by 799
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  9. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  10. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
15 more
  1. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  2. 2019Measuring Calibration in Deep Learningcited by 443
  3. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  4. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  5. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  6. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 48
  7. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 37
  8. 2018SCORES: Shape Composition with Recursive Substructure Priorscited by 23
  9. 2025"It Warned Me Just at the Right Moment": Exploring LLM-based Real-time Detection of Phone Scamscited by 12
  10. 2025TeleAntiFraud-28k: An Audio-Text Slow-Thinking Dataset for Telecom Fraud Detectioncited by 12
  11. 2024Combating Phone Scams with LLM-based Detection: Where Do We Stand?cited by 10
  12. 2024Voice-Enabled AI Agents can Perform Common Scamscited by 5
  13. 2026Anatomy of a Scam Call: What 10,000 real scam and spam calls reveal about how phone scammers operatecited by 1
  14. 2026Vishing-Tactics-Bench: Forecasting Exploitation Trajectories in Voice Phishing Callscited by 1
  15. 2026The Machines Are Calling: Measuring Automated and Synthetic Voices in Unwanted Inbound Callscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026From Text Decisions to Pixels: An Study of Jev-Style Visual Choice Modelcited by 1
  2. 2026Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?cited by 1
  3. 2026Typed Decision Models: An Early Evidence Audit and Evaluation Checklistcited by 0
  4. 2026Koa-action: Fast and Consistent Structured Decision Making with Generative LLMscited by 0
  5. 2026Benchmarking System One decision models against trained classifiers and language models for automated decision gatescited by 0
  6. 2026OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System Onecited by 0
  7. 2026Can Jev be Your Q or Policy in Reinforcement Learning?cited by 0