# JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

arXiv: https://arxiv.org/abs/2609.26550 · HTML: https://mlsanity.com/paper/2609.26550/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 22 Sep 2026.

## Placed in

- LLM reasoning

## Builds on (most cited first, 26 of 26)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy](https://mlsanity.com/paper/2507.01352/) (2025)
- [RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style](https://mlsanity.com/paper/2410.16184/) (2024)
- [RewardBench 2: Advancing Reward Model Evaluation](https://mlsanity.com/paper/2506.01937/) (2025)
- [Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement](https://mlsanity.com/paper/2407.18370/) (2024)
- [How to Evaluate Reward Models for RLHF](https://mlsanity.com/paper/2410.14872/) (2024)
- [Reasoning Models Better Express Their Confidence](https://mlsanity.com/paper/2505.14489/) (2025)
- [PEDANTS: Cheap but Effective and Interpretable Answer Equivalence](https://mlsanity.com/paper/2402.11161/) (2024)
- [Scaling Evaluation-time Compute with Reasoning Models as Evaluators](https://mlsanity.com/paper/2503.19877/) (2025)
- [Ask a Strong LLM Judge when Your Reward Model is Uncertain](https://mlsanity.com/paper/2510.20369/) (2025)
- [BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation](https://mlsanity.com/paper/2604.09497/) (2026)
- [Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation](https://mlsanity.com/paper/2609.02942/) (2026)
- [Luna-2: Scalable Single-Token Evaluation with Small Language Models](https://mlsanity.com/paper/2602.18583/) (2026)
- [Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models](https://mlsanity.com/paper/2609.10996/) (2026)

## Built on by (most cited first, 30 of 30)

- [Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?](https://mlsanity.com/paper/2609.33209/) (2026)
- [Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents](https://mlsanity.com/paper/2609.28940/) (2026)
- [Typed Decision Models: An Early Evidence Audit and Evaluation Checklist](https://mlsanity.com/paper/2609.32160/) (2026)
- [More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models](https://mlsanity.com/paper/2609.38827/) (2026)
- [Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation](https://mlsanity.com/paper/2609.33843/) (2026)
- [Calibrated to Whom? Persona and Language Effects on Cultural Values in JEV](https://mlsanity.com/paper/2609.36399/) (2026)
- [Benchmarking System One decision models against trained classifiers and language models for automated decision gates](https://mlsanity.com/paper/2610.00346/) (2026)
- [AnyJev Technical Report](https://mlsanity.com/paper/2610.00831/) (2026)
- [Labels Override Definitions in Jev-Style Typed Decision Models](https://mlsanity.com/paper/2610.02586/) (2026)
- [Hidden Risks of Jev: An Empirical Study of Security, Privacy, and Dual Use](https://mlsanity.com/paper/2610.04985/) (2026)
- [Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation](https://mlsanity.com/paper/2609.33401/) (2026)
- [OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System One](https://mlsanity.com/paper/2610.00381/) (2026)
- [Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model](https://mlsanity.com/paper/2610.01006/) (2026)
- [CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks](https://mlsanity.com/paper/2610.07177/) (2026)
- [SanSi: A Looped Typed Decision Model for System 1.5 Thinking](https://mlsanity.com/paper/2610.07730/) (2026)
- [Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge](https://mlsanity.com/paper/2610.08675/) (2026)
- [Emo-Jev: Probabilistic Reasoning for Emotion Classification with Jev](https://mlsanity.com/paper/2610.08829/) (2026)
- [From Probabilities to Decisions: Search and Multi-Teacher Distillation with Jev](https://mlsanity.com/paper/2610.09188/) (2026)
- [Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentation](https://mlsanity.com/paper/2610.11436/) (2026)
- [Can Jev be Your Q or Policy in Reinforcement Learning?](https://mlsanity.com/paper/2610.11692/) (2026)
- [Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks](https://mlsanity.com/paper/2610.11978/) (2026)
- [Does System One Transfer? Calibrated Typed-Decision Models, Label Budgets, and Translation for Indonesian Regional Languages](https://mlsanity.com/paper/2610.14112/) (2026)
- [Suggested Is Not Superseded: A Pre-Registered Evaluation of a System-One Model as a Memory-Retirement Gate](https://mlsanity.com/paper/2610.15710/) (2026)
- [Review Rarely Beats the Stronger Model: Separating Verification from Capability Transfer in LLM Pipelines](https://mlsanity.com/paper/2610.15897/) (2026)
- [Correct Choices, Failed Sequences: Choice2Seq Benchmarks Correctness Retention Across Output Interfaces](https://mlsanity.com/paper/2610.16191/) (2026)
- [Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges](https://mlsanity.com/paper/2610.18122/) (2026)
- [Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments](https://mlsanity.com/paper/2610.18497/) (2026)
- [Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops](https://mlsanity.com/paper/2610.18579/) (2026)
- [Jev-ToM: Evaluating Theory of Mind in Typed Decision Models](https://mlsanity.com/paper/2610.18610/) (2026)
- [Efficient contextual partial monitoring with general function approximation](https://mlsanity.com/paper/2610.19804/) (2026)

## Abstract (arXiv)

LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that returns label probabilities instead of text, and whose confidence decides whether to accept its verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges, with blinded human adjudication, JEV comes within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee and a 0.15-second median latency, and falls behind where the verdict must be derived, as in math, code, and logic. Its confidence marks this boundary. With a threshold frozen in advance, accepting confident verdicts and escalating the rest is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs at 41% of its fee, and in a pre-specified live test on two new workloads the cascade matches GPT-6's accuracy exactly. Confidence routing weakens on style-adversarial pairs and reference-free prose; we close with a simple recipe for validating thresholds locally.