mlsanityData to 7 Oct 2026

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

arXiv 2609.26550 ↗First posted 22 Sep 2026cs.AIcited by 14

New in LLM reasoning; no direction yet. LLM reasoning

Abstract (arXiv)

LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that returns label probabilities instead of text, and whose confidence decides whether to accept its verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges, with blinded human adjudication, JEV comes within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee and a 0.15-second median latency, and falls behind where the verdict must be derived, as in math, code, and logic. Its confidence marks this boundary. With a threshold frozen in advance, accepting confident verdicts and escalating the rest is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs at 41% of its fee, and in a pre-specified live test on two new workloads the cascade matches GPT-6's accuracy exactly. Confidence routing weakens on style-adversarial pairs and reference-free prose; we close with a simple recipe for validating thresholds locally.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

26 of 38 references.

References · 26

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  5. 2017Selective Classification for Deep Neural Networkscited by 799
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  8. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  10. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  11. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
14 more
  1. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  3. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  4. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  5. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  6. 2024How to Evaluate Reward Models for RLHFcited by 84
  7. 2025Reasoning Models Better Express Their Confidencecited by 63
  8. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  9. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluatorscited by 13
  10. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
  11. 2026BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluationcited by 3
  12. 2026Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluationcited by 2
  13. 2026Luna-2: Scalable Single-Token Evaluation with Small Language Modelscited by 1
  14. 2026Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agentscited by 2
  2. 2026Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?cited by 1
  3. 2026Typed Decision Models: An Early Evidence Audit and Evaluation Checklistcited by 0
  4. 2026Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automationcited by 0
  5. 2026Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalationcited by 0
  6. 2026Calibrated to Whom? Persona and Language Effects on Cultural Values in JEVcited by 0
  7. 2026More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Modelscited by 0
  8. 2026Benchmarking System One decision models against trained classifiers and language models for automated decision gatescited by 0
  9. 2026OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System Onecited by 0
  10. 2026AnyJev Technical Reportcited by 0
  11. 2026Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Modelcited by 0
  12. 2026Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentationcited by 0
2 more
  1. 2026Can Jev be Your Q or Policy in Reinforcement Learning?cited by 0
  2. 2026Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
  2. 2026VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
  3. 2026Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistence
  4. 2026JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
  5. 2025Verdict: A Library for Scaling Judge-Time Compute
  6. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
  7. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
  8. 2025Confidence as a Reward: Transforming LLMs into Reward Models