mlsanityData to 14 Oct 2026

VERA: Verdict-Conditioned Reliability for Adaptive LLM Judges

arXiv 2610.05452 ↗First posted 4 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Accurately estimating judgment reliability is a central challenge in adapting LLM judges to newly verified feedback while preserving previously learned behavior. However, existing approaches often rely on output-level confidence, which can be overconfident and poorly aligned with judgment correctness. We propose VERA, a VErdict-conditioned Reliability Axis that estimates reliability from hidden activations by distinguishing correct from incorrect judgments within each predicted-verdict group. Using VERA as a control signal, we develop a VERA-guided periodic adaptation framework that integrates reliability-ranked corrective updates, reliability-residual replay, and periodic refresh of the reliability directions. After VERA-guided adaptation on Chatbot Arena, 8B- and 14B-parameter judges outperform the strongest baseline on each of four held-out public benchmarks, with relative gains of up to 23.01%. The framework also improves focal-class recall by up to 16.1% relative to the strongest adaptive baselines on a separate proprietary temporal auditing task.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 26 references that are in the map. First posted on arXiv 4 Oct 2026.

Builds on

25 of 26 references.

References · 25

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  7. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  9. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  10. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  11. 2024Self-Rewarding Language Modelscited by 681
  12. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
13 more
  1. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  2. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  3. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  4. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  5. 2023Generative Judge for Evaluating Alignmentcited by 169
  6. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  7. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  8. 2024Factual Confidence of LLMs: on Reliability and Robustness of Current Estimatorscited by 61
  9. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 58
  10. 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 30
  11. 2024Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Modelscited by 19
  12. 2024Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuningcited by 10
  13. 2026The Value Axis: Language Models Encode Whether They're on the Right Trackcited by 7

Built on by

Not cited yet.