mlsanityData to 14 Oct 2026

Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models

arXiv 2609.10996 ↗First posted 10 Sep 2026cs.CLcited by 1

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Verbalized confidence, long dismissed as overconfident, coarse, and prone to round-number clustering, is now the more robust soft-scoring mechanism for LLM-as-a-Judge on top-tier proprietary models. Across SummEval, AggreFact, and HelpSteer2, spanning up to 18 LLMs, we show that the standard advice to prefer log-probabilities no longer holds on post-2025 models, where verbalized confidence is the better signal. We call this a compatibility shift. On top of a standard verbalized-confidence baseline, we introduce two new ingredients: an overconfidence advisory and self-debate. Together they improve calibration, score-distribution spread, and robustness to task subjectivity. We further observe a generation effect: post-2025 models accommodate these two additions with little balanced-accuracy cost, whereas pre-2025 models pay a measurable penalty. Compared with logprob-based G-Eval, verbalized confidence is the more subjectivity-robust soft signal on GPT-family top-tier releases. The shift is invisible under accuracy-only reporting. Rather than defaulting to hard predictions, we recommend broader use of soft scoring in LLM-as-a-Judge. More broadly, verbalized confidence has moved from a weaker substitute for logprobs to a practical soft-scoring mechanism for contemporary LLM judges.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 6 of its 26 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

6 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 5

  1. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  2. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  3. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  4. 2026How do LLMs Compute Verbal Confidencecited by 26
  5. 2026Rescaling Confidence: What Scale Design Reveals About LLM Metacognitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 30

Similar, not cited

Close in wording, no citation either way

  1. 2024On Verbalized Confidence Scores for LLMs
  2. 2026VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
  3. 2026All Verdicts are Not Equal: Rethinking LLM Judge Reliability
  4. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
  5. 2026Quantifying and Mitigating Self-Preference Bias of LLM Judges
  6. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judges
  7. 2026Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
  8. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges