# Calibrating LLM Confidence by Probing Perturbed Representation Stability

arXiv: https://arxiv.org/abs/2505.21772 · HTML: https://mlsanity.com/paper/2505.21772/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2025.

## Built on by (most cited first, 23 of 23)

- [Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibration](https://mlsanity.com/paper/2605.27752/) (2026)
- [How Reliable are Confidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domains](https://mlsanity.com/paper/2601.08134/) (2026)
- [Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning](https://mlsanity.com/paper/2601.22323/) (2026)
- [Same Answer, Different Representations: Hidden instability in VLMs](https://mlsanity.com/paper/2602.06652/) (2026)
- [Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models](https://mlsanity.com/paper/2604.17274/) (2026)
- [Correctness-Optimized Residual Activation Lens (CORAL): Transferrable and Calibration-Aware Inference-Time Steering](https://mlsanity.com/paper/2602.06022/) (2026)
- [Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models](https://mlsanity.com/paper/2606.15910/) (2026)
- [Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents](https://mlsanity.com/paper/2607.12397/) (2026)
- [Resisting Manipulative Bots in Meme Coin Copy Trading: A Multi-Agent Approach with Chain-of-Thought Reasoning](https://mlsanity.com/paper/2601.08641/) (2026)
- [Evaluating LLM Safety Under Repeated Inference via Accelerated Prompt Stress Testing](https://mlsanity.com/paper/2602.11786/) (2026)
- [Physics-based phenomenological characterization of cross-modal bias in multimodal models](https://mlsanity.com/paper/2602.20624/) (2026)
- [Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling](https://mlsanity.com/paper/2604.09606/) (2026)
- [Margin-Adaptive Confidence Ranking for Reliable LLM Judgement](https://mlsanity.com/paper/2605.15416/) (2026)
- [Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Models](https://mlsanity.com/paper/2605.31220/) (2026)
- [CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction](https://mlsanity.com/paper/2606.05799/) (2026)
- [Calibrating Overconfidence Without Sacrificing Confidence: Probe-Conditioned Head Intervention for LLMs](https://mlsanity.com/paper/2606.09876/) (2026)
- [Localize-Then-Decide Guarantees for LLM Judgments](https://mlsanity.com/paper/2608.25824/) (2026)
- [SABER: Stability-Aware Early Exit for LLM Reasoning via Adversarial Branch Probing](https://mlsanity.com/paper/2608.27963/) (2026)
- [From Performance to Purpose: A Sociotechnical Taxonomy for Evaluating Large Language Model Utility](https://mlsanity.com/paper/2602.20513/) (2026)
- [Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding](https://mlsanity.com/paper/2608.06532/) (2026)
- [Vision-Language Model Confidence Is Not a Property of the Answer](https://mlsanity.com/paper/2608.06571/) (2026)
- [What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels](https://mlsanity.com/paper/2608.22960/) (2026)
- [Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models](https://mlsanity.com/paper/2609.24881/) (2026)

## Abstract (arXiv)

Miscalibration in Large Language Models (LLMs) undermines their reliability, highlighting the need for accurate confidence estimation. We introduce CCPS (Calibrating LLM Confidence by Probing Perturbed Representation Stability), a novel method analyzing internal representational stability in LLMs. CCPS applies targeted adversarial perturbations to final hidden states, extracts features reflecting the model's response to these perturbations, and uses a lightweight classifier to predict answer correctness. CCPS was evaluated on LLMs from 8B to 32B parameters (covering Llama, Qwen, and Mistral architectures) using MMLU and MMLU-Pro benchmarks in both multiple-choice and open-ended formats. Our results show that CCPS significantly outperforms current approaches. Across four LLMs and three MMLU variants, CCPS reduces Expected Calibration Error by approximately 55% and Brier score by 21%, while increasing accuracy by 5 percentage points, Area Under the Precision-Recall Curve by 4 percentage points, and Area Under the Receiver Operating Characteristic Curve by 6 percentage points, all relative to the strongest prior method. CCPS delivers an efficient, broadly applicable, and more accurate solution for estimating LLM confidence, thereby improving their trustworthiness.