The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust
Not on the map. Why?
Abstract (arXiv)
As language models improve and become increasingly deployed to solve a variety of tasks, trustworthiness becomes essential. Calibration is a good proxy for trust: well-calibrated confidence estimates help inform the risk versus reward tradeoff when trusting a specific model output. Unfortunately, even as models improve, they remain poorly calibrated, often biasing towards overconfidence. Additionally, calibration can be gamed: a policy that always predicts the base rate is perfectly calibrated, but completely uninformative. To resolve this, we develop a new metric, expected utility renormalized by the oracle (EURO), that balances calibration and informativeness. We also propose a general-purpose activation-based confidence, utility, and trust estimation protocol (ACUTE) to appropriately adjudicate uncertainty. The ACUTE protocol provides flexible, sample-efficient, and compute-efficient confidence estimators for 3 tasks including multiple choice question answering, tool-calling, and scientific document summarization across 6 models from 4 model families. ACUTE outperforms strong baselines on EURO, while maintaining low calibration error. Taken together, our work shows that equipping LLMs with the ACUTE protocol can improve calibration, utility, and trustworthiness in numerous settings.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 56 references that are in the map. First posted on arXiv 5 Jun 2026.
Builds on
38 of 56 references.
References · 38
- 2025Qwen3 Technical Reportcited by 8,195
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
- 2020A Primer in BERTology: What we know about how BERT workscited by 807
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
- 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
- 2020Calibration of Pre-trained Transformerscited by 307
26 more
- 2023In-Context Learning Creates Task Vectorscited by 302
- 2023Function Vectors in Large Language Modelscited by 292
- 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
- 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
- 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
- 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
- 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
- 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 106
- 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
- 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 59
- 2024Linguistic Calibration of Long-Form Generationscited by 59
- 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
- 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 29
- 2024Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairscited by 24
- 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 22
- 2015Binary Classifier Calibration using an Ensemble of Near Isotonic Regression Modelscited by 21
- 2019Can Unconditional Language Models Recover Arbitrary Sentences?cited by 15
- 2025MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Toolscited by 15
- 2022Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQLcited by 12
- 2025LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilizationcited by 11
- 2026Differential syntactic and semantic encoding in LLMscited by 6
- 2023Did You Mean...? Confidence-based Trade-offs in Semantic Parsingcited by 5
- 2025SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alonecited by 5
- 2025Random Forest Calibrationcited by 3
- 2026Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansioncited by 2
- 2026How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuitscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
- 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
- 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models
- 2026CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
- 2025Trusted Uncertainty in Large Language Models: A Unified Framework for Confidence Calibration and Risk-Controlled Refusal
- 2024Calibrating the Confidence of Large Language Models by Eliciting Fidelity
- 2026Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning
- 2026Rethinking Uncertainty Evaluation in Large Language Models