# Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals

arXiv: https://arxiv.org/abs/2605.26045 · HTML: https://mlsanity.com/paper/2605.26045/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 20 of 20)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Reducing conversational agents' overconfidence through linguistic calibration](https://mlsanity.com/paper/2012.14983/) (2020)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers](https://mlsanity.com/paper/2512.15674/) (2025)
- [LatentQA: Teaching LLMs to Decode Activations Into Natural Language](https://mlsanity.com/paper/2412.08686/) (2024)
- [AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors](https://mlsanity.com/paper/2602.22755/) (2026)
- [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025)
- [Eliciting Secret Knowledge from Language Models](https://mlsanity.com/paper/2510.01070/) (2025)
- [Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representations](https://mlsanity.com/paper/2603.18353/) (2026)
- [Closing the Confidence-Faithfulness Gap in Large Language Models](https://mlsanity.com/paper/2603.25052/) (2026)
- [Hidden Error Awareness in Chain-of-Thought Reasoning: The Signal Is Diagnostic, Not Causal](https://mlsanity.com/paper/2605.09502/) (2026)
- [Split Personality Training: Revealing Latent Knowledge Through Alternate Personalities](https://mlsanity.com/paper/2602.05532/) (2026)
- [Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformers](https://mlsanity.com/paper/2507.09406/) (2025)

## Built on by (most cited first, 1 of 1)

- [Selecting The Most Informative Tokens in Natural Language Autoencoders](https://mlsanity.com/paper/2609.37040/) (2026)

## Abstract (arXiv)

An activation oracle is a language model trained to read another model's internal activations and describe them in natural language, for example to name a secret word the other model was trained to hide. Oracle answers carry no measure of confidence, which limits their use in auditing. We compare five ways of attaching a confidence score to an oracle's answer on this secret-word task, across four oracles from two model families (Qwen and Gemma, 8B to 27B parameters), at 6,000 samples per method and oracle. The five methods rank the same way on all four oracles. Which method to use depends on one question: can the auditor list the possible answers in advance? If the auditor can, then having the oracle score each candidate answer roughly doubles accuracy and separates correct from wrong answers best of the five (AUROC 0.92 to 0.96). If the oracle must generate its answer freely and no labeled data exists, the agreement rate over twenty samples is the only confidence that is calibrated on every oracle. Once labeled data exists, a rescaled answer probability reaches the same calibration at one generation instead of twenty. Asking the oracle to state a confidence number gives no usable signal on any oracle. Code and the patched trainer are available at https://github.com/federicotorrielli/probabilisticactivationoracles.