mlsanityData to 7 Oct 2026

Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals

arXiv 2605.26045 ↗First posted 25 May 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

An activation oracle is a language model trained to read another model's internal activations and describe them in natural language, for example to name a secret word the other model was trained to hide. Oracle answers carry no measure of confidence, which limits their use in auditing. We compare five ways of attaching a confidence score to an oracle's answer on this secret-word task, across four oracles from two model families (Qwen and Gemma, 8B to 27B parameters), at 6,000 samples per method and oracle. The five methods rank the same way on all four oracles. Which method to use depends on one question: can the auditor list the possible answers in advance? If the auditor can, then having the oracle score each candidate answer roughly doubles accuracy and separates correct from wrong answers best of the five (AUROC 0.92 to 0.96). If the oracle must generate its answer freely and no labeled data exists, the agreement rate over twenty samples is the only confidence that is calibrated on every oracle. Once labeled data exists, a rescaled answer probability reaches the same calibration at one generation instead of twenty. Asking the oracle to state a confidence number gives no usable signal on any oracle. Code and the patched trainer are available at https://github.com/federicotorrielli/probabilisticactivationoracles.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

20 of 36 references.

References · 20

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  7. 2023Steering Language Models With Activation Engineeringcited by 824
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  10. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  11. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 49
  12. 2025Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainerscited by 41
8 more
  1. 2026AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviorscited by 32
  2. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 24
  3. 2025Eliciting Secret Knowledge from Language Modelscited by 23
  4. 2026Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representationscited by 13
  5. 2026Closing the Confidence-Faithfulness Gap in Large Language Modelscited by 8
  6. 2026Hidden Error Awareness in Chain-of-Thought Reasoning: The Signal Is Diagnostic, Not Causalcited by 5
  7. 2025Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformerscited by 2
  8. 2026Split Personality Training: Revealing Latent Knowledge Through Alternate Personalitiescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Selecting The Most Informative Tokens in Natural Language Autoencoderscited by 0