# Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

arXiv: https://arxiv.org/abs/2512.15674 · HTML: https://mlsanity.com/paper/2512.15674/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Builds on (most cited first, 23 of 23)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Automatically Interpreting Millions of Features in Large Language Models](https://mlsanity.com/paper/2410.13928/) (2024)
- [Model Organisms for Emergent Misalignment](https://mlsanity.com/paper/2506.11613/) (2025)
- [Multi-Dimensional Gender Bias Classification](https://mlsanity.com/paper/2005.00614/) (2020)
- [SelfIE: Self-Interpretation of Large Language Model Embeddings](https://mlsanity.com/paper/2403.10949/) (2024)
- [LatentQA: Teaching LLMs to Decode Activations Into Natural Language](https://mlsanity.com/paper/2412.08686/) (2024)
- [Training Language Models to Explain Their Own Computations](https://mlsanity.com/paper/2511.08579/) (2025)
- [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025)
- [Eliciting Secret Knowledge from Language Models](https://mlsanity.com/paper/2510.01070/) (2025)
- [Learning to Interpret Weight Differences in Language Models](https://mlsanity.com/paper/2510.05092/) (2025)
- [Do Activation Verbalization Methods Convey Privileged Information?](https://mlsanity.com/paper/2509.13316/) (2025)
- [Meta-Models: An Architecture for Decoding LLM Behaviors Through Interpreted Embeddings and Natural Language](https://mlsanity.com/paper/2410.02472/) (2024)

## Built on by (most cited first, 30 of 52)

- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Can LLMs Introspect? A Reality Check](https://mlsanity.com/paper/2605.26242/) (2026)
- [Introspection Adapters: Training LLMs to Report Their Learned Behaviors](https://mlsanity.com/paper/2604.16812/) (2026)
- [Tracing Persona Vectors Through LLM Pretraining](https://mlsanity.com/paper/2605.13329/) (2026)
- [Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden States](https://mlsanity.com/paper/2608.02657/) (2026)
- [The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology](https://mlsanity.com/paper/2607.01033/) (2026)
- [Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimes](https://mlsanity.com/paper/2603.04426/) (2026)
- [Training Deliberative Monitors for Black-Box Scheming Detection](https://mlsanity.com/paper/2605.29601/) (2026)
- ["Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms](https://mlsanity.com/paper/2606.12618/) (2026)
- [Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision](https://mlsanity.com/paper/2606.32038/) (2026)
- [Training on Documents About Monitoring Leads to CoT Obfuscation](https://mlsanity.com/paper/2605.15257/) (2026)
- [How Transparent is DiffusionGemma?](https://mlsanity.com/paper/2606.20560/) (2026)
- [Language Models Can Explain Visual Features via Steering](https://mlsanity.com/paper/2603.22593/) (2026)
- [Shared Lexical Task Representations Explain Behavioral Variability In LLMs](https://mlsanity.com/paper/2604.22027/) (2026)
- [How LLMs Are Persuaded: A Few Attention Heads, Rerouted](https://mlsanity.com/paper/2605.09314/) (2026)
- [Decoding Hidden Deception in Reasoning LLMs: Activation Explainers for Deception Auditing](https://mlsanity.com/paper/2606.17478/) (2026)
- [ADAG: Automatically Describing Attribution Graphs](https://mlsanity.com/paper/2604.07615/) (2026)
- [From Correlation to Cause: A Five-Stage Methodology for Feature Analysis in Transformer Language Models](https://mlsanity.com/paper/2605.22462/) (2026)
- [Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals](https://mlsanity.com/paper/2605.26045/) (2026)
- [Building Better Activation Oracles](https://mlsanity.com/paper/2606.02609/) (2026)
- [SAEExplainer: Interpreting SAE Features with Activation-Guided Preference Optimization](https://mlsanity.com/paper/2606.08496/) (2026)
- [The Impossibility of Eliciting Latent Knowledge](https://mlsanity.com/paper/2606.12268/) (2026)
- [Model Forensics: Investigating Whether Concerning Behavior Reflects Misalignment](https://mlsanity.com/paper/2606.26071/) (2026)
- [Radical AI Interpretability](https://mlsanity.com/paper/2606.26523/) (2026)
- [Introspection Fine-Tuning (IFT): Training Small LLMs to Introspect](https://mlsanity.com/paper/2607.14111/) (2026)
- [Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent](https://mlsanity.com/paper/2607.16448/) (2026)
- [When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles](https://mlsanity.com/paper/2607.23379/) (2026)
- [Matryoshka attribution: Learning to attribute language model outputs to representations and weights](https://mlsanity.com/paper/2609.25518/) (2026)
- [Stress-Testing Alignment Audits With Prompt-Level Strategic Deception](https://mlsanity.com/paper/2602.08877/) (2026)
- [Learning Self-Interpretation from Interpretability Artifacts: Training Lightweight Adapters on Vector-Label Pairs](https://mlsanity.com/paper/2602.10352/) (2026)

## Abstract (arXiv)

Large language model (LLM) activations are notoriously difficult to understand, with most existing techniques using complex, specialized methods for interpreting them. Recent work has proposed a simpler approach known as LatentQA: training LLMs to directly accept LLM activations as inputs and answer arbitrary questions about them in natural language. However, prior work has focused on narrow task settings for both training and evaluation. In this paper, we instead take a generalist perspective. We evaluate LatentQA-trained models, which we call Activation Oracles (AOs), in far out-of-distribution settings and examine how performance scales with training data diversity. We find that AOs can recover information fine-tuned into a model (e.g., biographical knowledge or malign propensities) that does not appear in the input text, despite never being trained with activations from a fine-tuned model. Our main evaluations are four downstream tasks where we can compare to prior white- and black-box techniques. We find that even narrowly-trained LatentQA models can generalize well, and that adding additional training datasets (such as classification tasks and a self-supervised context prediction task) yields consistent further improvements. Our best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4. These results suggest that diversified training to answer natural-language queries imparts a general capability to verbalize information about LLM activations.