# Towards eliciting latent knowledge from LLMs with mechanistic interpretability

arXiv: https://arxiv.org/abs/2505.14352 · HTML: https://mlsanity.com/paper/2505.14352/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Large Language Models can Strategically Deceive their Users when Put Under Pressure](https://mlsanity.com/paper/2311.07590/) (2023)
- [Tell me about yourself: LLMs are aware of their learned behaviors](https://mlsanity.com/paper/2501.11120/) (2025)
- [Auditing language models for hidden objectives](https://mlsanity.com/paper/2503.10965/) (2025)
- [LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers](https://mlsanity.com/paper/2502.15007/) (2025)

## Built on by (most cited first, 19 of 19)

- [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025)
- [Difficulties with Evaluating a Deception Detector for AIs](https://mlsanity.com/paper/2511.22662/) (2025)
- [The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models](https://mlsanity.com/paper/2604.06374/) (2026)
- [The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology](https://mlsanity.com/paper/2607.01033/) (2026)
- [Prompting Large Language Models with Partial Knowledge for Answering Questions with Unseen Entities](https://mlsanity.com/paper/2508.01290/) (2025)
- [A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models](https://mlsanity.com/paper/2603.29676/) (2026)
- ["Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms](https://mlsanity.com/paper/2606.12618/) (2026)
- [Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models](https://mlsanity.com/paper/2510.06107/) (2025)
- [Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AI](https://mlsanity.com/paper/2511.00230/) (2025)
- [Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's Taxonomy](https://mlsanity.com/paper/2602.17229/) (2026)
- [MUX: Continuous Reasoning via Multiplexed Tokens](https://mlsanity.com/paper/2607.18264/) (2026)
- [Building Better Activation Oracles](https://mlsanity.com/paper/2606.02609/) (2026)
- [Measure, Don't Optimize: Forecasting Recovery in LLM Unlearning](https://mlsanity.com/paper/2608.11408/) (2026)
- [DECOR: Auditing LLM Deception via Information Manipulation Theory](https://mlsanity.com/paper/2605.19270/) (2026)
- ["Don't Say It!": Constraints, Compliance, and Communication when Language Models Play Taboo](https://mlsanity.com/paper/2607.00601/) (2026)
- [Diff Mining: Logit Differences Reveal Finetuning Objectives](https://mlsanity.com/paper/2608.26462/) (2026)
- [Detecting Hidden Behaviors in LLMs via Activation-matched Finetuning](https://mlsanity.com/paper/2609.00351/) (2026)
- [On Emergent Capabilities and Model Merging](https://mlsanity.com/paper/2609.24504/) (2026)
- [Selecting The Most Informative Tokens in Natural Language Autoencoders](https://mlsanity.com/paper/2609.37040/) (2026)

## Abstract (arXiv)

As language models become more powerful and sophisticated, it is crucial that they remain trustworthy and reliable. There is concerning preliminary evidence that models may attempt to deceive or keep secrets from their operators. To explore the ability of current techniques to elicit such hidden knowledge, we train a Taboo model: a language model that describes a specific secret word without explicitly stating it. Importantly, the secret word is not presented to the model in its training data or prompt. We then investigate methods to uncover this secret. First, we evaluate non-interpretability (black-box) approaches. Subsequently, we develop largely automated strategies based on mechanistic interpretability techniques, including logit lens and sparse autoencoders. Evaluation shows that both approaches are effective in eliciting the secret word in our proof-of-concept setting. Our findings highlight the promise of these approaches for eliciting hidden knowledge and suggest several promising avenues for future work, including testing and refining these methods on more complex model organisms. This work aims to be a step towards addressing the crucial problem of eliciting secret knowledge from language models, thereby contributing to their safe and reliable deployment.