mlsanityData to 14 Oct 2026

Towards eliciting latent knowledge from LLMs with mechanistic interpretability

arXiv 2505.14352 ↗First posted 20 May 2025cs.LGcited by 19

On the map: Mechanistic interpretability → Knowledge and representations inside LLMs

Abstract (arXiv)

As language models become more powerful and sophisticated, it is crucial that they remain trustworthy and reliable. There is concerning preliminary evidence that models may attempt to deceive or keep secrets from their operators. To explore the ability of current techniques to elicit such hidden knowledge, we train a Taboo model: a language model that describes a specific secret word without explicitly stating it. Importantly, the secret word is not presented to the model in its training data or prompt. We then investigate methods to uncover this secret. First, we evaluate non-interpretability (black-box) approaches. Subsequently, we develop largely automated strategies based on mechanistic interpretability techniques, including logit lens and sparse autoencoders. Evaluation shows that both approaches are effective in eliciting the secret word in our proof-of-concept setting. Our findings highlight the promise of these approaches for eliciting hidden knowledge and suggest several promising avenues for future work, including testing and refining these methods on more complex model organisms. This work aims to be a step towards addressing the crucial problem of eliciting secret knowledge from language models, thereby contributing to their safe and reliable deployment.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability. Builds on 15 of its 26 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

15 of 26 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  8. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  9. 2024Alignment faking in large language modelscited by 360
  10. 2024Frontier Models are Capable of In-context Schemingcited by 287
  11. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  12. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
3 more
  1. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
  2. 2025Auditing language models for hidden objectivescited by 83
  3. 2025LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformerscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 14.

19 papers.

Sort:

By area:

  1. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 28
  2. 2025Difficulties with Evaluating a Deception Detector for AIscited by 13
  3. 2026The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Modelscited by 7
  4. 2026The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodologycited by 6
  5. 2025Prompting Large Language Models with Partial Knowledge for Answering Questions with Unseen Entitiescited by 4
  6. 2026A Comprehensive Information-Decomposition Analysis of Large Vision-Language Modelscited by 4
  7. 2026"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organismscited by 4
  8. 2025Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Modelscited by 3
  9. 2025Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AIcited by 3
  10. 2026Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's Taxonomycited by 2
  11. 2026MUX: Continuous Reasoning via Multiplexed Tokenscited by 2
  12. 2026Building Better Activation Oraclescited by 1
7 more
  1. 2026Measure, Don't Optimize: Forecasting Recovery in LLM Unlearningcited by 1
  2. 2026DECOR: Auditing LLM Deception via Information Manipulation Theorycited by 0
  3. 2026"Don't Say It!": Constraints, Compliance, and Communication when Language Models Play Taboocited by 0
  4. 2026Diff Mining: Logit Differences Reveal Finetuning Objectivescited by 0
  5. 2026Detecting Hidden Behaviors in LLMs via Activation-matched Finetuningcited by 0
  6. 2026On Emergent Capabilities and Model Mergingcited by 0
  7. 2026Selecting The Most Informative Tokens in Natural Language Autoencoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Eliciting Secret Knowledge from Language Models
  2. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
  3. 2026MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Models
  4. 2026Can You Keep a Secret? Involuntary Information Leakage in Language Model Writing
  5. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspective
  6. 2025Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding
  7. 2026Causally Grounded Mechanistic Interpretability for LLMs with Faithful Natural-Language Explanations
  8. 2026Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it