# SelfIE: Self-Interpretation of Large Language Model Embeddings

arXiv: https://arxiv.org/abs/2403.10949 · HTML: https://mlsanity.com/paper/2403.10949/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 16 Mar 2024.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Interpreting CLIP's Image Representation via Text-Based Decomposition](https://mlsanity.com/paper/2310.05916/) (2023)
- [Synthesizing the preferred inputs for neurons in neural networks via deep generator networks](https://mlsanity.com/paper/1605.09304/) (2016)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [Generating Visual Explanations](https://mlsanity.com/paper/1603.08507/) (2016)
- [Inspecting and Editing Knowledge Representations in Language Models](https://mlsanity.com/paper/2304.00740/) (2023)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)

## Built on by (most cited first, 30 of 67)

- [Automatically Interpreting Millions of Features in Large Language Models](https://mlsanity.com/paper/2410.13928/) (2024)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers](https://mlsanity.com/paper/2512.15674/) (2025)
- [LatentQA: Teaching LLMs to Decode Activations Into Natural Language](https://mlsanity.com/paper/2412.08686/) (2024)
- [Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers](https://mlsanity.com/paper/2411.08745/) (2024)
- [Who's asking? User personas and the mechanics of latent misalignment](https://mlsanity.com/paper/2406.12094/) (2024)
- [One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2410.22366/) (2024)
- [ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2411.15268/) (2024)
- [Training Language Models to Explain Their Own Computations](https://mlsanity.com/paper/2511.08579/) (2025)
- [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025)
- [Controllable Context Sensitivity and the Knob Behind It](https://mlsanity.com/paper/2411.07404/) (2024)
- [MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs](https://mlsanity.com/paper/2505.24858/) (2025)
- [Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race](https://mlsanity.com/paper/2506.00253/) (2025)
- [Subliminal Steering: Stronger Encoding of Hidden Signals](https://mlsanity.com/paper/2604.25783/) (2026)
- [Monet: Mixture of Monosemantic Experts for Transformers](https://mlsanity.com/paper/2412.04139/) (2024)
- [RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs](https://mlsanity.com/paper/2502.07424/) (2025)
- [Beyond the Black Box: Interpretability of LLMs in Finance](https://mlsanity.com/paper/2505.24650/) (2025)
- [Learning to Interpret Weight Differences in Language Models](https://mlsanity.com/paper/2510.05092/) (2025)
- [Do Activation Verbalization Methods Convey Privileged Information?](https://mlsanity.com/paper/2509.13316/) (2025)
- [Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation](https://mlsanity.com/paper/2505.23701/) (2025)
- [Efficient and accurate steering of Large Language Models through attention-guided feature learning](https://mlsanity.com/paper/2602.00333/) (2026)
- [Learning a Generative Meta-Model of LLM Activations](https://mlsanity.com/paper/2602.06964/) (2026)
- [LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMs](https://mlsanity.com/paper/2505.10838/) (2025)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [Rep2Text: Decoding Full Text from a Single LLM Token Representation](https://mlsanity.com/paper/2511.06571/) (2025)
- [Designing Role Vectors to Improve LLM Inference Behaviour](https://mlsanity.com/paper/2502.12055/) (2025)
- [Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation](https://mlsanity.com/paper/2508.12040/) (2025)
- [Meta-Models: An Architecture for Decoding LLM Behaviors Through Interpreted Embeddings and Natural Language](https://mlsanity.com/paper/2410.02472/) (2024)
- [Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models](https://mlsanity.com/paper/2411.04291/) (2024)

## Abstract (arXiv)

How do large language models (LLMs) obtain their answers? The ability to explain and control an LLM's reasoning process is key for reliability, transparency, and future model developments. We propose SelfIE (Self-Interpretation of Embeddings), a framework that enables LLMs to interpret their own embeddings in natural language by leveraging their ability to respond to inquiries about a given passage. Capable of interpreting open-world concepts in the hidden embeddings, SelfIE reveals LLM internal reasoning in cases such as making ethical decisions, internalizing prompt injection, and recalling harmful knowledge. SelfIE's text descriptions on hidden embeddings also open up new avenues to control LLM reasoning. We propose Supervised Control, which allows editing open-ended concepts while only requiring gradient computation of individual layer. We extend RLHF to hidden embeddings and propose Reinforcement Control that erases harmful knowledge in LLM without supervision targets.