# Rep2Text: Decoding Full Text from a Single LLM Token Representation

arXiv: https://arxiv.org/abs/2511.06571 · HTML: https://mlsanity.com/paper/2511.06571/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 9 Nov 2025.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [Text Embeddings Reveal (Almost) As Much As Text](https://mlsanity.com/paper/2310.06816/) (2023)
- [Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence](https://mlsanity.com/paper/2305.03010/) (2023)
- [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025)
- [SelfIE: Self-Interpretation of Large Language Model Embeddings](https://mlsanity.com/paper/2403.10949/) (2024)
- [LatentQA: Teaching LLMs to Decode Activations Into Natural Language](https://mlsanity.com/paper/2412.08686/) (2024)
- [Transferable Embedding Inversion Attack: Uncovering Privacy Risks in Text Embeddings without Model Queries](https://mlsanity.com/paper/2406.10280/) (2024)
- [Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patching](https://mlsanity.com/paper/2311.15131/) (2023)
- [Depth Gives a False Sense of Privacy: LLM Internal States Inversion](https://mlsanity.com/paper/2507.16372/) (2025)
- [InversionView: A General-Purpose Method for Reading Information from Neural Activations](https://mlsanity.com/paper/2405.17653/) (2024)
- [InverseScope: Scalable Activation Inversion for Interpreting Large Language Models](https://mlsanity.com/paper/2506.07406/) (2025)

## Built on by (most cited first, 7 of 7)

- [A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious Minima](https://mlsanity.com/paper/2512.05534/) (2025)
- [When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models](https://mlsanity.com/paper/2605.08245/) (2026)
- [Capsule Lens: Locating and Tracking Concept Geometry in Model Representations](https://mlsanity.com/paper/2609.05575/) (2026)
- [Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2607.16295/) (2026)
- [Understanding and Exploiting Anisotropy in Post-Training](https://mlsanity.com/paper/2609.32792/) (2026)
- [Faithful Activation Verbalization: Reducing Hallucinations in LLM Representation Interpretation](https://mlsanity.com/paper/2609.34033/) (2026)
- [HeadEdit: Calibrating Language Model Behavior Through the Frozen Unembedding Matrix](https://mlsanity.com/paper/2610.01170/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved remarkable progress across diverse tasks, yet their internal mechanisms remain largely opaque. In this work, we investigate a fundamental question: to what extent can the original input text be recovered from a single last-token representation in an LLM? To this end, we propose Rep2Text, a novel framework for decoding text from last-token representations. Rep2Text employs a trainable adapter that maps a target model's last-token representation into the token embedding space of a decoding language model, which then autoregressively reconstructs the input text. Experiments across various model combinations (Llama-3.1-8B, Gemma-7B, Mistral-7B-v0.1, Llama-3.2-3B, etc.) show that, on average, roughly half of the tokens in 16-token sequences can be recovered from this compressed representation while preserving strong semantic coherence. Further analysis reveals a clear information bottleneck effect: as sequence length increases, token-level recovery declines, while semantic information remains relatively well preserved. We also find that scaling effects are less pronounced in inversion tasks. Finally, our framework demonstrates robust generalization to out-of-distribution clinical data.