# Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings

arXiv: https://arxiv.org/abs/2506.15001 · HTML: https://mlsanity.com/paper/2506.15001/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models](https://mlsanity.com/paper/2210.03162/) (2022)
- [Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence](https://mlsanity.com/paper/2305.03010/) (2023)
- [Prompt Compression for Large Language Models: A Survey](https://mlsanity.com/paper/2410.12388/) (2024)
- [InvBERT: Reconstructing Text from Contextualized Word Embeddings by inverting the BERT pipeline](https://mlsanity.com/paper/2109.10104/) (2021)

## Built on by (most cited first, 6 of 6)

- [TokMem: One-Token Procedural Memory for Large Language Models](https://mlsanity.com/paper/2510.00444/) (2025)
- [Skill Neologisms: Towards Skill-based Continual Learning](https://mlsanity.com/paper/2605.04970/) (2026)
- [Refine Thought: A Test-Time Inference Method for Embedding Model Reasoning](https://mlsanity.com/paper/2511.13726/) (2025)
- [Learning a Single Token to Replace Long System Prompts in LLMs](https://mlsanity.com/paper/2511.23271/) (2025)
- [Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings](https://mlsanity.com/paper/2601.11124/) (2026)
- [Concept Tokens: Learning Behavioral Embeddings Through Concept Definitions](https://mlsanity.com/paper/2601.04465/) (2026)

## Abstract (arXiv)

In this work, we observe an interesting phenomenon: it is possible to generate reversible sentence embeddings that allow an LLM to reconstruct the original text exactly, without modifying the model's weights. This is achieved by introducing a special memory token, whose embedding is optimized through training on a fixed sequence. When prompted with this embedding, the model reconstructs the fixed sequence exactly. We evaluate this phenomenon across English and Spanish datasets, sequences of up to approximately 240 tokens, and model scales ranging from 100M to 8B parameters. Notably, Llama 3.1 8B successfully reconstructs all tested sequences. Our findings highlight an interesting capability of LLMs and suggest potential applications in memory-based retrieval, compression, and controlled text generation.