# Language Model Inversion

arXiv: https://arxiv.org/abs/2311.13647 · HTML: https://mlsanity.com/paper/2311.13647/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 44 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 23 of 23)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [Inverting Visual Representations with Convolutional Networks](https://mlsanity.com/paper/1506.02753/) (2015)
- [Thieves on Sesame Street! Model Extraction of BERT-based APIs](https://mlsanity.com/paper/1910.12366/) (2019)
- [LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions](https://mlsanity.com/paper/2304.14402/) (2023)
- [Imitation Attacks and Defenses for Black-box Machine Translation Systems](https://mlsanity.com/paper/2004.15015/) (2020)
- [Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence](https://mlsanity.com/paper/2305.03010/) (2023)
- [High Fidelity Visualization of What Your Self-Supervised Representation Knows About](https://mlsanity.com/paper/2112.09164/) (2021)
- [Text Revealer: Private Text Reconstruction via Model Inversion Attacks against Transformers](https://mlsanity.com/paper/2209.10505/) (2022)
- [A framework for the extraction of Deep Neural Networks by leveraging public data](https://mlsanity.com/paper/1905.09165/) (2019)
- [Understanding Invariance via Feedforward Inversion of Discriminatively Trained Classifiers](https://mlsanity.com/paper/2103.07470/) (2021)
- [Breaching FedMD: Image Recovery via Paired-Logits Inversion Attack](https://mlsanity.com/paper/2304.11436/) (2023)

## Built on by (most cited first, 30 of 68)

- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Weak-to-Strong Jailbreaking on Large Language Models](https://mlsanity.com/paper/2401.17256/) (2024)
- [Coercing LLMs to do and reveal (almost) anything](https://mlsanity.com/paper/2402.14020/) (2024)
- [Query-Based Adversarial Prompt Generation](https://mlsanity.com/paper/2402.12329/) (2024)
- [Discrete Diffusion in Large Language and Multimodal Models: A Survey](https://mlsanity.com/paper/2506.13759/) (2025)
- [Harnessing the Universal Geometry of Embeddings](https://mlsanity.com/paper/2505.12540/) (2025)
- [PAL: Proxy-Guided Black-Box Attack on Large Language Models](https://mlsanity.com/paper/2402.09674/) (2024)
- [Extracting Prompts by Inverting LLM Outputs](https://mlsanity.com/paper/2405.15012/) (2024)
- [Logits of API-Protected LLMs Leak Proprietary Information](https://mlsanity.com/paper/2403.09539/) (2024)
- [Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defenses](https://mlsanity.com/paper/2402.04013/) (2024)
- [Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models](https://mlsanity.com/paper/2404.01231/) (2024)
- [Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking](https://mlsanity.com/paper/2409.08045/) (2024)
- [Privacy Preserving Prompt Engineering: A Survey](https://mlsanity.com/paper/2404.06001/) (2024)
- [Eliciting Language Model Behaviors with Investigator Agents](https://mlsanity.com/paper/2502.01236/) (2025)
- [Privacy Auditing of Large Language Models](https://mlsanity.com/paper/2503.06808/) (2025)
- [LLM-based Multi-Agent Systems: Techniques and Business Perspectives](https://mlsanity.com/paper/2411.14033/) (2024)
- [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026)
- [PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs](https://mlsanity.com/paper/2409.14729/) (2024)
- [Depth Gives a False Sense of Privacy: LLM Internal States Inversion](https://mlsanity.com/paper/2507.16372/) (2025)
- [Black-Box Adversarial Attacks on LLM-Based Code Completion](https://mlsanity.com/paper/2408.02509/) (2024)
- [PRSA: Prompt Stealing Attacks against Real-World Prompt Services](https://mlsanity.com/paper/2402.19200/) (2024)
- [Single-pass Detection of Jailbreaking Input in Large Language Models](https://mlsanity.com/paper/2502.15435/) (2025)
- [Better Language Model Inversion by Compactly Representing Next-Token Distributions](https://mlsanity.com/paper/2506.17090/) (2025)
- [Conditional Data Synthesis Augmentation](https://mlsanity.com/paper/2504.07426/) (2025)
- [LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasures](https://mlsanity.com/paper/2505.01177/) (2025)
- [Diffusion LLMs are Natural Adversaries for any LLM](https://mlsanity.com/paper/2511.00203/) (2025)
- [DP-Fusion: Token-Level Differentially Private Inference for Large Language Models](https://mlsanity.com/paper/2507.04531/) (2025)
- [Privacy Risks of LLM-Empowered Recommender Systems: An Inversion Attack Perspective](https://mlsanity.com/paper/2508.03703/) (2025)
- [Beyond Data Privacy: New Privacy Risks for Large Language Models](https://mlsanity.com/paper/2509.14278/) (2025)
- [Privacy-Preserving LLM Embedding Transmission for End-Cloud Collaboration](https://mlsanity.com/paper/2503.12896/) (2025)

## Abstract (arXiv)

Language models produce a distribution over the next token; can we use this information to recover the prompt tokens? We consider the problem of language model inversion and show that next-token probabilities contain a surprising amount of information about the preceding text. Often we can recover the text in cases where it is hidden from the user, motivating a method for recovering unknown prompts given only the model's current distribution output. We consider a variety of model access scenarios, and show how even without predictions for every token in the vocabulary we can recover the probability vector through search. On Llama-2 7b, our inversion method reconstructs prompts with a BLEU of 59 and token-level F1 of 78 and recovers 27% of prompts exactly. Code for reproducing all experiments is available at http://github.com/jxmorris12/vec2text.