mlsanityData to 7 Oct 2026

Extracting Prompts by Inverting LLM Outputs

arXiv 2405.15012 ↗First posted 23 May 2024cs.CL cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

We consider the problem of language model inversion: given outputs of a language model, we seek to extract the prompt that generated these outputs. We develop a new black-box method, output2prompt, that learns to extract prompts without access to the model's logits and without adversarial or jailbreaking queries. In contrast to previous work, output2prompt only needs outputs of normal user queries. To improve memory efficiency, output2prompt employs a new sparse encoding techique. We measure the efficacy of output2prompt on a variety of user and system prompts and demonstrate zero-shot transferability across different LLMs.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

27 of 39 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Mistral 7Bcited by 3,065
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2024Mixtral of Expertscited by 1,785
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  12. 2016Stealing Machine Learning Models via Prediction APIscited by 602
15 more
  1. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  2. 2014Understanding Deep Image Representations by Inverting Themcited by 399
  3. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  4. 2019High Accuracy and High Fidelity Extraction of Neural Networkscited by 208
  5. 2020Information Leakage in Embedding Modelscited by 199
  6. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  7. 2015Inverting Visual Representations with Convolutional Networkscited by 168
  8. 2024Stealing Part of a Production Language Modelcited by 152
  9. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  10. 2023Effective Prompt Extraction from Language Modelscited by 83
  11. 2023Language Model Inversioncited by 66
  12. 2024Query-Based Adversarial Prompt Generationcited by 55
  13. 2024Prompt Stealing Attacks Against Large Language Modelscited by 39
  14. 2024Logits of API-Protected LLMs Leak Proprietary Informationcited by 38
  15. 2020Vec2Face: Unveil Human Faces from their Blackbox Features in Face Recognitioncited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 22; 2026: 11.

40 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  4. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  5. 2025Harnessing the Universal Geometry of Embeddingscited by 54
  6. 2025Language Models are Injective and Hence Invertiblecited by 31
  7. 2025A Survey on Model Extraction Attacks and Defenses for Large Language Modelscited by 18
  8. 2025GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluationcited by 17
  9. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 16
  10. 2025Depth Gives a False Sense of Privacy: LLM Internal States Inversioncited by 14
  11. 2024Model Inversion Attacks: A Survey of Approaches and Countermeasurescited by 11
  12. 2025Better Language Model Inversion by Compactly Representing Next-Token Distributionscited by 9
28 more
  1. 2025Universal Zero-shot Embedding Inversioncited by 8
  2. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  3. 2025Federated In-Context Learning: Iterative Refinement for Improved Answer Qualitycited by 7
  4. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
  5. 2025Privacy Risks of LLM-Empowered Recommender Systems: An Inversion Attack Perspectivecited by 6
  6. 2025When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Pluginscited by 6
  7. 2024Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizerscited by 5
  8. 2025DP-Fusion: Token-Level Differentially Private Inference for Large Language Modelscited by 5
  9. 2024Rank Matters: Understanding and Defending Model Inversion Attacks via Low-Rank Feature Filteringcited by 4
  10. 2024Mitigating Paraphrase Attacks on Machine-Text Detectors via Paraphrase Inversioncited by 4
  11. 2024PromptKeeper: Safeguarding System Prompts for LLMscited by 4
  12. 2025Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Promptscited by 3
  13. 2025SoK: Semantic Privacy in Large Language Modelscited by 3
  14. 2026Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMscited by 3
  15. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  16. 2025IPAD: Inverse Prompt for AI Detection - A Robust and Interpretable LLM-Generated Text Detectorcited by 2
  17. 2025Prada: Black-Box LLM Adaptation with Private Data on Resource-Constrained Devicescited by 2
  18. 2026Small Reward Models via Backward Inferencecited by 2
  19. 2025Label Anything: An Interpretable, High-Fidelity and Prompt-Free Annotatorcited by 1
  20. 2026Buy versus Build an LLM: A Decision Framework for Governmentscited by 1
  21. 2025A global log for medical AIcited by 0
  22. 2026Synapse: Federated Tool Routing via Typed Compendium Artifactscited by 0
  23. 2026Language Model Inversion through End-to-End Differentiationcited by 0
  24. 2026PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstructioncited by 0
  25. 2026SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channelscited by 0
  26. 2026Auditing and Mitigating Privacy Leakage in Cloud-Edge Collaborative Decodingcited by 0
  27. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  28. 2026HeadEdit: Calibrating Language Model Behavior Through the Frozen Unembedding Matrixcited by 0