mlsanityData to 7 Oct 2026

Information Leakage in Embedding Models

arXiv 2004.00053 ↗First posted 31 Mar 2020cs.LG cs.CL cs.CRcited by 199

Not on the map. Why?

Cited by 40 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Embeddings are functions that map raw input data to low-dimensional vector representations, while preserving important semantic information about the inputs. Pre-training embeddings on a large amount of unlabeled data and fine-tuning them for downstream tasks is now a de facto standard in achieving state of the art learning in many domains. We demonstrate that embeddings, in addition to encoding generic semantics, often also present a vector that leaks sensitive information about the input data. We develop three classes of attacks to systematically study information that might be leaked by embeddings. First, embedding vectors can be inverted to partially recover some of the input data. As an example, we show that our attacks on popular sentence embeddings recover between 50%--70% of the input words (F1 scores of 0.5--0.7). Second, embeddings may reveal sensitive attributes inherent in inputs and independent of the underlying semantic task at hand. Attributes such as authorship of text can be easily extracted by training an inference model on just a handful of labeled embedding vectors. Third, embedding models leak moderate amount of membership information for infrequent training data inputs. We extensively evaluate our attacks on various state-of-the-art embedding models in the text domain. We also propose and evaluate defenses that can prevent the leakage to some extent at a minor cost in utility.

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 27; 2023: 34; 2024: 49; 2025: 40; 2026: 33.

199 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  3. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  4. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  5. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  6. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  7. 2023Large Language Model Alignment: A Surveycited by 241
  8. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  9. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  10. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  11. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  12. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
38 more
  1. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  2. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  3. 2023Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentencecited by 81
  4. 2021ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Modelscited by 79
  5. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  6. 2021Differential Privacy for Text Analytics via Natural Text Sanitizationcited by 74
  7. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  8. 2021EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learningcited by 68
  9. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  10. 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
  11. 2023Language Model Inversioncited by 66
  12. 2022The Limits of Word Level Differential Privacycited by 63
  13. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 55
  14. 2021Natural Language Understanding with Privacy-Preserving BERTcited by 54
  15. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  16. 2025Harnessing the Universal Geometry of Embeddingscited by 54
  17. 2022A Customized Text Sanitization Mechanism with Differential Privacycited by 51
  18. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  19. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  20. 2023Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Servicescited by 43
  21. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  22. 2024Extracting Prompts by Inverting LLM Outputscited by 40
  23. 2023Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimationcited by 36
  24. 2021Quantifying and Mitigating Privacy Risks of Contrastive Learningcited by 35
  25. 2023PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Modelscited by 35
  26. 2021Membership Inference on Word Embedding and Beyondcited by 34
  27. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 31
  28. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  29. 2024Membership Inference Attacks against Large Vision-Language Modelscited by 31
  30. 2022Bilateral Dependency Optimization: Defending Against Model-inversion Attackscited by 30
  31. 2022M^4I: Multi-modal Models Membership Inferencecited by 30
  32. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  33. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
  34. 2023Integration of Large Language Models and Federated Learningcited by 28
  35. 2024Text Embedding Inversion Security for Multilingual Language Modelscited by 26
  36. 2024Transferable Embedding Inversion Attack: Uncovering Privacy Risks in Text Embeddings without Model Queriescited by 25
  37. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  38. 2024Digger: Detecting Copyright Content Mis-usage in Large Language Model Trainingcited by 24

50 most cited of 199 All 199 (JSON)