# SoK: Semantic Privacy in Large Language Models

arXiv: https://arxiv.org/abs/2506.23603 · HTML: https://mlsanity.com/paper/2506.23603/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 17 of 17)

- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies](https://mlsanity.com/paper/2407.19354/) (2024)
- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [PLeak: Prompt Leaking Attacks against Large Language Model Applications](https://mlsanity.com/paper/2405.06823/) (2024)
- [Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents](https://mlsanity.com/paper/2411.09523/) (2024)
- [Privacy Issues in Large Language Models: A Survey](https://mlsanity.com/paper/2312.06717/) (2023)
- [Membership Inference Attacks Against In-Context Learning](https://mlsanity.com/paper/2409.01380/) (2024)
- [A Comprehensive Overview of Backdoor Attacks in Large Language Models within Communication Networks](https://mlsanity.com/paper/2308.14367/) (2023)
- [Extracting Prompts by Inverting LLM Outputs](https://mlsanity.com/paper/2405.15012/) (2024)
- [Strong and weak alignment of large language models with human values](https://mlsanity.com/paper/2408.04655/) (2024)
- [Noisy Neighbors: Efficient membership inference attacks against LLMs](https://mlsanity.com/paper/2406.16565/) (2024)
- [TexShape: Information Theoretic Sentence Embedding for Language Models](https://mlsanity.com/paper/2402.05132/) (2024)
- [SentinelLMs: Encrypted Input Adaptation and Fine-tuning of Language Models for Private and Secure Inference](https://mlsanity.com/paper/2312.17342/) (2023)
- [LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasures](https://mlsanity.com/paper/2505.01177/) (2025)
- [UniErase: Towards Balanced and Precise Unlearning in Language Models](https://mlsanity.com/paper/2505.15674/) (2025)
- [Information Leakage of Sentence Embeddings via Generative Embedding Inversion Attacks](https://mlsanity.com/paper/2504.16609/) (2025)

## Built on by (most cited first, 3 of 3)

- [SoK: Agentic Skills -- Beyond Tool Use in LLM Agents](https://mlsanity.com/paper/2602.20867/) (2026)
- [Prompt to Pwn: Automated Exploit Generation for Smart Contracts](https://mlsanity.com/paper/2508.01371/) (2025)
- [PlanTwin: Privacy-Preserving Planning Abstractions for Cloud-Assisted LLM Agents](https://mlsanity.com/paper/2603.18377/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) are increasingly deployed in sensitive domains, traditional data privacy measures prove inadequate for protecting information that is implicit, contextual, or inferable - what we define as semantic privacy. This Systematization of Knowledge (SoK) introduces a lifecycle-centric framework to analyze how semantic privacy risks emerge across input processing, pretraining, fine-tuning, and alignment stages of LLMs. We categorize key attack vectors and assess how current defenses, such as differential privacy, embedding encryption, edge computing, and unlearning, address these threats. Our analysis reveals critical gaps in semantic-level protection, especially against contextual inference and latent representation leakage. We conclude by outlining open challenges, including quantifying semantic leakage, protecting multimodal inputs, balancing de-identification with generation quality, and ensuring transparency in privacy enforcement. This work aims to inform future research on designing robust, semantically aware privacy-preserving techniques for LLMs.