# Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory

arXiv: https://arxiv.org/abs/2408.10053 · HTML: https://mlsanity.com/paper/2408.10053/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Builds on (most cited first, 20 of 20)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [On the Exploitability of Instruction Tuning](https://mlsanity.com/paper/2306.17194/) (2023)
- [Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence](https://mlsanity.com/paper/2305.03010/) (2023)
- [GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theory](https://mlsanity.com/paper/2406.11149/) (2024)
- [LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Model](https://mlsanity.com/paper/2406.04614/) (2024)
- [You Don't Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers' Private Personas](https://mlsanity.com/paper/2205.10228/) (2022)
- [OPPO: An Ontology for Describing Fine-Grained Data Practices in Privacy Policies of Online Social Networks](https://mlsanity.com/paper/2309.15971/) (2023)

## Built on by (most cited first, 18 of 18)

- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliance](https://mlsanity.com/paper/2502.17041/) (2025)
- [MCIP: Protecting MCP Safety via Model Contextual Integrity Protocol](https://mlsanity.com/paper/2505.14590/) (2025)
- [1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning](https://mlsanity.com/paper/2508.07667/) (2025)
- [Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent](https://mlsanity.com/paper/2411.01344/) (2024)
- [PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI](https://mlsanity.com/paper/2512.24848/) (2025)
- [Privacy Reasoning in Ambiguous Contexts](https://mlsanity.com/paper/2506.12241/) (2025)
- [Context Reasoner: Incentivizing Reasoning Capability for Contextualized Privacy and Safety Compliance via Reinforcement Learning](https://mlsanity.com/paper/2505.14585/) (2025)
- [Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric](https://mlsanity.com/paper/2409.03735/) (2024)
- [Can Large Language Models Really Recognize Your Name?](https://mlsanity.com/paper/2505.14549/) (2025)
- [It's My Agent, Not Me: Personal Privacy Boundaries in AI-Delegated Information Sharing](https://mlsanity.com/paper/2509.21712/) (2025)
- [Backdoor-Powered Prompt Injection Attacks Nullify Defense Methods](https://mlsanity.com/paper/2510.03705/) (2025)
- [Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance](https://mlsanity.com/paper/2509.22250/) (2025)
- [PrivacyReasoner: Can LLM Emulate a Human-like Privacy Mind?](https://mlsanity.com/paper/2601.09152/) (2026)
- [It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs](https://mlsanity.com/paper/2605.20258/) (2026)
- [The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques](https://mlsanity.com/paper/2603.08207/) (2026)
- [ContextLens: Modeling Imperfect Privacy and Safety Context for Legal Compliance](https://mlsanity.com/paper/2604.12308/) (2026)
- [RedactionBench](https://mlsanity.com/paper/2606.18782/) (2026)

## Abstract (arXiv)

Privacy research has attracted wide attention as individuals worry that their private data can be easily leaked during interactions with smart devices, social platforms, and AI applications. Computer science researchers, on the other hand, commonly study privacy issues through privacy attacks and defenses on segmented fields. Privacy research is conducted on various sub-fields, including Computer Vision (CV), Natural Language Processing (NLP), and Computer Networks. Within each field, privacy has its own formulation. Though pioneering works on attacks and defenses reveal sensitive privacy issues, they are narrowly trapped and cannot fully cover people's actual privacy concerns. Consequently, the research on general and human-centric privacy research remains rather unexplored. In this paper, we formulate the privacy issue as a reasoning problem rather than simple pattern matching. We ground on the Contextual Integrity (CI) theory which posits that people's perceptions of privacy are highly correlated with the corresponding social context. Based on such an assumption, we develop the first comprehensive checklist that covers social identities, private attributes, and existing privacy regulations. Unlike prior works on CI that either cover limited expert annotated norms or model incomplete social context, our proposed privacy checklist uses the whole Health Insurance Portability and Accountability Act of 1996 (HIPAA) as an example, to show that we can resort to large language models (LLMs) to completely cover the HIPAA's regulations. Additionally, our checklist also gathers expert annotations across multiple ontologies to determine private information including but not limited to personally identifiable information (PII). We use our preliminary results on the HIPAA to shed light on future context-centric privacy research to cover more privacy regulations, social norms and standards.