# GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theory

arXiv: https://arxiv.org/abs/2406.11149 · HTML: https://mlsanity.com/paper/2406.11149/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 55 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Instruction Induction: From Few Examples to Natural Language Task Descriptions](https://mlsanity.com/paper/2205.10782/) (2022)
- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [JEC-QA: A Legal-Domain Question Answering Dataset](https://mlsanity.com/paper/1911.12011/) (2019)
- [Towards Zero-Label Language Learning](https://mlsanity.com/paper/2109.09193/) (2021)
- [A Brief Report on LawGPT 1.0: A Virtual Legal Assistant Based on GPT-3](https://mlsanity.com/paper/2302.05729/) (2023)
- [LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Model](https://mlsanity.com/paper/2406.04614/) (2024)
- [CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoning](https://mlsanity.com/paper/2401.07286/) (2024)
- [Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners](https://mlsanity.com/paper/2210.02969/) (2022)
- [AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimation](https://mlsanity.com/paper/2402.10646/) (2024)

## Built on by (most cited first, 30 of 37)

- [PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliance](https://mlsanity.com/paper/2502.17041/) (2025)
- [MCIP: Protecting MCP Safety via Model Contextual Integrity Protocol](https://mlsanity.com/paper/2505.14590/) (2025)
- [A Survey on Privacy Risks and Protection in Large Language Models](https://mlsanity.com/paper/2505.01976/) (2025)
- [Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory](https://mlsanity.com/paper/2408.10053/) (2024)
- [1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning](https://mlsanity.com/paper/2508.07667/) (2025)
- [Position: Privacy Is Not Just Memorization!](https://mlsanity.com/paper/2510.01645/) (2025)
- [CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agents](https://mlsanity.com/paper/2604.21308/) (2026)
- [Privacy Reasoning in Ambiguous Contexts](https://mlsanity.com/paper/2506.12241/) (2025)
- [Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation](https://mlsanity.com/paper/2512.16310/) (2025)
- [Large Language Models Meet Legal Artificial Intelligence: A Survey](https://mlsanity.com/paper/2509.09969/) (2025)
- [Context Reasoner: Incentivizing Reasoning Capability for Contextualized Privacy and Safety Compliance via Reinforcement Learning](https://mlsanity.com/paper/2505.14585/) (2025)
- [Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric](https://mlsanity.com/paper/2409.03735/) (2024)
- [Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliance](https://mlsanity.com/paper/2503.04736/) (2025)
- [Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answering](https://mlsanity.com/paper/2505.15805/) (2025)
- [Can Large Language Models Really Recognize Your Name?](https://mlsanity.com/paper/2505.14549/) (2025)
- [Personalizing Agent Privacy Decisions via Logical Entailment](https://mlsanity.com/paper/2512.05065/) (2025)
- [Modeling Subjectivity in Cognitive Appraisal with Language Models](https://mlsanity.com/paper/2503.11381/) (2025)
- [MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations](https://mlsanity.com/paper/2606.23217/) (2026)
- [EcomEdit: An Automated E-commerce Knowledge Editing Framework for Enhanced Product and Purchase Intention Understanding](https://mlsanity.com/paper/2410.14276/) (2024)
- [How Private are Language Models in Abstractive Summarization?](https://mlsanity.com/paper/2412.12040/) (2024)
- [Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Framework](https://mlsanity.com/paper/2508.12257/) (2025)
- [TERAG: Token-Efficient Graph-Based Retrieval-Augmented Generation](https://mlsanity.com/paper/2509.18667/) (2025)
- [Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance](https://mlsanity.com/paper/2509.22250/) (2025)
- [MPCI-Bench: A Benchmark for Multimodal Pairwise Contextual Integrity Evaluation of Language Model Agents](https://mlsanity.com/paper/2601.08235/) (2026)
- [Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations](https://mlsanity.com/paper/2604.00209/) (2026)
- [PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems](https://mlsanity.com/paper/2607.05318/) (2026)
- [Ustnlp16 at SemEval-2025 Task 9: Improving Model Performance through Imbalance Handling and Focal Loss](https://mlsanity.com/paper/2505.00021/) (2025)
- [PrivacyXray: Detecting Privacy Breaches in LLMs through Semantic Consistency and Probability Certainty](https://mlsanity.com/paper/2506.19563/) (2025)
- [All Required, In Order: Phase-Level Evaluation for AI-Human Dialogue in Healthcare and Beyond](https://mlsanity.com/paper/2601.08690/) (2026)
- [Safer Policy Compliance with Dynamic Epistemic Fallback](https://mlsanity.com/paper/2601.23094/) (2026)

## Abstract (arXiv)

Privacy issues arise prominently during the inappropriate transmission of information between entities. Existing research primarily studies privacy by exploring various privacy attacks, defenses, and evaluations within narrowly predefined patterns, while neglecting that privacy is not an isolated, context-free concept limited to traditionally sensitive data (e.g., social security numbers), but intertwined with intricate social contexts that complicate the identification and analysis of potential privacy violations. The advent of Large Language Models (LLMs) offers unprecedented opportunities for incorporating the nuanced scenarios outlined in privacy laws to tackle these complex privacy issues. However, the scarcity of open-source relevant case studies restricts the efficiency of LLMs in aligning with specific legal statutes. To address this challenge, we introduce a novel framework, GoldCoin, designed to efficiently ground LLMs in privacy laws for judicial assessing privacy violations. Our framework leverages the theory of contextual integrity as a bridge, creating numerous synthetic scenarios grounded in relevant privacy statutes (e.g., HIPAA), to assist LLMs in comprehending the complex contexts for identifying privacy risks in the real world. Extensive experimental results demonstrate that GoldCoin markedly enhances LLMs' capabilities in recognizing privacy risks across real court cases, surpassing the baselines on different judicial tasks.