# Rethinking Privacy in Machine Learning Pipelines from an Information Flow Control Perspective

arXiv: https://arxiv.org/abs/2311.15792 · HTML: https://mlsanity.com/paper/2311.15792/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Differentially Private Learning Needs Better Features (or Much More Data)](https://mlsanity.com/paper/2011.11660/) (2020)
- [Distilling Knowledge from Reader to Retriever for Question Answering](https://mlsanity.com/paper/2012.04584/) (2020)
- [Unlocking High-Accuracy Differentially Private Image Classification through Scale](https://mlsanity.com/paper/2204.13650/) (2022)
- [Re-Imagen: Retrieval-Augmented Text-to-Image Generator](https://mlsanity.com/paper/2209.14491/) (2022)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Machine Learning with Membership Privacy using Adversarial Regularization](https://mlsanity.com/paper/1807.05852/) (2018)
- [KNN-Diffusion: Image Generation via Large-Scale Retrieval](https://mlsanity.com/paper/2204.02849/) (2022)
- [Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining](https://mlsanity.com/paper/2212.06470/) (2022)
- [Adaptive Semiparametric Language Models](https://mlsanity.com/paper/2102.02557/) (2021)
- [AttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine Learning](https://mlsanity.com/paper/1805.04810/) (2018)
- [Augmenting Transformers with KNN-Based Composite Memory for Dialogue](https://mlsanity.com/paper/2004.12744/) (2020)
- [Passage Retrieval for Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2105.03938/) (2021)
- [Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Search](https://mlsanity.com/paper/2006.07548/) (2020)
- [Information Flow Control in Machine Learning through Modular Model Architecture](https://mlsanity.com/paper/2306.03235/) (2023)

## Built on by (most cited first, 16 of 16)

- [Instruction Fine-Tuning: Does Prompt Loss Matter?](https://mlsanity.com/paper/2401.13586/) (2024)
- [Privacy Preserving Prompt Engineering: A Survey](https://mlsanity.com/paper/2404.06001/) (2024)
- [ConfusedPilot: Confused Deputy Risks in RAG-based LLMs](https://mlsanity.com/paper/2408.04870/) (2024)
- [Permissive Information-Flow Analysis for Large Language Models](https://mlsanity.com/paper/2410.03055/) (2024)
- [MeMemo: On-device Retrieval Augmentation for Private and Personalized Text Generation](https://mlsanity.com/paper/2407.01972/) (2024)
- [TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval](https://mlsanity.com/paper/2502.20969/) (2025)
- [Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.19101/) (2025)
- [DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning](https://mlsanity.com/paper/2403.14421/) (2024)
- [DOMBA: Double Model Balancing for Access-Controlled Language Models via Minimum-Bounded Aggregation](https://mlsanity.com/paper/2408.11121/) (2024)
- [Position: Capability Control Should be a Separate Goal From Alignment](https://mlsanity.com/paper/2602.05164/) (2026)
- [SoK: A Systems Perspective on Compound AI Threats and Countermeasures](https://mlsanity.com/paper/2411.13459/) (2024)
- [WebANNS: Fast and Efficient Approximate Nearest Neighbor Search in Web Browsers](https://mlsanity.com/paper/2507.00521/) (2025)
- [How To Think About End-To-End Encryption and AI: Training, Processing, Disclosure, and Consent](https://mlsanity.com/paper/2412.20231/) (2024)
- [CoRAG: Collaborative Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.01883/) (2025)
- [Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Direction](https://mlsanity.com/paper/2510.15422/) (2025)
- [Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories](https://mlsanity.com/paper/2609.36414/) (2026)

## Abstract (arXiv)

Modern machine learning systems use models trained on ever-growing corpora. Typically, metadata such as ownership, access control, or licensing information is ignored during training. Instead, to mitigate privacy risks, we rely on generic techniques such as dataset sanitization and differentially private model training, with inherent privacy/utility trade-offs that hurt model performance. Moreover, these techniques have limitations in scenarios where sensitive information is shared across multiple participants and fine-grained access control is required. By ignoring metadata, we therefore miss an opportunity to better address security, privacy, and confidentiality challenges. In this paper, we take an information flow control perspective to describe machine learning systems, which allows us to leverage metadata such as access control policies and define clear-cut privacy and confidentiality guarantees with interpretable information flows. Under this perspective, we contrast two different approaches to achieve user-level non-interference: 1) fine-tuning per-user models, and 2) retrieval augmented models that access user-specific datasets at inference time. We compare these two approaches to a trivially non-interfering zero-shot baseline using a public model and to a baseline that fine-tunes this model on the whole corpus. We evaluate trained models on two datasets of scientific articles and demonstrate that retrieval augmented architectures deliver the best utility, scalability, and flexibility while satisfying strict non-interference guarantees.