mlsanityData to 7 Oct 2026

Information Flow Control in Machine Learning through Modular Model Architecture

arXiv 2306.03235 ↗First posted 5 Jun 2023cs.LG cs.CRcited by 11

Not on the map. Why?

Abstract (arXiv)

In today's machine learning (ML) models, any part of the training data can affect the model output. This lack of control for information flow from training data to model output is a major obstacle in training models on sensitive data when access control only allows individual users to access a subset of data. To enable secure machine learning for access-controlled data, we propose the notion of information flow control for machine learning, and develop an extension to the Transformer language model architecture that strictly adheres to the IFC definition we propose. Our architecture controls information flow by limiting the influence of training data from each security domain to a single expert module, and only enables a subset of experts at inference time based on the access control policy.The evaluation using large text and code datasets show that our proposed parametric IFC architecture has minimal (1.9%) performance overhead and can significantly improve model accuracy (by 38% for the text dataset, and between 44%--62% for the code datasets) by enabling training on access-controlled data.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 52 of its 80 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

52 of 80 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2016Deep Learning with Differential Privacycited by 2,682
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
38 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2019Machine Unlearningcited by 1,020
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  9. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  11. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  12. 2023Extracting Training Data from Diffusion Modelscited by 723
  13. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  14. 2019ELI5: Long Form Question Answeringcited by 567
  15. 2019Certified Data Removal from Machine Learning Modelscited by 512
  16. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  17. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  18. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  19. 2021Remember What You Want to Forget: Algorithms for Machine Unlearningcited by 360
  20. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  21. 2020The Pushshift Reddit Datasetcited by 289
  22. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 241
  23. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  24. 2021Adaptive Machine Unlearningcited by 164
  25. 2022Reconstructing Training Data with Informed Adversariescited by 143
  26. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  27. 2019Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learningcited by 117
  28. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  29. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 64
  30. 2018Privacy-preserving Predictioncited by 48
  31. 2020Formalizing Data Deletion in the Context of the Right to be Forgottencited by 46
  32. 2020Towards Probabilistic Verification of Machine Unlearningcited by 46
  33. 2022Bounding Training Data Reconstruction in Private (Deep) Learningcited by 43
  34. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  35. 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
  36. 2022Cocktail Party Attack: Breaking Aggregation-Based Privacy in Federated Learning using Independent Component Analysiscited by 29
  37. 2022Analyzing Privacy Leakage in Machine Learning via Multiple Hypothesis Testing: A Lesson From Fanocited by 16
  38. 2022Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learnerscited by 15

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 3; 2026: 3.

11 papers.

  1. 2024ConfusedPilot: Confused Deputy Risks in RAG-based LLMscited by 22
  2. 2023Rethinking Privacy in Machine Learning Pipelines from an Information Flow Control Perspectivecited by 16
  3. 2025Systems Security Foundations for Agentic Computingcited by 9
  4. 2024DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuningcited by 4
  5. 2025Enterprise AI Must Enforce Participant-Aware Access Controlcited by 3
  6. 2025Can LLMs Make (Personalized) Access Control Decisions?cited by 3
  7. 2024DOMBA: Double Model Balancing for Access-Controlled Language Models via Minimum-Bounded Aggregationcited by 2
  8. 2026Agent Security is a Systems Problemcited by 2
  9. 2024SoK: A Systems Perspective on Compound AI Threats and Countermeasurescited by 1
  10. 2026Chain-of-Authorization: Embedding authorization into large language modelscited by 0
  11. 2026Policy-Masked Private Experts: Auditable and Reversible Capability Access Control in Sparse MoE Modelscited by 0