Cocoon: A System Architecture for Differentially Private Training with Correlated Noises
Not on the map. Why?
Abstract (arXiv)
Machine learning (ML) models memorize and leak training data, causing serious privacy issues to data owners. Training algorithms with differential privacy (DP) have been gaining attention as a solution. However, these algorithms add noise at each training iteration and degrade accuracy, limiting their real-world adoption. To improve accuracy, a new family of approaches adds carefully designed correlated noises, so that noises cancel out each other across iterations. We performed an extensive characterization study of these new mechanisms and show they incur non-negligible overheads when the model is relatively large or uses large embedding tables compared to the hardware capacity. Motivated by the analysis, we propose Cocoon, a framework for efficient training with correlated noises. Cocoon stores and processes the large noise history across CPU, GPU, and memory extension module, introduces optimizations for sparse embedding tables, and leverages to-be-commercialized near-memory processing (NMP) devices. On a real system with an FPGA-based NMP device prototype, Cocoon improves the performance by 1.23-10.82x.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 78 of its 114 references that are in the map. First posted on arXiv 8 Oct 2025.
Builds on
78 of 114 references.
References · 78
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2016Deep Learning with Differential Privacycited by 2,728
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
- 2023Extracting Training Data from Diffusion Modelscited by 732
- 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
- 2021Differentially Private Fine-tuning of Language Modelscited by 400
- 2020Label-Only Membership Inference Attackscited by 388
- 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
- 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
38 more
- 2013The Composition Theorem for Differential Privacycited by 297
- 2014RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Responsecited by 272
- 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
- 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
- 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
- 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
- 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 199
- 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
- 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 147
- 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 122
- 2018Privacy Amplification by Iterationcited by 100
- 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
- 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 88
- 2023Federated Learning of Gboard Language Models with Differential Privacycited by 66
- 2022Differentially Private Optimization on Large Model at Small Costcited by 65
- 2020DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inferencecited by 54
- 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
- 2019RecNMP: Accelerating Personalized Recommendation with Near-Memory Processingcited by 49
- 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
- 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 48
- 2022Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learningcited by 48
- 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 47
- 2022Bounding Training Data Reconstruction in Private (Deep) Learningcited by 46
- 2023Bounding Training Data Reconstruction in DP-SGDcited by 46
- 2017Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12cited by 43
- 2019TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learningcited by 43
- 2024Privacy-Preserving Retrieval-Augmented Generation with Differential Privacycited by 39
- 2021RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inferencecited by 35
- 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
- 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 34
- 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
- 2022Cocktail Party Attack: Breaking Aggregation-Based Privacy in Federated Learning using Independent Component Analysiscited by 29
- 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 29
- 2023DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)cited by 28
- 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
- 2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 24
- 2023Correlated Noise Provably Beats Independent Noise for Differentially Private Learningcited by 23
- 2020The Trade-Offs of Private Predictioncited by 22
50 most cited of 78 All 78 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.