mlsanityData to 14 Oct 2026

Continual Learning with Differential Privacy

arXiv 2110.05223 ↗First posted 11 Oct 2021cs.LG cs.CRcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we focus on preserving differential privacy (DP) in continual learning (CL), in which we train ML models to learn a sequence of new tasks while memorizing previous tasks. We first introduce a notion of continual adjacent databases to bound the sensitivity of any data record participating in the training process of CL. Based upon that, we develop a new DP-preserving algorithm for CL with a data sampling strategy to quantify the privacy risk of training data in the well-known Averaged Gradient Episodic Memory (A-GEM) approach by applying a moments accountant. Our algorithm provides formal guarantees of privacy for data records across tasks in CL. Preliminary theoretical analysis and evaluations show that our mechanism tightens the privacy loss while maintaining a promising model utility.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 24 of its 44 references that are in the map. First posted on arXiv 11 Oct 2021.

Builds on

24 of 44 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  6. 2016Progressive Neural Networkscited by 1,361
  7. 2017Continual Learning with Deep Generative Replaycited by 1,158
  8. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  9. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  10. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  11. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  12. 2018Progress & Compress: A scalable framework for continual learningcited by 520
12 more
  1. 2020Few-Shot Class-Incremental Learningcited by 251
  2. 2018Task-Free Continual Learningcited by 221
  3. 2019Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learningcited by 182
  4. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  5. 2020iTAML: An Incremental Task-Agnostic Meta-learning Approachcited by 90
  6. 2018Memory Replay GANs: learning to generate images from new categories without forgettingcited by 77
  7. 2017Adaptive Laplace Mechanism: Differential Privacy Preservation in Deep Learningcited by 34
  8. 2019The Value of Collaboration in Convex Machine Learning with Differential Privacycited by 24
  9. 2013Deep learning for neuroimaging: a validation studycited by 21
  10. 2019Heterogeneous Gaussian Mechanism: Preserving Differential Privacy in Deep Learning with Provable Robustnesscited by 15
  11. 2019Differentially Private Continual Learningcited by 3
  12. 2019Scalable Differential Privacy with Certified Robustness in Adversarial Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 0; 2024: 0; 2025: 3; 2026: 1.

6 papers.

  1. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  2. 2025Federated Continual Learning: Concepts, Challenges, and Solutionscited by 13
  3. 2022Lifelong DP: Consistently Bounded Differential Privacy in Lifelong Machine Learningcited by 2
  4. 2025Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learningcited by 1
  5. 2025What is the role of memorization in Continual Learning?cited by 0
  6. 2026Continual Learning for fMRI-Based Brain Disorder Diagnosis via Functional Connectivity Matrices Generative Replaycited by 0