mlsanityData to 7 Oct 2026

Distributed Continual Learning with CoCoA in High-dimensional Linear Regression

arXiv 2312.01795 ↗First posted 4 Dec 2023cs.LG eess.SPcited by 0

Not on the map. Why?

Abstract (arXiv)

We consider estimation under scenarios where the signals of interest exhibit change of characteristics over time. In particular, we consider the continual learning problem where different tasks, e.g., data with different distributions, arrive sequentially and the aim is to perform well on the newly arrived task without performance degradation on the previously seen tasks. In contrast to the continual learning literature focusing on the centralized setting, we investigate the problem from a distributed estimation perspective. We consider the well-established distributed learning algorithm COCOA, which distributes the model parameters and the corresponding features over the network. We provide exact analytical characterization for the generalization error of COCOA under continual learning for linear regression in a range of scenarios, where overparameterization is of particular interest. These analytical results characterize how the generalization error depends on the network structure, the task similarity and the number of tasks, and show how these dependencies are intertwined. In particular, our results show that the generalization error can be significantly reduced by adjusting the network size, where the most favorable network size depends on task similarity and the number of tasks. We present numerical results verifying the theoretical analysis and illustrate the continual learning performance of COCOA with a digit classification task.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 66 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

42 of 66 references.

References · 42

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  4. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  5. 2019Three scenarios for continual learningcited by 610
  6. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  7. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  8. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  9. 2020Federated Continual Learning with Weighted Inter-client Transfercited by 158
  10. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  11. 2018Towards Robust Evaluations of Continual Learningcited by 148
  12. 2019Federated Learning for Wireless Communications: Motivation, Opportunities and Challengescited by 128
30 more
  1. 2022Federated Class-Incremental Learningcited by 126
  2. 2019Harmless interpolation of noisy data in regressioncited by 121
  3. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  4. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  5. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  6. 2016CoCoA: A General Framework for Communication-Efficient Distributed Optimizationcited by 69
  7. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  8. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  9. 2019Toward Understanding Catastrophic Forgetting in Continual Learningcited by 67
  10. 2014Communication-Efficient Distributed Dual Coordinate Ascentcited by 30
  11. 2018COLA: Decentralized Linear Learningcited by 30
  12. 2023Asynchronous Federated Continual Learningcited by 29
  13. 2020Multitask learning over graphs: An Approach for Distributed, Streaming Machine Learningcited by 26
  14. 2015Distributed Optimization with Arbitrary Local Solverscited by 23
  15. 2019One-Shot Federated Learning: Theoretical Limits and Algorithms to Achieve Themcited by 23
  16. 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
  17. 2021Learning to Continuously Optimize Wireless Resource in a Dynamic Environment: A Bilevel Optimization Perspectivecited by 12
  18. 2013Multitask Diffusion Adaptation over Networkscited by 11
  19. 2021Local SGD Optimizes Overparameterized Neural Networks in Polynomial Timecited by 10
  20. 2014Dictionary Learning over Distributed Modelscited by 8
  21. 2019Privacy-preserving Distributed Machine Learning via Local Randomization and ADMM Perturbationcited by 8
  22. 2015DUAL-LOCO: Distributing Statistical Estimation Using Random Projectionscited by 7
  23. 2019Order Optimal One-Shot Distributed Learningcited by 4
  24. 2021Linear Regression with Distributed Learning: A Generalization Error Perspectivecited by 4
  25. 2014LOCO: Distributing Ridge Regression with Random Projectionscited by 3
  26. 2019Distributed Optimization for Over-Parameterized Learningcited by 2
  27. 2020Diffusion LMS with Communication Delays: Stability and Performance Analysiscited by 2
  28. 2020Efficient model selection in switching linear dynamic systems by graph clusteringcited by 2
  29. 2022Estimation under Model Misspecification with Fake Featurescited by 2
  30. 2022Continual Learning with Distributed Optimization: Does CoCoA Forget?cited by 1

Built on by

Not cited yet.