# Continual Learning with Distributed Optimization: Does CoCoA Forget?

arXiv: https://arxiv.org/abs/2211.16994 · HTML: https://mlsanity.com/paper/2211.16994/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 16 of 16)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix](https://mlsanity.com/paper/2010.04003/) (2020)
- [How catastrophic can catastrophic forgetting be in linear regression?](https://mlsanity.com/paper/2205.09588/) (2022)
- [CoCoA: A General Framework for Communication-Efficient Distributed Optimization](https://mlsanity.com/paper/1611.02189/) (2016)
- [Communication-Efficient Distributed Dual Coordinate Ascent](https://mlsanity.com/paper/1409.1458/) (2014)
- [COLA: Decentralized Linear Learning](https://mlsanity.com/paper/1808.04883/) (2018)
- [Distributed Optimization with Arbitrary Local Solvers](https://mlsanity.com/paper/1512.04039/) (2015)
- [Multitask Diffusion Adaptation over Networks](https://mlsanity.com/paper/1311.4894/) (2013)
- [Distributed Online Linear Regression](https://mlsanity.com/paper/1902.04774/) (2019)
- [Dictionary Learning over Distributed Models](https://mlsanity.com/paper/1402.1515/) (2014)
- [Diffusion LMS over Multitask Networks](https://mlsanity.com/paper/1404.6813/) (2014)
- [Linear Regression with Distributed Learning: A Generalization Error Perspective](https://mlsanity.com/paper/2101.09001/) (2021)
- [Diffusion LMS with Communication Delays: Stability and Performance Analysis](https://mlsanity.com/paper/2004.08881/) (2020)
- [Efficient model selection in switching linear dynamic systems by graph clustering](https://mlsanity.com/paper/2012.04543/) (2020)

## Built on by (most cited first, 1 of 1)

- [Distributed Continual Learning with CoCoA in High-dimensional Linear Regression](https://mlsanity.com/paper/2312.01795/) (2023)

## Abstract (arXiv)

We focus on the continual learning problem where the tasks arrive sequentially and the aim is to perform well on the newly arrived task without performance degradation on the previously seen tasks. In contrast to the continual learning literature focusing on the centralized setting, we investigate the distributed estimation framework. We consider the well-established distributed learning algorithm COCOA. We derive closed form expressions for the iterations for the overparametrized case. We illustrate the convergence and the error performance of the algorithm based on the over/under-parameterization of the problem. Our results show that depending on the problem dimensions and data generation assumptions, COCOA can perform continual learning over a sequence of tasks, i.e., it can learn a new task without forgetting previously learned tasks, with access only to one task at a time.