# A distillation-based approach integrating continual learning and federated learning for pervasive services

arXiv: https://arxiv.org/abs/2109.04197 · HTML: https://mlsanity.com/paper/2109.04197/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Federated Learning in Mobile Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/1909.11875/) (2019)
- [Federated Continual Learning with Weighted Inter-client Transfer](https://mlsanity.com/paper/2003.03196/) (2020)
- [Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge based Framework](https://mlsanity.com/paper/2002.10671/) (2020)
- [Continual Local Training for Better Initialization of Federated Models](https://mlsanity.com/paper/2005.12657/) (2020)

## Built on by (most cited first, 30 of 42)

- [Decentralized Federated Learning: A Survey and Perspective](https://mlsanity.com/paper/2306.01603/) (2023)
- [TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation](https://mlsanity.com/paper/2303.06937/) (2023)
- [Better Generative Replay for Continual Federated Learning](https://mlsanity.com/paper/2302.13001/) (2023)
- [Federated Continual Learning via Knowledge Fusion: A Survey](https://mlsanity.com/paper/2312.16475/) (2023)
- [A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Tasks](https://mlsanity.com/paper/2311.07784/) (2023)
- [Learning Federated Visual Prompt in Null Space for MRI Reconstruction](https://mlsanity.com/paper/2303.16181/) (2023)
- [MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare](https://mlsanity.com/paper/2206.08516/) (2022)
- [Non-IID data and Continual Learning processes in Federated Learning: A long road ahead](https://mlsanity.com/paper/2111.13394/) (2021)
- [Graph Lifelong Learning: A Survey](https://mlsanity.com/paper/2202.10688/) (2022)
- [Towards Federated Learning on Time-Evolving Heterogeneous Data](https://mlsanity.com/paper/2112.13246/) (2021)
- [Text-Enhanced Data-free Approach for Federated Class-Incremental Learning](https://mlsanity.com/paper/2403.14101/) (2024)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Federated Continual Learning: Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2502.07059/) (2025)
- [Fed-CPrompt: Contrastive Prompt for Rehearsal-Free Federated Continual Learning](https://mlsanity.com/paper/2307.04869/) (2023)
- [Ex-Model: Continual Learning from a Stream of Trained Models](https://mlsanity.com/paper/2112.06511/) (2021)
- [Federated Continual Learning for Edge-AI: A Comprehensive Survey](https://mlsanity.com/paper/2411.13740/) (2024)
- [Accurate Forgetting for Heterogeneous Federated Continual Learning](https://mlsanity.com/paper/2502.14205/) (2025)
- [Continual Horizontal Federated Learning for Heterogeneous Data](https://mlsanity.com/paper/2203.02108/) (2022)
- [Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients](https://mlsanity.com/paper/2201.10899/) (2022)
- [Evaluation and comparison of federated learning algorithms for Human Activity Recognition on smartphones](https://mlsanity.com/paper/2210.16918/) (2022)
- [Zone-based Federated Learning for Mobile Sensing Data](https://mlsanity.com/paper/2303.06246/) (2023)
- [Don't Memorize; Mimic The Past: Federated Class Incremental Learning Without Episodic Memory](https://mlsanity.com/paper/2307.00497/) (2023)
- [Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator](https://mlsanity.com/paper/2407.11078/) (2024)
- [Evaluation of Regularization-based Continual Learning Approaches: Application to HAR](https://mlsanity.com/paper/2304.13327/) (2023)
- [Coordinated Replay Sample Selection for Continual Federated Learning](https://mlsanity.com/paper/2310.15054/) (2023)
- [Data-Free Federated Class Incremental Learning with Diffusion-Based Generative Memory](https://mlsanity.com/paper/2405.17457/) (2024)
- [Federated Learning and catastrophic forgetting in pervasive computing: demonstration in HAR domain](https://mlsanity.com/paper/2207.08180/) (2022)
- [CoDeC: Communication-Efficient Decentralized Continual Learning](https://mlsanity.com/paper/2303.15378/) (2023)
- [PIP: Prototypes-Injected Prompt for Federated Class Incremental Learning](https://mlsanity.com/paper/2407.20705/) (2024)
- [Decentralized Dynamic Cooperation of Personalized Models for Federated Continual Learning](https://mlsanity.com/paper/2509.23683/) (2025)

## Abstract (arXiv)

Federated Learning, a new machine learning paradigm enhancing the use of edge devices, is receiving a lot of attention in the pervasive community to support the development of smart services. Nevertheless, this approach still needs to be adapted to the specificity of the pervasive domain. In particular, issues related to continual learning need to be addressed. In this paper, we present a distillation-based approach dealing with catastrophic forgetting in federated learning scenario. Specifically, Human Activity Recognition tasks are used as a demonstration domain.