# Heterogeneous Decentralized Machine Unlearning with Seed Model Distillation

arXiv: https://arxiv.org/abs/2308.13269 · HTML: https://mlsanity.com/paper/2308.13269/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 28 of its 63 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [A Unified Theory of Decentralized SGD with Changing Topology and Local Updates](https://mlsanity.com/paper/2003.10422/) (2020)
- [Analyzing Federated Learning through an Adversarial Lens](https://mlsanity.com/paper/1811.12470/) (2018)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Unrolling SGD: Understanding Factors Influencing Machine Unlearning](https://mlsanity.com/paper/2109.13398/) (2021)
- [Variational Bayesian Unlearning](https://mlsanity.com/paper/2010.12883/) (2020)
- [Federated Unlearning with Knowledge Distillation](https://mlsanity.com/paper/2201.09441/) (2022)
- [Concentrated Differentially Private Gradient Descent with Adaptive per-Iteration Privacy Budget](https://mlsanity.com/paper/1808.09501/) (2018)
- [Semi-decentralized Federated Ego Graph Learning for Recommendation](https://mlsanity.com/paper/2302.10900/) (2023)
- [Learning Elastic Embeddings for Customizing On-Device Recommenders](https://mlsanity.com/paper/2106.02223/) (2021)
- [ReFRS: Resource-efficient Federated Recommender System for Dynamic and Diversified User Preferences](https://mlsanity.com/paper/2207.13897/) (2022)
- [Decentralized Collaborative Learning Framework for Next POI Recommendation](https://mlsanity.com/paper/2204.06516/) (2022)
- [Model-Agnostic Decentralized Collaborative Learning for On-Device POI Recommendation](https://mlsanity.com/paper/2304.03947/) (2023)
- [Decomposable Submodular Function Minimization: Discrete and Continuous](https://mlsanity.com/paper/1703.01830/) (2017)
- [AI-based Fog and Edge Computing: A Systematic Review, Taxonomy and Future Directions](https://mlsanity.com/paper/2212.04645/) (2022)
- [Heterogeneous Collaborative Learning for Personalized Healthcare Analytics via Messenger Distillation](https://mlsanity.com/paper/2205.13705/) (2022)
- [Personalized On-Device E-health Analytics with Decentralized Block Coordinate Descent](https://mlsanity.com/paper/2112.09341/) (2021)

## Built on by (most cited first, 4 of 4)

- [A Survey on Federated Unlearning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/2310.20448/) (2023)
- [Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metrics](https://mlsanity.com/paper/2401.05146/) (2024)
- [A Survey on Federated Unlearning: Challenges and Opportunities](https://mlsanity.com/paper/2403.02437/) (2024)
- [Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning](https://mlsanity.com/paper/2606.22878/) (2026)

## Abstract (arXiv)

As some recent information security legislation endowed users with unconditional rights to be forgotten by any trained machine learning model, personalized IoT service providers have to put unlearning functionality into their consideration. The most straightforward method to unlearn users' contribution is to retrain the model from the initial state, which is not realistic in high throughput applications with frequent unlearning requests. Though some machine unlearning frameworks have been proposed to speed up the retraining process, they fail to match decentralized learning scenarios. In this paper, we design a decentralized unlearning framework called HDUS, which uses distilled seed models to construct erasable ensembles for all clients. Moreover, the framework is compatible with heterogeneous on-device models, representing stronger scalability in real-world applications. Extensive experiments on three real-world datasets show that our HDUS achieves state-of-the-art performance.