# Joint Superposition Coding and Training for Federated Learning over Multi-Width Neural Networks

arXiv: https://arxiv.org/abs/2112.02543 · HTML: https://mlsanity.com/paper/2112.02543/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 5 Dec 2021.

## Builds on (most cited first, 13 of 13)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [Tighter Theory for Local SGD on Identical and Heterogeneous Data](https://mlsanity.com/paper/1909.04746/) (2019)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [Wireless Network Intelligence at the Edge](https://mlsanity.com/paper/1812.02858/) (2018)
- [Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications](https://mlsanity.com/paper/2008.02608/) (2020)
- [Measuring the Algorithmic Efficiency of Neural Networks](https://mlsanity.com/paper/2005.04305/) (2020)

## Built on by (most cited first, 14 of 14)

- [Slimmable Quantum Federated Learning](https://mlsanity.com/paper/2207.10221/) (2022)
- [Quantum Distributed Deep Learning Architectures: Models, Discussions, and Applications](https://mlsanity.com/paper/2202.11200/) (2022)
- [AnycostFL: Efficient On-Demand Federated Learning over Heterogeneous Edge Devices](https://mlsanity.com/paper/2301.03062/) (2023)
- [SlimFL: Federated Learning with Superposition Coding over Slimmable Neural Networks](https://mlsanity.com/paper/2203.14094/) (2022)
- [Cooperative Multi-Agent Deep Reinforcement Learning for Reliable and Energy-Efficient Mobile Access via Multi-UAV Control](https://mlsanity.com/paper/2210.00945/) (2022)
- [Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning](https://mlsanity.com/paper/2308.11464/) (2023)
- [Quantum Federated Learning with Entanglement Controlled Circuits and Superposition Coding](https://mlsanity.com/paper/2212.01732/) (2022)
- [Two-stage architectural fine-tuning with neural architecture search using early-stopping in image classification](https://mlsanity.com/paper/2202.08604/) (2022)
- [Exploiting Features and Logits in Heterogeneous Federated Learning](https://mlsanity.com/paper/2210.15527/) (2022)
- [Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Network](https://mlsanity.com/paper/2311.10002/) (2023)
- [Parallelized and Randomized Adversarial Imitation Learning for Safety-Critical Self-Driving Vehicles](https://mlsanity.com/paper/2112.14710/) (2021)
- [FedIN: Federated Intermediate Layers Learning for Model Heterogeneity](https://mlsanity.com/paper/2304.00759/) (2023)
- [Search Space Adaptation for Differentiable Neural Architecture Search in Image Classification](https://mlsanity.com/paper/2206.02098/) (2022)
- [Federated Explainable Artificial Intelligence: Roles, Architectures, Evaluation, and Open Challenges](https://mlsanity.com/paper/2607.13045/) (2026)

## Abstract (arXiv)

This paper aims to integrate two synergetic technologies, federated learning (FL) and width-adjustable slimmable neural network (SNN) architectures. FL preserves data privacy by exchanging the locally trained models of mobile devices. By adopting SNNs as local models, FL can flexibly cope with the time-varying energy capacities of mobile devices. Combining FL and SNNs is however non-trivial, particularly under wireless connections with time-varying channel conditions. Furthermore, existing multi-width SNN training algorithms are sensitive to the data distributions across devices, so are ill-suited to FL. Motivated by this, we propose a communication and energy-efficient SNN-based FL (named SlimFL) that jointly utilizes superposition coding (SC) for global model aggregation and superposition training (ST) for updating local models. By applying SC, SlimFL exchanges the superposition of multiple width configurations that are decoded as many as possible for a given communication throughput. Leveraging ST, SlimFL aligns the forward propagation of different width configurations, while avoiding the inter-width interference during backpropagation. We formally prove the convergence of SlimFL. The result reveals that SlimFL is not only communication-efficient but also can counteract non-IID data distributions and poor channel conditions, which is also corroborated by simulations.