# Partial Variable Training for Efficient On-Device Federated Learning

arXiv: https://arxiv.org/abs/2110.05607 · HTML: https://mlsanity.com/paper/2110.05607/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Expanding the Reach of Federated Learning by Reducing Client Resource Requirements](https://mlsanity.com/paper/1812.07210/) (2018)
- [Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning](https://mlsanity.com/paper/1611.05128/) (2016)
- [Are All Layers Created Equal?](https://mlsanity.com/paper/1902.01996/) (2019)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization](https://mlsanity.com/paper/2104.00031/) (2021)
- [Distributed Fixed Point Methods with Compressed Iterates](https://mlsanity.com/paper/1912.09925/) (2019)

## Built on by (most cited first, 14 of 14)

- [FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction](https://mlsanity.com/paper/2212.01548/) (2022)
- [TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Training](https://mlsanity.com/paper/2304.06947/) (2023)
- [Scaling Language Model Size in Cross-Device Federated Learning](https://mlsanity.com/paper/2204.09715/) (2022)
- [A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness](https://mlsanity.com/paper/2212.09006/) (2022)
- [Federated Select: A Primitive for Communication- and Memory-Efficient Federated Learning](https://mlsanity.com/paper/2208.09432/) (2022)
- [CoCoFL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantization](https://mlsanity.com/paper/2203.05468/) (2022)
- [Online Model Compression for Federated Learning with Large Models](https://mlsanity.com/paper/2205.03494/) (2022)
- [Federated Domain Adaptation for ASR with Full Self-Supervision](https://mlsanity.com/paper/2203.15966/) (2022)
- [Decoupled Federated Learning for ASR with Non-IID Data](https://mlsanity.com/paper/2206.09102/) (2022)
- [Towards a Better Theoretical Understanding of Independent Subnetwork Training](https://mlsanity.com/paper/2306.16484/) (2023)
- [Why Go Full? Elevating Federated Learning Through Partial Network Updates](https://mlsanity.com/paper/2410.11559/) (2024)
- [Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beat](https://mlsanity.com/paper/2309.03237/) (2023)
- [Feature-based Federated Transfer Learning: Communication Efficiency, Robustness and Privacy](https://mlsanity.com/paper/2405.09014/) (2024)
- [Federated Learning of Large ASR Models in the Real World](https://mlsanity.com/paper/2408.10443/) (2024)

## Abstract (arXiv)

This paper aims to address the major challenges of Federated Learning (FL) on edge devices: limited memory and expensive communication. We propose a novel method, called Partial Variable Training (PVT), that only trains a small subset of variables on edge devices to reduce memory usage and communication cost. With PVT, we show that network accuracy can be maintained by utilizing more local training steps and devices, which is favorable for FL involving a large population of devices. According to our experiments on two state-of-the-art neural networks for speech recognition and two different datasets, PVT can reduce memory usage by up to 1.9× and communication cost by up to 593× while attaining comparable accuracy when compared with full network training.