Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point
Not on the map. Why?
Abstract (arXiv)
Recent work has shown that 8-bit floating point (FP8) can be used for efficiently training neural networks with reduced computational cost compared to training in FP32/FP16. In this work, we investigate the use of FP8 training in a federated learning context. This approach brings not only the usual benefits of FP8 which are desirable for on-device training at the edge, but also reduces client-server communication costs due to significant weight compression. We present a novel method for combining FP8 client training while maintaining a global FP32 server model and provide convergence analysis. Experiments with various machine learning models and datasets show that our method consistently yields communication reductions of at least 2.9x across a variety of tasks and models compared to an FP32 baseline to achieve the same trained model accuracy.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 2 Jul 2024.
Builds on
29 of 39 references.
References · 29
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2018Group Normalizationcited by 1,514
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
- 2018Federated Learning for Mobile Keyboard Predictioncited by 565
- 2021Federated Learning Based on Dynamic Regularizationcited by 555
- 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
- 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
- 2020On Biased Compression for Distributed Learningcited by 140
- 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 131
17 more
- 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
- 2018Federated Learning for Keyword Spottingcited by 95
- 2022FP8 Quantization: The Power of the Exponentcited by 87
- 2023FP8-LM: Training FP8 Large Language Modelscited by 75
- 2017Training Quantized Nets: A Deeper Understandingcited by 74
- 2021Keyword Transformer: A Self-Attention Model for Keyword Spottingcited by 59
- 2020Design and Analysis of Uplink and Downlink Communications for Federated Learningcited by 48
- 2021DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learningcited by 40
- 2020MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognitioncited by 36
- 2023Efficient Post-training Quantization with FP8 Formatscited by 29
- 2023Randomized Quantization is All You Need for Differential Privacy in Federated Learningcited by 16
- 2022Wireless Quantized Federated Learning: A Joint Computation and Communication Designcited by 15
- 2022Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devicescited by 12
- 2022Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantizationcited by 11
- 2022QReg: On Regularization Effects of Quantizationcited by 9
- 2020CosSGD: Communication-Efficient Federated Learning with a Simple Cosine-Based Quantizationcited by 6
- 2022Avoid Overfitting User Specific Information in Federated Keyword Spottingcited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.