# Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point

arXiv: https://arxiv.org/abs/2407.02610 · HTML: https://mlsanity.com/paper/2407.02610/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [On Biased Compression for Distributed Learning](https://mlsanity.com/paper/2002.12410/) (2020)
- [EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback](https://mlsanity.com/paper/2106.05203/) (2021)
- [Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be](https://mlsanity.com/paper/2304.13960/) (2023)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [FP8 Quantization: The Power of the Exponent](https://mlsanity.com/paper/2208.09225/) (2022)
- [FP8-LM: Training FP8 Large Language Models](https://mlsanity.com/paper/2310.18313/) (2023)
- [Training Quantized Nets: A Deeper Understanding](https://mlsanity.com/paper/1706.02379/) (2017)
- [Keyword Transformer: A Self-Attention Model for Keyword Spotting](https://mlsanity.com/paper/2104.00769/) (2021)
- [Design and Analysis of Uplink and Downlink Communications for Federated Learning](https://mlsanity.com/paper/2012.04057/) (2020)
- [DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning](https://mlsanity.com/paper/2111.00465/) (2021)
- [MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition](https://mlsanity.com/paper/2004.08531/) (2020)
- [Efficient Post-training Quantization with FP8 Formats](https://mlsanity.com/paper/2309.14592/) (2023)
- [Randomized Quantization is All You Need for Differential Privacy in Federated Learning](https://mlsanity.com/paper/2306.11913/) (2023)
- [Wireless Quantized Federated Learning: A Joint Computation and Communication Design](https://mlsanity.com/paper/2203.05878/) (2022)
- [Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devices](https://mlsanity.com/paper/2206.10844/) (2022)
- [Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantization](https://mlsanity.com/paper/2202.11453/) (2022)
- [QReg: On Regularization Effects of Quantization](https://mlsanity.com/paper/2206.12372/) (2022)
- [CosSGD: Communication-Efficient Federated Learning with a Simple Cosine-Based Quantization](https://mlsanity.com/paper/2012.08241/) (2020)
- [Avoid Overfitting User Specific Information in Federated Keyword Spotting](https://mlsanity.com/paper/2206.08864/) (2022)

## Built on by (most cited first, 1 of 1)

- [On-Device Training of PV Power Forecasting Models in a Smart Meter for Grid Edge Intelligence](https://mlsanity.com/paper/2507.07016/) (2025)

## Abstract (arXiv)

Recent work has shown that 8-bit floating point (FP8) can be used for efficiently training neural networks with reduced computational cost compared to training in FP32/FP16. In this work, we investigate the use of FP8 training in a federated learning context. This approach brings not only the usual benefits of FP8 which are desirable for on-device training at the edge, but also reduces client-server communication costs due to significant weight compression. We present a novel method for combining FP8 client training while maintaining a global FP32 server model and provide convergence analysis. Experiments with various machine learning models and datasets show that our method consistently yields communication reductions of at least 2.9x across a variety of tasks and models compared to an FP32 baseline to achieve the same trained model accuracy.