# Federated Learning of Large ASR Models in the Real World

arXiv: https://arxiv.org/abs/2408.10443 · HTML: https://mlsanity.com/paper/2408.10443/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Builds on (most cited first, 15 of 15)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework](https://mlsanity.com/paper/2010.15965/) (2020)
- [Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entities](https://mlsanity.com/paper/1912.09251/) (2019)
- [4-bit Conformer with Native Quantization Aware Training for Speech Recognition](https://mlsanity.com/paper/2203.15952/) (2022)
- [Partial Variable Training for Efficient On-Device Federated Learning](https://mlsanity.com/paper/2110.05607/) (2021)
- [Enabling On-Device Training of Speech Recognition Models with Federated Dropout](https://mlsanity.com/paper/2110.03634/) (2021)
- [A Unified Cascaded Encoder ASR Model for Dynamic Model Sizes](https://mlsanity.com/paper/2204.06164/) (2022)
- [Federated Pruning: Improving Neural Network Efficiency with Federated Learning](https://mlsanity.com/paper/2209.06359/) (2022)
- [Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shift](https://mlsanity.com/paper/2111.12150/) (2021)
- [Online Model Compression for Federated Learning with Large Models](https://mlsanity.com/paper/2205.03494/) (2022)
- [Sub-8-bit quantization for on-device speech recognition: a regularization-free approach](https://mlsanity.com/paper/2210.09188/) (2022)

## Abstract (arXiv)

Federated learning (FL) has shown promising results on training machine learning models with privacy preservation. However, for large models with over 100 million parameters, the training resource requirement becomes an obstacle for FL because common devices do not have enough memory and computation power to finish the FL tasks. Although efficient training methods have been proposed, it is still a challenge to train the large models like Conformer based ASR. This paper presents a systematic solution to train the full-size ASR models of 130M parameters with FL. To our knowledge, this is the first real-world FL application of the Conformer model, which is also the largest model ever trained with FL so far. And this is the first paper showing FL can improve the ASR model quality with a set of proposed methods to refine the quality of data and labels of clients. We demonstrate both the training efficiency and the model quality improvement in real-world experiments.