# Federated Learning for ASR based on Wav2vec 2.0

arXiv: https://arxiv.org/abs/2302.10790 · HTML: https://mlsanity.com/paper/2302.10790/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 20 Feb 2023.

## Builds on (most cited first, 26 of 26)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Fair Resource Allocation in Federated Learning](https://mlsanity.com/paper/1905.10497/) (2019)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [Introducing the VoicePrivacy Initiative](https://mlsanity.com/paper/2005.01387/) (2020)
- [LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech](https://mlsanity.com/paper/2104.11462/) (2021)
- [The VoicePrivacy 2020 Challenge: Results and findings](https://mlsanity.com/paper/2109.00648/) (2021)
- [Cross-Silo Federated Learning: Challenges and Opportunities](https://mlsanity.com/paper/2206.12949/) (2022)
- [Training Keyword Spotting Models on Non-IID Data with Federated Learning](https://mlsanity.com/paper/2005.10406/) (2020)
- [Improving on-device speaker verification using federated learning with privacy](https://mlsanity.com/paper/2008.02651/) (2020)
- [End-to-End Speech Recognition from Federated Acoustic Models](https://mlsanity.com/paper/2104.14297/) (2021)
- [Federated Learning in ASR: Not as Easy as You Think](https://mlsanity.com/paper/2109.15108/) (2021)
- [Privacy attacks for automatic speech recognition acoustic models in a federated learning framework](https://mlsanity.com/paper/2111.03777/) (2021)
- [Federated Self-supervised Speech Representations: Are We There Yet?](https://mlsanity.com/paper/2204.02804/) (2022)
- [Production federated keyword spotting via distillation, filtering, and joint federated-centralized training](https://mlsanity.com/paper/2204.06322/) (2022)
- [Federated Domain Adaptation for ASR with Full Self-Supervision](https://mlsanity.com/paper/2203.15966/) (2022)
- [Decoupled Federated Learning for ASR with Non-IID Data](https://mlsanity.com/paper/2206.09102/) (2022)
- [FedNST: Federated Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2206.02797/) (2022)
- [Retrieving Speaker Information from Personalized Acoustic Models for Speech Recognition](https://mlsanity.com/paper/2111.04194/) (2021)

## Built on by (most cited first, 7 of 7)

- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks](https://mlsanity.com/paper/2401.10070/) (2024)
- [Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clients](https://mlsanity.com/paper/2405.17376/) (2024)
- [SVSNet+: Enhancing Speaker Voice Similarity Assessment Models with Representations from Speech Foundation Models](https://mlsanity.com/paper/2406.08445/) (2024)
- [Sequence-to-sequence models in peer-to-peer learning: A practical application](https://mlsanity.com/paper/2406.02565/) (2024)
- [SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies](https://mlsanity.com/paper/2607.25716/) (2026)
- [Federated Multilingual Speech-LLMs: Architecture and Aggregation Strategy Benchmarking](https://mlsanity.com/paper/2609.23825/) (2026)

## Abstract (arXiv)

This paper presents a study on the use of federated learning to train an ASR model based on a wav2vec 2.0 model pre-trained by self supervision. Carried out on the well-known TED-LIUM 3 dataset, our experiments show that such a model can obtain, with no use of a language model, a word error rate of 10.92% on the official TED-LIUM 3 test set, without sharing any data from the different users. We also analyse the ASR performance for speakers depending to their participation to the federated learning. Since federated learning was first introduced for privacy purposes, we also measure its ability to protect speaker identity. To do that, we exploit an approach to analyze information contained in exchanged models based on a neural network footprint on an indicator dataset. This analysis is made layer-wise and shows which layers in an exchanged wav2vec 2.0 based model bring the speaker identity information.