# A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization

arXiv: https://arxiv.org/abs/2609.25471 · HTML: https://mlsanity.com/paper/2609.25471/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)
- [FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](https://mlsanity.com/paper/2205.07246/) (2022)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!](https://mlsanity.com/paper/2202.09357/) (2022)
- [Iterative Pseudo-Labeling for Speech Recognition](https://mlsanity.com/paper/2005.09267/) (2020)
- [SlimIPL: Language-Model-Free Iterative Pseudo-Labeling](https://mlsanity.com/paper/2010.11524/) (2020)
- [Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework](https://mlsanity.com/paper/2010.15965/) (2020)
- [SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training](https://mlsanity.com/paper/2106.01432/) (2021)
- [Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition](https://mlsanity.com/paper/2106.08922/) (2021)
- [Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognition](https://mlsanity.com/paper/2406.02925/) (2024)
- [Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-End ASR](https://mlsanity.com/paper/2309.13102/) (2023)

## Abstract (arXiv)

Semi-supervised federated learning (SSFL) trains models on clients' unlabeled data using a teacher to generate pseudo-labels, with a small labeled seed dataset on the server. Automatic Speech Recognition (ASR) is particularly fragile here: pseudo-label errors compound across the output sequence and across training rounds into divergence, leaving a large gap to fully-supervised FL. We show that closing this gap turns on two coupled design axes -- the teacher (which model generates the pseudo-labels) and the anchor (the server-side updates on labeled data that stabilize training). On the teacher axis, a per-client online teacher (each client's own evolving model) diverges on its own, but once stabilized it matches or beats the broadcast global teacher (one server model, fixed within a round) -- decisively in-domain and competitively under domain shift. As the seed grows stronger and the online teacher's advantage narrows, a transitioning teacher (global rightarrow online at round r) matches or beats both. On the anchor axis, the server must keep training on labeled data between rounds -- otherwise the online teacher drifts -- and this interleaving, more than the seed model, governs convergence. The two axes are inseparable: aggressive teacher choices pay off only once the anchor stabilizes training, which is highly sensitive to data augmentation and batch size -- the settings that govern how much input and gradient noise the server injects. How much stabilization is needed is domain-dependent, governed by the dispersion of the seed data and its overlap with client data. These findings yield guidelines for SSFL in ASR training, improving over the strongest prior method on 9 of 11 pairs, by 20.8% on average in-domain and 10.0% cross-domain, narrowing the gap to fully-supervised FL.