# Self-Train Before You Transcribe

arXiv: https://arxiv.org/abs/2406.12937 · HTML: https://mlsanity.com/paper/2406.12937/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-rewarding alignment and self-training](https://mlsanity.com/synth/self-rewarding-alignment-and-self-training/)

## Builds on (most cited first, 22 of 22)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models](https://mlsanity.com/paper/1702.03275/) (2017)
- [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://mlsanity.com/paper/2305.05084/) (2023)
- [Rethinking Evaluation in ASR: Are Our Models Robust Enough?](https://mlsanity.com/paper/2010.11745/) (2020)
- [Dynamic Evaluation of Neural Sequence Models](https://mlsanity.com/paper/1709.07432/) (2017)
- [Earnings-22: A Practical Benchmark for Accents in the Wild](https://mlsanity.com/paper/2203.15591/) (2022)
- [Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition](https://mlsanity.com/paper/2106.08922/) (2021)
- [Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training](https://mlsanity.com/paper/2011.13439/) (2020)
- [Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization](https://mlsanity.com/paper/2101.11075/) (2021)
- [Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2001.01798/) (2020)
- [Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognition](https://mlsanity.com/paper/2203.14222/) (2022)
- [ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition](https://mlsanity.com/paper/2210.13352/) (2022)
- [Advancing Momentum Pseudo-Labeling with Conformer and Initialization Strategy](https://mlsanity.com/paper/2110.04948/) (2021)
- [How Much Context Does My Attention-Based ASR System Need?](https://mlsanity.com/paper/2310.15672/) (2023)

## Abstract (arXiv)

When there is a mismatch between the training and test domains, current speech recognition systems show significant performance degradation. Self-training methods, such as noisy student teacher training, can help address this and enable the adaptation of models under such domain shifts. However, self-training typically requires a collection of unlabelled target domain data. For settings where this is not practical, we investigate the benefit of performing noisy student teacher training on recordings in the test set as a test-time adaptation approach. Similarly to the dynamic evaluation approach in language modelling, this enables the transfer of information across utterance boundaries and functions as a method of domain adaptation. A range of in-domain and out-of-domain datasets are used for experiments demonstrating large relative gains of up to 32.2%. Interestingly, our method showed larger gains than the typical self-training setup that utilises separate adaptation data.