mlsanityData to 7 Oct 2026

Self-Train Before You Transcribe

arXiv 2406.12937 ↗First posted 17 Jun 2024eess.AS cs.CL cs.LGcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

When there is a mismatch between the training and test domains, current speech recognition systems show significant performance degradation. Self-training methods, such as noisy student teacher training, can help address this and enable the adaptation of models under such domain shifts. However, self-training typically requires a collection of unlabelled target domain data. For settings where this is not practical, we investigate the benefit of performing noisy student teacher training on recordings in the test set as a test-time adaptation approach. Similarly to the dynamic evaluation approach in language modelling, this enables the transfer of information across utterance boundaries and functions as a method of domain adaptation. A range of in-domain and out-of-domain datasets are used for experiments demonstrating large relative gains of up to 32.2%. Interestingly, our method showed larger gains than the typical self-training setup that utilises separate adaptation data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 22 of its 31 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

22 of 31 references.

References · 22

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  7. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  8. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  9. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  10. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 127
  11. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 120
  12. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 72
10 more
  1. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  2. 2022Earnings-22: A Practical Benchmark for Accents in the Wildcited by 46
  3. 2020Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Trainingcited by 38
  4. 2021Momentum Pseudo-Labeling for Semi-Supervised Speech Recognitioncited by 37
  5. 2021Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimizationcited by 27
  6. 2020Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognitioncited by 21
  7. 2022Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognitioncited by 19
  8. 2022ESB: A Benchmark For Multi-Domain End-to-End Speech Recognitioncited by 18
  9. 2021Advancing Momentum Pseudo-Labeling with Conformer and Initialization Strategycited by 7
  10. 2023How Much Context Does My Attention-Based ASR System Need?cited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Customizing Speech Recognition Model with Large Language Model Feedback
  2. 2023Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Models
  3. 2025Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection
  4. 2026Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards
  5. 2022UL2: Unifying Language Learning Paradigms
  6. 2022Avoid Overthinking in Self-Supervised Models for Speech Recognition
  7. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Models
  8. 2026Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding ↗