# O-1: Self-training with Oracle and 1-best Hypothesis

arXiv: https://arxiv.org/abs/2308.07486 · HTML: https://mlsanity.com/paper/2308.07486/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 30 of 30)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [An Actor-Critic Algorithm for Sequence Prediction](https://mlsanity.com/paper/1607.07086/) (2016)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [Sequence-to-Sequence Learning as Beam-Search Optimization](https://mlsanity.com/paper/1606.02960/) (2016)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network](https://mlsanity.com/paper/2104.02133/) (2021)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01818/) (2017)
- [Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation](https://mlsanity.com/paper/2105.08504/) (2021)
- [Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?](https://mlsanity.com/paper/1905.10617/) (2019)
- [Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognition](https://mlsanity.com/paper/2007.13802/) (2020)
- [Optimizing expected word error rate via sampling for speech recognition](https://mlsanity.com/paper/1706.02776/) (2017)
- [Optimal Completion Distillation for Sequence Learning](https://mlsanity.com/paper/1810.01398/) (2018)
- [A Fully Differentiable Beam Search Decoder](https://mlsanity.com/paper/1902.06022/) (2019)
- [Improving Rare Word Recognition with LM-aware MWER Training](https://mlsanity.com/paper/2204.07553/) (2022)
- [On Minimum Word Error Rate Training of the Hybrid Autoregressive Transducer](https://mlsanity.com/paper/2010.12673/) (2020)
- [Input Length Matters: Improving RNN-T and MWER Training for Long-form Telephony Speech Recognition](https://mlsanity.com/paper/2110.03841/) (2021)
- [Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR](https://mlsanity.com/paper/2210.05793/) (2022)
- [Promising Accurate Prefix Boosting for sequence-to-sequence ASR](https://mlsanity.com/paper/1811.02770/) (2018)
- [Ask2Mask: Guided Data Selection for Masked Speech Modeling](https://mlsanity.com/paper/2202.12719/) (2022)
- [Reducing Exposure Bias in Training Recurrent Neural Network Transducers](https://mlsanity.com/paper/2108.10803/) (2021)
- [Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss](https://mlsanity.com/paper/2303.05958/) (2023)

## Abstract (arXiv)

We introduce O-1, a new self-training objective to reduce training bias and unify training and evaluation metrics for speech recognition. O-1 is a faster variant of Expected Minimum Bayes Risk (EMBR), that boosts the oracle hypothesis and can accommodate both supervised and unsupervised data. We demonstrate the effectiveness of our approach in terms of recognition on publicly available SpeechStew datasets and a large-scale, in-house data set. On Speechstew, the O-1 objective closes the gap between the actual and oracle performance by 80% relative compared to EMBR which bridges the gap by 43% relative. O-1 achieves 13% to 25% relative improvement over EMBR on the various datasets that SpeechStew comprises of, and a 12% relative gap reduction with respect to the oracle WER over EMBR training on the in-house dataset. Overall, O-1 results in a 9% relative improvement in WER over EMBR, thereby speaking to the scalability of the proposed objective for large-scale datasets.