# End-to-End Speech Recognition and Disfluency Removal with Acoustic Language Model Pretraining

arXiv: https://arxiv.org/abs/2309.04516 · HTML: https://mlsanity.com/paper/2309.04516/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context](https://mlsanity.com/paper/2005.03191/) (2020)
- [Improving Disfluency Detection by Self-Training a Self-Attentive Model](https://mlsanity.com/paper/2004.05323/) (2020)
- [Multi-Task Self-Supervised Learning for Disfluency Detection](https://mlsanity.com/paper/1908.05378/) (2019)
- [Disfluency Detection with Unlabeled Data and Small BERT Models](https://mlsanity.com/paper/2104.10769/) (2021)
- [End-to-End Speech Recognition and Disfluency Removal](https://mlsanity.com/paper/2009.10298/) (2020)
- [Giving Attention to the Unexpected: Using Prosody Innovations in Disfluency Detection](https://mlsanity.com/paper/1904.04388/) (2019)
- [Neural Constituency Parsing of Speech Transcripts](https://mlsanity.com/paper/1904.08535/) (2019)
- [Auxiliary Sequence Labeling Tasks for Disfluency Detection](https://mlsanity.com/paper/2011.04512/) (2020)

## Abstract (arXiv)

The SOTA in transcription of disfluent and conversational speech has in recent years favored two-stage models, with separate transcription and cleaning stages. We believe that previous attempts at end-to-end disfluency removal have fallen short because of the representational advantage that large-scale language model pretraining has given to lexical models. Until recently, the high dimensionality and limited availability of large audio datasets inhibited the development of large-scale self-supervised pretraining objectives for learning effective audio representations, giving a relative advantage to the two-stage approach, which utilises pretrained representations for lexical tokens. In light of recent successes in large scale audio pretraining, we revisit the performance comparison between two-stage and end-to-end model and find that audio based language models pretrained using weak self-supervised objectives match or exceed the performance of similarly trained two-stage models, and further, that the choice of pretraining objective substantially effects a model's ability to be adapted to the disfluency removal task.