Learning to Jointly Transcribe and Subtitle for End-to-End Spontaneous Speech Recognition
Not on the map. Why?
Abstract (arXiv)
TV subtitles are a rich source of transcriptions of many types of speech, ranging from read speech in news reports to conversational and spontaneous speech in talk shows and soaps. However, subtitles are not verbatim (i.e. exact) transcriptions of speech, so they cannot be used directly to improve an Automatic Speech Recognition (ASR) model. We propose a multitask dual-decoder Transformer model that jointly performs ASR and automatic subtitling. The ASR decoder (possibly pre-trained) predicts the verbatim output and the subtitle decoder generates a subtitle, while sharing the encoder. The two decoders can be independent or connected. The model is trained to perform both tasks jointly, and is able to effectively use subtitle data. We show improvements on regular ASR and on spontaneous and conversational ASR by incorporating the additional subtitle decoder. The method does not require preprocessing (aligning, filtering, pseudo-labeling, ...) of the subtitles.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 19 of its 37 references that are in the map. First posted on arXiv 14 Oct 2022.
Builds on
19 of 37 references.
References · 19
- 2017Attention Is All You Needcited by 60,361
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
- 2018Self-Attention with Relative Position Representationscited by 1,120
- 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
- 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
- 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
- 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
- 2018Tied Multitask Learning for Neural Speech Translationcited by 54
7 more
- 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
- 2019Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decodingcited by 21
- 2020Adapting End-to-End Speech Recognition for Readable Subtitlescited by 10
- 2021Construction of a Large-scale Japanese ASR Corpus on TV Recordingscited by 8
- 2021Comparison of Self-Supervised Speech Pre-Training Methods on Flemish Dutchcited by 2
- 2015The 2015 Sheffield System for Transcription of Multi-Genre Broadcast Mediacited by 1
- 2019Training ASR models by Generation of Contextual Informationcited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.