# Learning to Jointly Transcribe and Subtitle for End-to-End Spontaneous Speech Recognition

arXiv: https://arxiv.org/abs/2210.07771 · HTML: https://mlsanity.com/paper/2210.07771/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Self-training and Pre-training are Complementary for Speech Recognition](https://mlsanity.com/paper/2010.11430/) (2020)
- [Tied Multitask Learning for Neural Speech Translation](https://mlsanity.com/paper/1802.06655/) (2018)
- [Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation](https://mlsanity.com/paper/2011.00747/) (2020)
- [Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding](https://mlsanity.com/paper/1912.07240/) (2019)
- [Adapting End-to-End Speech Recognition for Readable Subtitles](https://mlsanity.com/paper/2005.12143/) (2020)
- [Construction of a Large-scale Japanese ASR Corpus on TV Recordings](https://mlsanity.com/paper/2103.14736/) (2021)
- [Comparison of Self-Supervised Speech Pre-Training Methods on Flemish Dutch](https://mlsanity.com/paper/2109.14357/) (2021)
- [The 2015 Sheffield System for Transcription of Multi-Genre Broadcast Media](https://mlsanity.com/paper/1512.06643/) (2015)
- [Training ASR models by Generation of Contextual Information](https://mlsanity.com/paper/1910.12367/) (2019)

## Built on by (most cited first, 2 of 2)

- [FLAME: A New Dataset on FLemish Accounts of Momentary Experiences](https://mlsanity.com/paper/2504.14707/) (2025)
- [GLoRIA: Gated Low-Rank Interpretable Adaptation for Dialectal ASR](https://mlsanity.com/paper/2603.02464/) (2026)

## Abstract (arXiv)

TV subtitles are a rich source of transcriptions of many types of speech, ranging from read speech in news reports to conversational and spontaneous speech in talk shows and soaps. However, subtitles are not verbatim (i.e. exact) transcriptions of speech, so they cannot be used directly to improve an Automatic Speech Recognition (ASR) model. We propose a multitask dual-decoder Transformer model that jointly performs ASR and automatic subtitling. The ASR decoder (possibly pre-trained) predicts the verbatim output and the subtitle decoder generates a subtitle, while sharing the encoder. The two decoders can be independent or connected. The model is trained to perform both tasks jointly, and is able to effectively use subtitle data. We show improvements on regular ASR and on spontaneous and conversational ASR by incorporating the additional subtitle decoder. The method does not require preprocessing (aligning, filtering, pseudo-labeling, ...) of the subtitles.