# Dodging the Data Bottleneck: Automatic Subtitling with Automatically Segmented ST Corpora

arXiv: https://arxiv.org/abs/2209.10608 · HTML: https://mlsanity.com/paper/2209.10608/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 21 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Improving the Transformer Translation Model with Document-Level Context](https://mlsanity.com/paper/1810.03581/) (2018)
- [When and Why is Document-level Context Useful in Neural Machine Translation?](https://mlsanity.com/paper/1910.00294/) (2019)
- [CTC-based Compression for Direct Speech Translation](https://mlsanity.com/paper/2102.01578/) (2021)
- [Multilingual End-to-End Speech Translation](https://mlsanity.com/paper/1910.00254/) (2019)
- [Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution](https://mlsanity.com/paper/2103.06799/) (2021)
- [Multimodal Machine Translation through Visuals and Speech](https://mlsanity.com/paper/1911.12798/) (2019)
- [Is 42 the Answer to Everything in Subtitling-oriented Speech Translation?](https://mlsanity.com/paper/2006.01080/) (2020)
- [MuST-Cinema: a Speech-to-Subtitles corpus](https://mlsanity.com/paper/2002.10829/) (2020)
- [Contextualized Translation of Automatically Segmented Speech](https://mlsanity.com/paper/2008.02270/) (2020)
- [Dealing with training and test segmentation mismatch: FBK@IWSLT2021](https://mlsanity.com/paper/2106.12607/) (2021)
- [Evaluating Subtitle Segmentation for End-to-end Generation Systems](https://mlsanity.com/paper/2205.09360/) (2022)

## Built on by (most cited first, 5 of 5)

- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [Direct Speech Translation for Automatic Subtitling](https://mlsanity.com/paper/2209.13192/) (2022)
- [Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023](https://mlsanity.com/paper/2309.15554/) (2023)
- [Automating Easy Read Text Segmentation](https://mlsanity.com/paper/2406.11464/) (2024)
- [SBAAM! Eliminating Transcript Dependency in Automatic Subtitling](https://mlsanity.com/paper/2405.10741/) (2024)

## Abstract (arXiv)

Speech translation for subtitling (SubST) is the task of automatically translating speech data into well-formed subtitles by inserting subtitle breaks compliant to specific displaying guidelines. Similar to speech translation (ST), model training requires parallel data comprising audio inputs paired with their textual translations. In SubST, however, the text has to be also annotated with subtitle breaks. So far, this requirement has represented a bottleneck for system development, as confirmed by the dearth of publicly available SubST corpora. To fill this gap, we propose a method to convert existing ST corpora into SubST resources without human intervention. We build a segmenter model that automatically segments texts into proper subtitles by exploiting audio and text in a multimodal fashion, achieving high segmentation quality in zero-shot conditions. Comparative experiments with SubST systems respectively trained on manual and automatic segmentations result in similar performance, showing the effectiveness of our approach.