mlsanityData to 7 Oct 2026

Direct Speech Translation for Automatic Subtitling

arXiv 2209.13192 ↗First posted 27 Sep 2022cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Automatic subtitling is the task of automatically translating the speech of audiovisual content into short pieces of timed text, i.e. subtitles and their corresponding timestamps. The generated subtitles need to conform to space and time requirements, while being synchronised with the speech and segmented in a way that facilitates comprehension. Given its considerable complexity, the task has so far been addressed through a pipeline of components that separately deal with transcribing, translating, and segmenting text into subtitles, as well as predicting timestamps. In this paper, we propose the first direct ST model for automatic subtitling that generates subtitles in the target language along with their timestamps with a single model. Our experiments on 7 language pairs show that our approach outperforms a cascade system in the same data condition, also being competitive with production tools on both in-domain and newly-released out-domain benchmarks covering new scenarios.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 56 of its 125 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

56 of 125 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  12. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
38 more
  1. 2016Sequence-Level Knowledge Distillationcited by 761
  2. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  3. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  4. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  5. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  6. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 155
  7. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  8. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  9. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  10. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  11. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  12. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 75
  13. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  14. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  15. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  16. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  17. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 50
  18. 2021CTC-based Compression for Direct Speech Translationcited by 45
  19. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  20. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  21. 2019Multilingual End-to-End Speech Translationcited by 37
  22. 2019Controlling the Output Length of Neural Machine Translationcited by 36
  23. 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
  24. 2020Is 42 the Answer to Everything in Subtitling-oriented Speech Translation?cited by 21
  25. 2021Machine Translation Verbosity Control for Automatic Dubbingcited by 18
  26. 2021Isometric MT: Neural Machine Translation for Automatic Dubbingcited by 18
  27. 2020MuST-Cinema: a Speech-to-Subtitles corpuscited by 17
  28. 2020Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decodercited by 17
  29. 2021Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translationcited by 17
  30. 2021Speechformer: Reducing Information Loss in Direct Speech Translationcited by 15
  31. 2021Isochrony-Aware Neural Machine Translation for Automatic Dubbingcited by 14
  32. 2022Efficient yet Competitive Speech Translation: FBK@IWSLT2022cited by 14
  33. 2019Prosodic Phrase Alignment for Machine Dubbingcited by 13
  34. 2021Is "moby dick" a Whale or a Bird? Named Entities and Terminology in Speech Translationcited by 11
  35. 2020Adapting End-to-End Speech Recognition for Readable Subtitlescited by 10
  36. 2021Streaming Models for Joint Speech Recognition and Translationcited by 10
  37. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  38. 2022SubER: A Metric for Automatic Evaluation of Subtitle Qualitycited by 9

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 3; 2026: 2.

7 papers.

  1. 2023Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023cited by 6
  2. 2025EVE: Towards End-to-End Video Subtitle Extraction with Vision-Language Modelscited by 6
  3. 2025Summarizing Speech: A Comprehensive Surveycited by 5
  4. 2024SBAAM! Eliminating Transcript Dependency in Automatic Subtitlingcited by 1
  5. 2025V-SAT: Video Subtitle Annotation Toolcited by 0
  6. 2026FuseAlign: Forced Alignment in the Wildcited by 0
  7. 2026Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translationcited by 0