mlsanityData to 7 Oct 2026

Between Flexibility and Consistency: Joint Generation of Captions and Subtitles

arXiv 2107.06246 ↗First posted 13 Jul 2021cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Speech translation (ST) has lately received growing interest for the generation of subtitles without the need for an intermediate source language transcription and timing (i.e. captions). However, the joint generation of source captions and target subtitles does not only bring potential output quality advantages when the two decoding processes inform each other, but it is also often required in multilingual scenarios. In this work, we focus on ST models which generate consistent captions-subtitles in terms of structure and lexical content. We further introduce new metrics for evaluating subtitling consistency. Our findings show that joint decoding leads to increased performance and consistency between the generated captions and subtitles while still allowing for sufficient flexibility to produce subtitles conforming to language-specific needs and norms.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 20 of its 51 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

20 of 51 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  7. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  8. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  9. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  10. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  11. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  12. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
8 more
  1. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  2. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  3. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  4. 2019Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decodingcited by 21
  5. 2020Is 42 the Answer to Everything in Subtitling-oriented Speech Translation?cited by 21
  6. 2020MuST-Cinema: a Speech-to-Subtitles corpuscited by 17
  7. 2020Consistent Transcription and Translation of Speechcited by 14
  8. 2018Structured-based Curriculum Learning for End-to-end English-Japanese Speech Translationcited by 11

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 3.

7 papers.

  1. 2022LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducerscited by 10
  2. 2022Direct Speech Translation for Automatic Subtitlingcited by 7
  3. 2023Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignmentscited by 6
  4. 2023Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023cited by 6
  5. 2022Joint Generation of Captions and Subtitles with Dual Decodingcited by 5
  6. 2023Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translationcited by 4
  7. 2021One Source, Two Targets: Challenges and Rewards of Dual Decodingcited by 2