mlsanityData to 7 Oct 2026

Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023

arXiv 2309.15554 ↗First posted 27 Sep 2023cs.CL cs.AI cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper describes the FBK's participation in the Simultaneous Translation and Automatic Subtitling tracks of the IWSLT 2023 Evaluation Campaign. Our submission focused on the use of direct architectures to perform both tasks: for the simultaneous one, we leveraged the knowledge already acquired by offline-trained models and directly applied a policy to obtain the real-time inference; for the subtitling one, we adapted the direct ST model to produce well-formed subtitles and exploited the same architecture to produce timestamps needed for the subtitle synchronization with audiovisual content. Our English-German SimulST system shows a reduced computational-aware latency compared to the one achieved by the top-ranked systems in the 2021 and 2022 rounds of the task, with gains of up to 3.5 BLEU. Our automatic subtitling system outperforms the only existing solution based on a direct system by 3.7 and 1.7 SubER in English-German and English-Spanish respectively.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 43 of its 65 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

43 of 65 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  6. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  7. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  8. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  9. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  10. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  11. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
  12. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 56
31 more
  1. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  2. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  3. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 47
  4. 2021CTC-based Compression for Direct Speech Translationcited by 45
  5. 2020Simultaneous Translation Policies: From Fixed to Adaptivecited by 42
  6. 2022Information-Transport-based Policy for Simultaneous Translationcited by 40
  7. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 39
  8. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  9. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  10. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  11. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  12. 2022CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022cited by 30
  13. 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
  14. 2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
  15. 2020Is 42 the Answer to Everything in Subtitling-oriented Speech Translation?cited by 21
  16. 2020MuST-Cinema: a Speech-to-Subtitles corpuscited by 17
  17. 2022Average Token Delay: A Latency Metric for Simultaneous Translationcited by 15
  18. 2021An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategiescited by 14
  19. 2021Isochrony-Aware Neural Machine Translation for Automatic Dubbingcited by 14
  20. 2022Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translationcited by 14
  21. 2022Efficient yet Competitive Speech Translation: FBK@IWSLT2022cited by 14
  22. 2019Prosodic Phrase Alignment for Machine Dubbingcited by 13
  23. 2021The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021cited by 13
  24. 2020On Knowledge Distillation for Direct Speech Translationcited by 11
  25. 2022SubER: A Metric for Automatic Evaluation of Subtitle Qualitycited by 9
  26. 2021Simultaneous Speech Translation for Live Subtitling: from Delay to Displaycited by 8
  27. 2021Between Flexibility and Consistency: Joint Generation of Captions and Subtitlescited by 7
  28. 2022Direct Speech Translation for Automatic Subtitlingcited by 7
  29. 2021Dealing with training and test segmentation mismatch: FBK@IWSLT2021cited by 6
  30. 2022Dodging the Data Bottleneck: Automatic Subtitling with Automatically Segmented ST Corporacited by 5
  31. 2022Evaluating Subtitle Segmentation for End-to-end Generation Systemscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 1.

6 papers.

  1. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  2. 2024Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agentcited by 14
  3. 2024SBAAM! Eliminating Transcript Dependency in Automatic Subtitlingcited by 1
  4. 2024SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translationcited by 1
  5. 2025Vision-Grounded Machine Interpreting: Improving the Translation Process through Visual Cuescited by 0
  6. 2026Bridging the Usability Gap: Lessons from Interpreting Studies for Machine Interpreting Designcited by 0