mlsanityData to 7 Oct 2026

StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection

arXiv 2406.06097 ↗First posted 10 Jun 2024cs.SD cs.AI cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Streaming speech-to-text translation (StreamST) is the task of automatically translating speech while incrementally receiving an audio stream. Unlike simultaneous ST (SimulST), which deals with pre-segmented speech, StreamST faces the challenges of handling continuous and unbounded audio streams. This requires additional decisions about what to retain of the previous history, which is impractical to keep entirely due to latency and computational constraints. Despite the real-world demand for real-time ST, research on streaming translation remains limited, with existing works solely focusing on SimulST. To fill this gap, we introduce StreamAtt, the first StreamST policy, and propose StreamLAAL, the first StreamST latency metric designed to be comparable with existing metrics for SimulST. Extensive experiments across all 8 languages of MuST-C v1.0 show the effectiveness of StreamAtt compared to a naive streaming baseline and the related state-of-the-art SimulST policy, providing a first step in StreamST research.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 48 of its 69 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

48 of 69 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2016Sequence-Level Knowledge Distillationcited by 761
  10. 2015Listen, Attend and Spellcited by 542
  11. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  12. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
36 more
  1. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  2. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  3. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  4. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  5. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
  6. 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75
  7. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 56
  8. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 47
  9. 2021CTC-based Compression for Direct Speech Translationcited by 45
  10. 2022Information-Transport-based Policy for Simultaneous Translationcited by 40
  11. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 39
  12. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  13. 2020Accurate Word Alignment Induction from Neural Machine Translationcited by 32
  14. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  15. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  16. 2019Adding Interpretable Attention to Neural Translation Models Improves Word Alignmentcited by 30
  17. 2020Streaming Simultaneous Speech Translation with Augmented Memory Transformercited by 30
  18. 2021Learning When to Translate for Streaming Speechcited by 30
  19. 2022CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022cited by 30
  20. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  21. 2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
  22. 2023End-to-End Simultaneous Speech Translation with Differentiable Segmentationcited by 22
  23. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  24. 2018An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translationcited by 15
  25. 2021An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategiescited by 14
  26. 2022Efficient yet Competitive Speech Translation: FBK@IWSLT2022cited by 14
  27. 2020Contextualized Translation of Automatically Segmented Speechcited by 12
  28. 2021Stream-level Latency Evaluation for Simultaneous Machine Translationcited by 12
  29. 2020On Knowledge Distillation for Direct Speech Translationcited by 11
  30. 2021Streaming Models for Joint Speech Recognition and Translationcited by 10
  31. 2022From Simultaneous to Streaming Machine Translation by Leveraging Streaming Historycited by 9
  32. 2021Towards the evaluation of automatic simultaneous speech translation from a communicative perspectivecited by 8
  33. 2022Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translationcited by 6
  34. 2022SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentationscited by 6
  35. 2023Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inferencecited by 6
  36. 2023SpeechAlign: a Framework for Speech Translation Alignment Evaluationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 8.

19 papers.

Sort:

By area:

  1. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  2. 2025InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Modelcited by 8
  3. 2025Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025cited by 7
  4. 2025Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translationcited by 4
  5. 2025Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systemscited by 4
  6. 2025CMU's IWSLT 2025 Simultaneous Speech Translation Systemcited by 2
  7. 2025MLLP-VRAIN UPV system for the IWSLT 2025 Simultaneous Speech Translation Translation taskcited by 2
  8. 2025SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translationcited by 2
  9. 2025BeaverTalk: Oregon State University's IWSLT 2025 Simultaneous Speech Translation Systemcited by 1
  10. 2025REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translationcited by 1
  11. 2026Streaming Translation and Transcription Through Speech-to-Text Causal Alignmentcited by 1
  12. 2026Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speechcited by 1
7 more
  1. 2026DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMscited by 1
  2. 2026A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translationcited by 1
  3. 2024CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translationcited by 0
  4. 2026RASST: Retrieval-Augmented Simultaneous Speech Translationcited by 0
  5. 2026SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translationcited by 0
  6. 2026A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026cited by 0
  7. 2026AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Taskcited by 0