# StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection

arXiv: https://arxiv.org/abs/2406.06097 · HTML: https://mlsanity.com/paper/2406.06097/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 69 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2011.02048/) (2020)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)
- [Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation](https://mlsanity.com/paper/2206.05807/) (2022)
- [Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection](https://mlsanity.com/paper/2005.11185/) (2020)
- [CTC-based Compression for Direct Speech Translation](https://mlsanity.com/paper/2102.01578/) (2021)
- [RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer](https://mlsanity.com/paper/2106.04833/) (2021)
- [Information-Transport-based Policy for Simultaneous Translation](https://mlsanity.com/paper/2210.12357/) (2022)
- [AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2305.11408/) (2023)
- [Accurate Word Alignment Induction from Neural Machine Translation](https://mlsanity.com/paper/2004.14837/) (2020)
- [Attention as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2212.07850/) (2022)
- [Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR](https://mlsanity.com/paper/2106.06636/) (2021)
- [Adding Interpretable Attention to Neural Translation Models Improves Word Alignment](https://mlsanity.com/paper/1901.11359/) (2019)
- [Streaming Simultaneous Speech Translation with Augmented Memory Transformer](https://mlsanity.com/paper/2011.00033/) (2020)
- [Learning When to Translate for Streaming Speech](https://mlsanity.com/paper/2109.07368/) (2021)

## Built on by (most cited first, 19 of 19)

- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model](https://mlsanity.com/paper/2503.02969/) (2025)
- [Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025](https://mlsanity.com/paper/2506.17077/) (2025)
- [Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation](https://mlsanity.com/paper/2509.17349/) (2025)
- [Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems](https://mlsanity.com/paper/2512.17648/) (2025)
- [CMU's IWSLT 2025 Simultaneous Speech Translation System](https://mlsanity.com/paper/2506.13143/) (2025)
- [MLLP-VRAIN UPV system for the IWSLT 2025 Simultaneous Speech Translation Translation task](https://mlsanity.com/paper/2506.18828/) (2025)
- [SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translation](https://mlsanity.com/paper/2508.07781/) (2025)
- [BeaverTalk: Oregon State University's IWSLT 2025 Simultaneous Speech Translation System](https://mlsanity.com/paper/2505.24016/) (2025)
- [REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation](https://mlsanity.com/paper/2508.04946/) (2025)
- [Streaming Translation and Transcription Through Speech-to-Text Causal Alignment](https://mlsanity.com/paper/2603.11578/) (2026)
- [Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech](https://mlsanity.com/paper/2604.21045/) (2026)
- [DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMs](https://mlsanity.com/paper/2605.31432/) (2026)
- [A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2606.15059/) (2026)
- [CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translation](https://mlsanity.com/paper/2410.16011/) (2024)
- [RASST: Retrieval-Augmented Simultaneous Speech Translation](https://mlsanity.com/paper/2601.22777/) (2026)
- [SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2603.16924/) (2026)
- [A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026](https://mlsanity.com/paper/2606.03948/) (2026)
- [AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task](https://mlsanity.com/paper/2606.03967/) (2026)

## Abstract (arXiv)

Streaming speech-to-text translation (StreamST) is the task of automatically translating speech while incrementally receiving an audio stream. Unlike simultaneous ST (SimulST), which deals with pre-segmented speech, StreamST faces the challenges of handling continuous and unbounded audio streams. This requires additional decisions about what to retain of the previous history, which is impractical to keep entirely due to latency and computational constraints. Despite the real-world demand for real-time ST, research on streaming translation remains limited, with existing works solely focusing on SimulST. To fill this gap, we introduce StreamAtt, the first StreamST policy, and propose StreamLAAL, the first StreamST latency metric designed to be comparable with existing metrics for SimulST. Extensive experiments across all 8 languages of MuST-C v1.0 show the effectiveness of StreamAtt compared to a naive streaming baseline and the related state-of-the-art SimulST policy, providing a first step in StreamST research.