# Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems

arXiv: https://arxiv.org/abs/2512.17648 · HTML: https://mlsanity.com/paper/2512.17648/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 19 Dec 2025.

## Builds on (most cited first, 16 of 16)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST](https://mlsanity.com/paper/2509.14128/) (2025)
- [Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation](https://mlsanity.com/paper/1806.03661/) (2018)
- [AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2305.11408/) (2023)
- [Re-translation versus Streaming for Simultaneous Translation](https://mlsanity.com/paper/2004.03643/) (2020)
- [Re-Translation Strategies For Long Form, Simultaneous, Spoken Language Translation](https://mlsanity.com/paper/1912.03393/) (2019)
- [StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection](https://mlsanity.com/paper/2406.06097/) (2024)
- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks](https://mlsanity.com/paper/2507.19634/) (2025)
- [Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTS](https://mlsanity.com/paper/2011.04845/) (2020)
- [End-to-End Evaluation for Low-Latency Simultaneous Speech Translation](https://mlsanity.com/paper/2308.03415/) (2023)
- [Simultaneous Translation for Unsegmented Input: A Sliding Window Approach](https://mlsanity.com/paper/2210.09754/) (2022)
- [Long-Form End-to-End Speech Translation via Latent Alignment Segmentation](https://mlsanity.com/paper/2309.11384/) (2023)

## Built on by (most cited first, 4 of 4)

- [DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMs](https://mlsanity.com/paper/2605.31432/) (2026)
- [Benchmarking Speech-to-Speech Translation Models](https://mlsanity.com/paper/2606.03241/) (2026)
- [A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026](https://mlsanity.com/paper/2606.03948/) (2026)
- [AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task](https://mlsanity.com/paper/2606.03967/) (2026)

## Abstract (arXiv)

Streaming Speech-to-Text Translation (StreamST) requires producing translations concurrently with incoming speech under strict latency constraints, demanding models that balance low latency with high translation quality. Despite rapid progress, evaluation remains fragmented across existing frameworks, which make different assumptions about how systems operate - for example, whether they process continuous speech or short pre-segmented audio, and whether they support output revision (retranslation) or not (incremental). For instance, SimulEval, the most widely used framework, supports only incremental decoding, assumes short segmented inputs, and lacks a native support for system demonstrations. As a result, comparing systems fairly and consistently across studies remains challenging, with no unified solution for benchmarking and interactive demonstration. To address this gap, we introduce simulstream, the first open-source framework for StreamST evaluation and demonstration. It supports both incremental and re-translation decoding on long-form speech, provides fine-grained logging for quality and latency evaluation, and includes an interactive web interface for real-time visualization and comparison.