# Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture

arXiv: https://arxiv.org/abs/2504.11809 · HTML: https://mlsanity.com/paper/2504.11809/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates](https://mlsanity.com/paper/1911.03167/) (2019)
- [SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2011.02048/) (2020)
- [Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation](https://mlsanity.com/paper/2206.05807/) (2022)
- [RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer](https://mlsanity.com/paper/2106.04833/) (2021)
- [Information-Transport-based Policy for Simultaneous Translation](https://mlsanity.com/paper/2210.12357/) (2022)
- [AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2305.11408/) (2023)
- [TransLLaMa: LLM-based Simultaneous Translation System](https://mlsanity.com/paper/2402.04636/) (2024)
- [Attention as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2212.07850/) (2022)
- [Learning When to Translate for Streaming Speech](https://mlsanity.com/paper/2109.07368/) (2021)
- [Speech Translation with Large Language Models: An Industrial Practice](https://mlsanity.com/paper/2312.13585/) (2023)
- [End-to-End Simultaneous Speech Translation with Differentiable Segmentation](https://mlsanity.com/paper/2305.16093/) (2023)
- [Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks](https://mlsanity.com/paper/2305.03101/) (2023)
- [Simultaneous Masking, Not Prompting Optimization: A Paradigm Shift in Fine-tuning LLMs for Simultaneous Translation](https://mlsanity.com/paper/2405.10443/) (2024)
- [LLMs Are Zero-Shot Context-Aware Simultaneous Translators](https://mlsanity.com/paper/2406.13476/) (2024)
- [Tuning Large language model for End-to-end Speech Translation](https://mlsanity.com/paper/2310.02050/) (2023)
- [InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model](https://mlsanity.com/paper/2503.02969/) (2025)
- [FASST: Fast LLM-based Simultaneous Speech Translation](https://mlsanity.com/paper/2408.09430/) (2024)
- [Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inference](https://mlsanity.com/paper/2303.07914/) (2023)
- [LLMs Can Achieve High-quality Simultaneous Machine Translation as Efficiently as Offline](https://mlsanity.com/paper/2504.09570/) (2025)

## Built on by (most cited first, 12 of 12)

- [From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment](https://mlsanity.com/paper/2507.14900/) (2025)
- [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026)
- [SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translation](https://mlsanity.com/paper/2508.07781/) (2025)
- [REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation](https://mlsanity.com/paper/2508.04946/) (2025)
- [Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech](https://mlsanity.com/paper/2604.21045/) (2026)
- [SimulSense: Sense-Driven Interpreting for Efficient Simultaneous Speech Translation](https://mlsanity.com/paper/2509.21932/) (2025)
- [OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion](https://mlsanity.com/paper/2512.00234/) (2025)
- [RASST: Retrieval-Augmented Simultaneous Speech Translation](https://mlsanity.com/paper/2601.22777/) (2026)
- [Regularized Entropy Information Adaptation with Temporal-Awareness Networks for Simultaneous Speech Translation](https://mlsanity.com/paper/2604.09916/) (2026)
- [Do LLMs Need Architectural Changes for Simultaneous Speech Translation? A Prefix-to-Prefix Data Driven Approach](https://mlsanity.com/paper/2607.13158/) (2026)
- [WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs](https://mlsanity.com/paper/2608.22704/) (2026)
- [All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2609.30416/) (2026)

## Abstract (arXiv)

Simultaneous speech translation (SimulST) produces translations incrementally while processing partial speech input. Although large language models (LLMs) have shown strong capabilities in offline translation tasks, applying them to SimulST poses notable challenges. Existing LLM-based SimulST approaches either incur significant computational overhead due to repeated encoding of bidirectional speech encoder, or they depend on a fixed read/write policy, limiting the efficiency and performance. In this work, we introduce Efficient and Adaptive Simultaneous Speech Translation (EASiST) with fully unidirectional architecture, including both speech encoder and LLM. EASiST includes a multi-latency data curation strategy to generate semantically aligned SimulST training samples and redefines SimulST as an interleaved generation task with explicit read/write tokens. To facilitate adaptive inference, we incorporate a lightweight policy head that dynamically predicts read/write actions. Additionally, we employ a multi-stage training strategy to align speech-text modalities and optimize both translation and policy behavior. Experiments on both in-domain (MuST-C) and out-of-domain (Europarl-ST) En-De and En-Es datasets demonstrate that EASiST offers superior latency-quality trade-offs compared to several strong baselines.