# SeqPO-SiMT: Sequential Policy Optimization for Simultaneous Machine Translation

arXiv: https://arxiv.org/abs/2505.20622 · HTML: https://mlsanity.com/paper/2505.20622/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 27 May 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [Learning to Translate in Real-time with Neural Machine Translation](https://mlsanity.com/paper/1610.00388/) (2016)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus](https://mlsanity.com/paper/2002.01320/) (2020)
- [Information-Transport-based Policy for Simultaneous Translation](https://mlsanity.com/paper/2210.12357/) (2022)
- [Simpler and Faster Learning of Adaptive Policies for Simultaneous Translation](https://mlsanity.com/paper/1909.01559/) (2019)
- [TransLLaMa: LLM-based Simultaneous Translation System](https://mlsanity.com/paper/2402.04636/) (2024)
- [Hidden Markov Transformer for Simultaneous Machine Translation](https://mlsanity.com/paper/2303.00257/) (2023)
- [Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent](https://mlsanity.com/paper/2407.21646/) (2024)
- [Decoder-only Streaming Transformer for Simultaneous Translation](https://mlsanity.com/paper/2406.03878/) (2024)
- [Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation](https://mlsanity.com/paper/2102.11387/) (2021)
- [SimulPL: Aligning Human Preferences in Simultaneous Machine Translation](https://mlsanity.com/paper/2502.00634/) (2025)
- [Self-Modifying State Modeling for Simultaneous Machine Translation](https://mlsanity.com/paper/2406.02237/) (2024)

## Built on by (most cited first, 13 of 13)

- [Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice](https://mlsanity.com/paper/2507.17527/) (2025)
- [SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Model](https://mlsanity.com/paper/2507.02822/) (2025)
- [Simultaneous Speech-to-Speech Translation Without Aligned Data](https://mlsanity.com/paper/2602.11072/) (2026)
- [SAS: Simulated Attention Score](https://mlsanity.com/paper/2507.07694/) (2025)
- [Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2606.02020/) (2026)
- [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026)
- [SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translation](https://mlsanity.com/paper/2508.07781/) (2025)
- [Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks](https://mlsanity.com/paper/2511.03328/) (2025)
- [Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Cameras](https://mlsanity.com/paper/2510.09230/) (2025)
- [GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization](https://mlsanity.com/paper/2601.22095/) (2026)
- [Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech](https://mlsanity.com/paper/2604.21045/) (2026)
- [ProactiveLLM: Learning Active Interaction for Streaming Large Language Models](https://mlsanity.com/paper/2606.00523/) (2026)
- [NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2606.13121/) (2026)

## Abstract (arXiv)

We present Sequential Policy Optimization for Simultaneous Machine Translation (SeqPO-SiMT), a new policy optimization framework that defines the simultaneous machine translation (SiMT) task as a sequential decision making problem, incorporating a tailored reward to enhance translation quality while reducing latency. In contrast to popular Reinforcement Learning from Human Feedback (RLHF) methods, such as PPO and DPO, which are typically applied in single-step tasks, SeqPO-SiMT effectively tackles the multi-step SiMT task. This intuitive framework allows the SiMT LLMs to simulate and refine the SiMT process using a tailored reward. We conduct experiments on six datasets from diverse domains for En to Zh and Zh to En SiMT tasks, demonstrating that SeqPO-SiMT consistently achieves significantly higher translation quality with lower latency. In particular, SeqPO-SiMT outperforms the supervised fine-tuning (SFT) model by 1.13 points in COMET, while reducing the Average Lagging by 6.17 in the NEWSTEST2021 En to Zh dataset. While SiMT operates with far less context than offline translation, the SiMT results of SeqPO-SiMT on 7B LLM surprisingly rival the offline translation of high-performing LLMs, including Qwen-2.5-7B-Instruct and LLaMA-3-8B-Instruct.