# Improving Simultaneous Machine Translation with Monolingual Data

arXiv: https://arxiv.org/abs/2212.01188 · HTML: https://mlsanity.com/paper/2212.01188/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 32 of its 57 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [Understanding Knowledge Distillation in Non-autoregressive Machine Translation](https://mlsanity.com/paper/1911.02727/) (2019)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [Learning to Translate in Real-time with Neural Machine Translation](https://mlsanity.com/paper/1610.00388/) (2016)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [Norm-Based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2006.02014/) (2020)
- [Understanding and Improving Lexical Choice in Non-Autoregressive Translation](https://mlsanity.com/paper/2012.14583/) (2020)
- [Simultaneous Translation Policies: From Fixed to Adaptive](https://mlsanity.com/paper/2004.13169/) (2020)
- [Information-Transport-based Policy for Simultaneous Translation](https://mlsanity.com/paper/2210.12357/) (2022)
- [Prevent the Language Model from being Overconfident in Neural Machine Translation](https://mlsanity.com/paper/2105.11098/) (2021)
- [Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation](https://mlsanity.com/paper/2106.00903/) (2021)
- [Future-Guided Incremental Transformer for Simultaneous Translation](https://mlsanity.com/paper/2012.12465/) (2020)
- [JParaCrawl: A Large Scale Web-Based English-Japanese Parallel Corpus](https://mlsanity.com/paper/1911.10668/) (2019)
- [Improving Simultaneous Translation by Incorporating Pseudo-References with Fewer Reorderings](https://mlsanity.com/paper/2010.11247/) (2020)
- [Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation](https://mlsanity.com/paper/1808.09006/) (2018)
- [Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation](https://mlsanity.com/paper/2106.00941/) (2021)
- [The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021](https://mlsanity.com/paper/2107.00279/) (2021)
- [The USYD-JD Speech Translation System for IWSLT 2021](https://mlsanity.com/paper/2107.11572/) (2021)
- [Improving Non-autoregressive Neural Machine Translation with Monolingual Data](https://mlsanity.com/paper/2005.00932/) (2020)

## Built on by (most cited first, 13 of 13)

- [Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?](https://mlsanity.com/paper/2308.12898/) (2023)
- [Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation](https://mlsanity.com/paper/2404.14827/) (2024)
- [APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Training](https://mlsanity.com/paper/2506.03483/) (2025)
- [NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates](https://mlsanity.com/paper/2410.20814/) (2024)
- [Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inference](https://mlsanity.com/paper/2303.07914/) (2023)
- [DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization](https://mlsanity.com/paper/2411.14055/) (2024)
- [LLMs Can Achieve High-quality Simultaneous Machine Translation as Efficiently as Offline](https://mlsanity.com/paper/2504.09570/) (2025)
- [AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs](https://mlsanity.com/paper/2507.18584/) (2025)
- [Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models](https://mlsanity.com/paper/2602.05897/) (2026)
- [Simultaneous Interpretation Corpus Construction by Large Language Models in Distant Language Pair](https://mlsanity.com/paper/2404.12299/) (2024)
- [Holistic Exploration on Universal Decompositional Semantic Parsing: Architecture, Data Augmentation, and LLM Paradigm](https://mlsanity.com/paper/2307.13424/) (2023)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)
- [On the Hallucination in Simultaneous Machine Translation](https://mlsanity.com/paper/2406.07239/) (2024)

## Abstract (arXiv)

Simultaneous machine translation (SiMT) is usually done via sequence-level knowledge distillation (Seq-KD) from a full-sentence neural machine translation (NMT) model. However, there is still a significant performance gap between NMT and SiMT. In this work, we propose to leverage monolingual data to improve SiMT, which trains a SiMT student on the combination of bilingual data and external monolingual data distilled by Seq-KD. Preliminary experiments on En-Zh and En-Ja news domain corpora demonstrate that monolingual data can significantly improve translation quality (e.g., +3.15 BLEU on En-Zh). Inspired by the behavior of human simultaneous interpreters, we propose a novel monolingual sampling strategy for SiMT, considering both chunk length and monotonicity. Experimental results show that our sampling strategy consistently outperforms the random sampling strategy (and other conventional typical NMT monolingual sampling strategies) by avoiding the key problem of SiMT -- hallucination, and has better scalability. We achieve +0.72 BLEU improvements on average against random sampling on En-Zh and En-Ja. Data and codes can be found at https://github.com/hexuandeng/Mono4SiMT.