# Gaussian Multi-head Attention for Simultaneous Machine Translation arXiv: https://arxiv.org/abs/2203.09072 ยท HTML: https://mlsanity.com/paper/2203.09072/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 43 references that are in the map. - First posted on arXiv 17 Mar 2022. ## Builds on (most cited first, 21 of 21) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019) - [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019) - [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019) - [Learning to Translate in Real-time with Neural Machine Translation](https://mlsanity.com/paper/1610.00388/) (2016) - [Monotonic Multihead Attention](https://mlsanity.com/paper/1909.12406/) (2019) - [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016) - [Online and Linear-Time Attention by Enforcing Monotonic Alignments](https://mlsanity.com/paper/1704.00784/) (2017) - [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019) - [Efficient Wait-k Models for Simultaneous Machine Translation](https://mlsanity.com/paper/2005.08595/) (2020) - [Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation](https://mlsanity.com/paper/1806.03661/) (2018) - [Simpler and Faster Learning of Adaptive Policies for Simultaneous Translation](https://mlsanity.com/paper/1909.01559/) (2019) - [Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy](https://mlsanity.com/paper/2109.05238/) (2021) - [Neural Machine Translation with Supervised Attention](https://mlsanity.com/paper/1609.04186/) (2016) - [Future-Guided Incremental Transformer for Simultaneous Translation](https://mlsanity.com/paper/2012.12465/) (2020) - [Simultaneous Translation with Flexible Policy via Restricted Imitation Learning](https://mlsanity.com/paper/1906.01135/) (2019) - [Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model](https://mlsanity.com/paper/2109.05244/) (2021) - [Learning Coupled Policies for Simultaneous Machine Translation using Imitation Learning](https://mlsanity.com/paper/2002.04306/) (2020) - [Neural Simultaneous Speech Translation Using Alignment-Based Chunking](https://mlsanity.com/paper/2005.14489/) (2020) - [Online Versus Offline NMT Quality: An In-depth Analysis on English-German and German-English](https://mlsanity.com/paper/2006.00814/) (2020) ## Built on by (most cited first, 17 of 17) - [Information-Transport-based Policy for Simultaneous Translation](https://mlsanity.com/paper/2210.12357/) (2022) - [Hidden Markov Transformer for Simultaneous Machine Translation](https://mlsanity.com/paper/2303.00257/) (2023) - [End-to-End Simultaneous Speech Translation with Differentiable Segmentation](https://mlsanity.com/paper/2305.16093/) (2023) - [StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning](https://mlsanity.com/paper/2406.03049/) (2024) - [Wait-info Policy: Balancing Source and Target at Information Level for Simultaneous Machine Translation](https://mlsanity.com/paper/2210.11220/) (2022) - [Learning Optimal Policy for Simultaneous Machine Translation via Binary Search](https://mlsanity.com/paper/2305.12774/) (2023) - [Turning Fixed to Adaptive: Integrating Post-Evaluation into Simultaneous Machine Translation](https://mlsanity.com/paper/2210.11900/) (2022) - [Decoder-only Streaming Transformer for Simultaneous Translation](https://mlsanity.com/paper/2406.03878/) (2024) - [Glancing Future for Simultaneous Machine Translation](https://mlsanity.com/paper/2309.06179/) (2023) - [Simultaneous Machine Translation with Tailored Reference](https://mlsanity.com/paper/2310.13588/) (2023) - [Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inference](https://mlsanity.com/paper/2303.07914/) (2023) - [Unified Segment-to-Segment Framework for Simultaneous Sequence Generation](https://mlsanity.com/paper/2310.17940/) (2023) - [Self-Modifying State Modeling for Simultaneous Machine Translation](https://mlsanity.com/paper/2406.02237/) (2024) - [Enhanced Simultaneous Machine Translation with Word-level Policies](https://mlsanity.com/paper/2310.16417/) (2023) - [CBSiMT: Mitigating Hallucination in Simultaneous Machine Translation with Weighted Prefix-to-Prefix Training](https://mlsanity.com/paper/2311.03672/) (2023) - [Rethinking the Reasonability of the Test Set for Simultaneous Machine Translation](https://mlsanity.com/paper/2303.00969/) (2023) - [Attention-Based Adaptive Policies for Simultaneous Speech-to-Text Translation](https://mlsanity.com/paper/2609.30839/) (2026) ## Abstract (arXiv) Simultaneous machine translation (SiMT) outputs translation while receiving the streaming source inputs, and hence needs a policy to determine where to start translating. The alignment between target and source words often implies the most informative source word for each target word, and hence provides the unified control over translation quality and latency, but unfortunately the existing SiMT methods do not explicitly model the alignment to perform the control. In this paper, we propose Gaussian Multi-head Attention (GMA) to develop a new SiMT policy by modeling alignment and translation in a unified manner. For SiMT policy, GMA models the aligned source position of each target word, and accordingly waits until its aligned position to start translating. To integrate the learning of alignment into the translation model, a Gaussian distribution centered on predicted aligned position is introduced as an alignment-related prior, which cooperates with translation-related soft attention to determine the final attention. Experiments on En-Vi and De-En tasks show that our method outperforms strong baselines on the trade-off between translation and latency.