# Reducing Position Bias in Simultaneous Machine Translation with Length-Aware Framework

arXiv: https://arxiv.org/abs/2203.09053 · HTML: https://mlsanity.com/paper/2203.09053/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [Learning to Translate in Real-time with Neural Machine Translation](https://mlsanity.com/paper/1610.00388/) (2016)
- [Monotonic Multihead Attention](https://mlsanity.com/paper/1909.12406/) (2019)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [Online and Linear-Time Attention by Enforcing Monotonic Alignments](https://mlsanity.com/paper/1704.00784/) (2017)
- [Look at the First Sentence: Position Bias in Question Answering](https://mlsanity.com/paper/2004.14602/) (2020)
- [Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation](https://mlsanity.com/paper/2010.10907/) (2020)
- [Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior](https://mlsanity.com/paper/1908.07181/) (2019)
- [Addressing the Rare Word Problem in Neural Machine Translation](https://mlsanity.com/paper/1410.8206/) (2014)
- [Simpler and Faster Learning of Adaptive Policies for Simultaneous Translation](https://mlsanity.com/paper/1909.01559/) (2019)
- [Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy](https://mlsanity.com/paper/2109.05238/) (2021)
- [Future-Guided Incremental Transformer for Simultaneous Translation](https://mlsanity.com/paper/2012.12465/) (2020)
- [Synchronous Bidirectional Neural Machine Translation](https://mlsanity.com/paper/1905.04847/) (2019)
- [AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate](https://mlsanity.com/paper/2109.06481/) (2021)
- [Variational Recurrent Neural Machine Translation](https://mlsanity.com/paper/1801.05119/) (2018)
- [Simultaneous Translation with Flexible Policy via Restricted Imitation Learning](https://mlsanity.com/paper/1906.01135/) (2019)
- [Token-level Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2010.04380/) (2020)
- [Improving Simultaneous Translation by Incorporating Pseudo-References with Fewer Reorderings](https://mlsanity.com/paper/2010.11247/) (2020)
- [Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model](https://mlsanity.com/paper/2109.05244/) (2021)
- [Position Bias Mitigation: A Knowledge-Aware Graph Model for Emotion Cause Extraction](https://mlsanity.com/paper/2106.03518/) (2021)
- [Modeling Fluency and Faithfulness for Diverse Neural Machine Translation](https://mlsanity.com/paper/1912.00178/) (2019)
- [Scheduled Sampling Based on Decoding Steps for Neural Machine Translation](https://mlsanity.com/paper/2108.12963/) (2021)
- [Synchronous Bidirectional Inference for Neural Sequence Generation](https://mlsanity.com/paper/1902.08955/) (2019)
- [Learning Coupled Policies for Simultaneous Machine Translation using Imitation Learning](https://mlsanity.com/paper/2002.04306/) (2020)

## Built on by (most cited first, 16 of 16)

- [Information-Transport-based Policy for Simultaneous Translation](https://mlsanity.com/paper/2210.12357/) (2022)
- [Hidden Markov Transformer for Simultaneous Machine Translation](https://mlsanity.com/paper/2303.00257/) (2023)
- [End-to-End Simultaneous Speech Translation with Differentiable Segmentation](https://mlsanity.com/paper/2305.16093/) (2023)
- [StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning](https://mlsanity.com/paper/2406.03049/) (2024)
- [Wait-info Policy: Balancing Source and Target at Information Level for Simultaneous Machine Translation](https://mlsanity.com/paper/2210.11220/) (2022)
- [Learning Optimal Policy for Simultaneous Machine Translation via Binary Search](https://mlsanity.com/paper/2305.12774/) (2023)
- [Turning Fixed to Adaptive: Integrating Post-Evaluation into Simultaneous Machine Translation](https://mlsanity.com/paper/2210.11900/) (2022)
- [Decoder-only Streaming Transformer for Simultaneous Translation](https://mlsanity.com/paper/2406.03878/) (2024)
- [Glancing Future for Simultaneous Machine Translation](https://mlsanity.com/paper/2309.06179/) (2023)
- [Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inference](https://mlsanity.com/paper/2303.07914/) (2023)
- [Unified Segment-to-Segment Framework for Simultaneous Sequence Generation](https://mlsanity.com/paper/2310.17940/) (2023)
- [Agent-SiMT: Agent-assisted Simultaneous Machine Translation with Large Language Models](https://mlsanity.com/paper/2406.06910/) (2024)
- [PsFuture: A Pseudo-Future-based Zero-Shot Adaptive Policy for Simultaneous Machine Translation](https://mlsanity.com/paper/2410.04075/) (2024)
- [CBSiMT: Mitigating Hallucination in Simultaneous Machine Translation with Weighted Prefix-to-Prefix Training](https://mlsanity.com/paper/2311.03672/) (2023)
- [On the Hallucination in Simultaneous Machine Translation](https://mlsanity.com/paper/2406.07239/) (2024)
- [Context Consistency between Training and Testing in Simultaneous Machine Translation](https://mlsanity.com/paper/2311.07066/) (2023)

## Abstract (arXiv)

Simultaneous machine translation (SiMT) starts translating while receiving the streaming source inputs, and hence the source sentence is always incomplete during translating. Different from the full-sentence MT using the conventional seq-to-seq architecture, SiMT often applies prefix-to-prefix architecture, which forces each target word to only align with a partial source prefix to adapt to the incomplete source in streaming inputs. However, the source words in the front positions are always illusoryly considered more important since they appear in more prefixes, resulting in position bias, which makes the model pay more attention on the front source positions in testing. In this paper, we first analyze the phenomenon of position bias in SiMT, and develop a Length-Aware Framework to reduce the position bias by bridging the structural gap between SiMT and full-sentence MT. Specifically, given the streaming inputs, we first predict the full-sentence length and then fill the future source position with positional encoding, thereby turning the streaming inputs into a pseudo full-sentence. The proposed framework can be integrated into most existing SiMT methods to further improve performance. Experiments on two representative SiMT methods, including the state-of-the-art adaptive policy, show that our method successfully reduces the position bias and thereby achieves better SiMT performance.