# Simultaneous Machine Translation with Visual Context

arXiv: https://arxiv.org/abs/2009.07310 · HTML: https://mlsanity.com/paper/2009.07310/

## Key facts

- Cited by 17 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 15 Sep 2020.

## Built on by (most cited first, 17 of 17)

- [Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/2105.14462/) (2021)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study](https://mlsanity.com/paper/2410.00545/) (2024)
- [Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation](https://mlsanity.com/paper/2102.11387/) (2021)
- [Simultaneous Multi-Pivot Neural Machine Translation](https://mlsanity.com/paper/2104.07410/) (2021)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation](https://mlsanity.com/paper/2502.20420/) (2025)
- [Monotonic Simultaneous Translation with Chunk-wise Reordering and Refinement](https://mlsanity.com/paper/2110.09646/) (2021)
- [BERTGEN: Multi-task Generation through BERT](https://mlsanity.com/paper/2106.03484/) (2021)
- [Detecting Euphemisms with Literal Descriptions and Visual Imagery](https://mlsanity.com/paper/2211.04576/) (2022)
- [Visual Cues and Error Correction for Translation Robustness](https://mlsanity.com/paper/2103.07352/) (2021)
- [Supervised Visual Attention for Simultaneous Multimodal Machine Translation](https://mlsanity.com/paper/2201.09324/) (2022)
- [Enhanced Simultaneous Machine Translation with Word-level Policies](https://mlsanity.com/paper/2310.16417/) (2023)
- [Exploring Supervised and Unsupervised Rewards in Machine Translation](https://mlsanity.com/paper/2102.11403/) (2021)
- [Recent Trends in Artificial Intelligence Technology: A Scoping Review](https://mlsanity.com/paper/2305.04532/) (2023)
- [Toward Machine Interpreting: Lessons from Human Interpreting Studies](https://mlsanity.com/paper/2508.07964/) (2025)

## Abstract (arXiv)

Simultaneous machine translation (SiMT) aims to translate a continuous input text stream into another language with the lowest latency and highest quality possible. The translation thus has to start with an incomplete source text, which is read progressively, creating the need for anticipation. In this paper, we seek to understand whether the addition of visual information can compensate for the missing source context. To this end, we analyse the impact of different multimodal approaches and visual features on state-of-the-art SiMT frameworks. Our results show that visual context is helpful and that visually-grounded models based on explicit object region information are much better than commonly used global features, reaching up to 3 BLEU points improvement under low latency scenarios. Our qualitative analysis illustrates cases where only the multimodal systems are able to translate correctly from English into gender-marked languages, as well as deal with differences in word order, such as adjective-noun placement between English and French.