mlsanityData to 7 Oct 2026

Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation

arXiv 2102.11387 ↗First posted 22 Feb 2021cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper addresses the problem of simultaneous machine translation (SiMT) by exploring two main concepts: (a) adaptive policies to learn a good trade-off between high translation quality and low latency; and (b) visual information to support this process by providing additional (visual) contextual information which may be available before the textual input is produced. For that, we propose a multimodal approach to simultaneous machine translation using reinforcement learning, with strategies to integrate visual and textual information in both the agent and the environment. We provide an exploration on how different types of visual information and integration strategies affect the quality and latency of simultaneous translation models, and demonstrate that visual cues lead to higher quality while keeping the latency low.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

37 of 55 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2015Neural Module Networkscited by 375
  10. 2017Embodied Question Answeringcited by 335
  11. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  12. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
25 more
  1. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  2. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  3. 2016Learning to Translate in Real-time with Neural Machine Translationcited by 97
  4. 2016Can neural machine translation do simultaneous translation?cited by 85
  5. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  6. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  7. 2018Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translationcited by 51
  8. 2017Video Captioning via Hierarchical Reinforcement Learningcited by 47
  9. 2017Doubly-Attentive Decoder for Multi-modal Neural Machine Translationcited by 46
  10. 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
  11. 2017Deep Reinforcement Learning-based Image Captioning with Embedding Rewardcited by 39
  12. 2017Imagination improves Multimodal Translationcited by 38
  13. 2019Simpler and Faster Learning of Adaptive Policies for Simultaneous Translationcited by 36
  14. 2018Low-Latency Neural Speech Translationcited by 35
  15. 2017Attention Strategies for Multi-Source Sequence-to-Sequence Learningcited by 32
  16. 2019Distilling Translations with Visual Awarenesscited by 30
  17. 2020Re-translation versus Streaming for Simultaneous Translationcited by 30
  18. 2019Multimodal Machine Translation through Visuals and Speechcited by 29
  19. 2018The MeMAD Submission to the WMT18 Multimodal Translation Taskcited by 22
  20. 2016Multimodal Pivots for Image Caption Translationcited by 18
  21. 2020Simultaneous Machine Translation with Visual Contextcited by 17
  22. 2020Learning Coupled Policies for Simultaneous Machine Translation using Imitation Learningcited by 10
  23. 2018LIUM-CVC Submissions for WMT18 Multimodal Translation Taskcited by 8
  24. 2016Does Multimodality Help Human and Machine for Translation and Image Captioning?cited by 7
  25. 2020Towards Multimodal Simultaneous Neural Machine Translationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 1; 2024: 1; 2025: 1; 2026: 1.

7 papers.

  1. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  2. 2023Exploring Better Text Image Translation with Multimodal Codebookcited by 23
  3. 2025SeqPO-SiMT: Sequential Policy Optimization for Simultaneous Machine Translationcited by 13
  4. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  5. 2022Supervised Visual Attention for Simultaneous Multimodal Machine Translationcited by 2
  6. 2026Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speechcited by 1
  7. 2021Reinforcement Learning for on-line Sequence Transformationcited by 0