mlsanityData to 7 Oct 2026

Distilling Translations with Visual Awareness

arXiv 1906.07701 ↗First posted 18 Jun 2019cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Previous work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient. As a consequence, models tend to learn to ignore this information. We propose a translate-and-refine approach to this problem where images are only used by a second stage decoder. This approach is trained jointly to generate a good first draft translation and to improve over this draft by (i) making better use of the target language textual context (both left and right-side contexts) and (ii) making use of visual context. This approach leads to the state of the art results. Additionally, we show that it has the ability to recover from erroneous or missing words in the source language.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 10; 2023: 5; 2024: 4; 2025: 3.

30 papers.

Sort:

By area:

  1. 2021Gender Bias in Machine Translationcited by 116
  2. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  3. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  4. 2022VALHALLA: Visual Hallucination for Machine Translationcited by 27
  5. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  6. 2022Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluationcited by 26
  7. 2023Exploring Better Text Image Translation with Multimodal Codebookcited by 23
  8. 2025Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Modelscited by 22
  9. 2021Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Modelscited by 21
  10. 2022LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translationcited by 21
  11. 2021Gumbel-Attention for Multi-modal Machine Translationcited by 20
  12. 2022MSCTD: A Multimodal Sentiment Chat Translation Datasetcited by 15
18 more
  1. 2023Universal Multimodal Representation for Language Understandingcited by 14
  2. 2022Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translationcited by 13
  3. 2021Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-trainingcited by 10
  4. 2022Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspectivecited by 10
  5. 2021Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translationcited by 7
  6. 2023M2C: Towards Automatic Multimodal Manga Complementcited by 7
  7. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  8. 2022Low-resource Neural Machine Translation with Cross-modal Alignmentcited by 5
  9. 2022Multimodal Neural Machine Translation with Search Engine Based Image Retrievalcited by 4
  10. 2024Adding Multimodal Capabilities to a Text-only Translation Modelcited by 4
  11. 2024Detecting Concrete Visual Tokens for Multimodal Machine Translationcited by 3
  12. 2021Visual Cues and Error Correction for Translation Robustnesscited by 2
  13. 2021Transfer Learning for Sequence Generation: from Single-source to Multi-sourcecited by 2
  14. 2023Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languagescited by 2
  15. 2025Dual-branch Prompting for Multimodal Machine Translationcited by 2
  16. 2023Few-shot Domain-Adaptive Visually-fused Event Detection from Textcited by 1
  17. 2024Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasetscited by 1
  18. 2025GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translationcited by 1