mlsanityData to 14 Oct 2026

Probing the Need for Visual Context in Multimodal Machine Translation

arXiv 1903.08678 ↗First posted 20 Mar 2019cs.CLcited by 54

Not on the map. Why?

Abstract (arXiv)

Current work on multimodal machine translation (MMT) has suggested that the visual modality is either unnecessary or only marginally beneficial. We posit that this is a consequence of the very simple, short and repetitive sentences used in the only available dataset for the task (Multi30K), rendering the source text sufficient as context. In the general case, however, we believe that it is possible to combine visual and textual information in order to ground translations. In this paper we probe the contribution of the visual modality to state-of-the-art MMT models by conducting a systematic analysis where we partially deprive the models from source-side textual context. Our results show that under limited textual context, models are capable of leveraging the visual input to generate better translations. This contradicts the current belief that MMT models disregard the visual modality because of either the quality of the image features or the way they are integrated into the model.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 11; 2023: 13; 2024: 8; 2025: 5; 2026: 2.

54 papers.

Sort:

By area:

  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  2. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  3. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  4. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  5. 2022On Vision Features in Multimodal Machine Translationcited by 31
  6. 2022VALHALLA: Visual Hallucination for Machine Translationcited by 27
  7. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  8. 2021A Survey on Multi-modal Summarizationcited by 26
  9. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  10. 2022Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluationcited by 26
  11. 2021ReFormer: The Relational Transformer for Image Captioningcited by 23
  12. 2023Exploring Better Text Image Translation with Multimodal Codebookcited by 23
38 more
  1. 2021Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Modelscited by 21
  2. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  3. 2021Cross-lingual Visual Pre-training for Multimodal Machine Translationcited by 19
  4. 2021Journalistic Guidelines Aware News Image Captioningcited by 14
  5. 2021Guiding Visual Question Generationcited by 14
  6. 2023Universal Multimodal Representation for Language Understandingcited by 14
  7. 2023CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translationcited by 14
  8. 2022Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translationcited by 13
  9. 2022VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translationcited by 11
  10. 2021Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Expertscited by 9
  11. 2023BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translationcited by 9
  12. 2022Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translationcited by 7
  13. 2022MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysiscited by 6
  14. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  15. 2021"Wikily" Supervised Neural Translation Tailored to Cross-Lingual Taskscited by 5
  16. 2021ViTA: Visual-Linguistic Translation by Aligning Object Tagscited by 5
  17. 2022Low-resource Neural Machine Translation with Cross-modal Alignmentcited by 5
  18. 2022BAN-Cap: A Multi-Purpose English-Bangla Image Descriptions Datasetcited by 4
  19. 2022Multimodal Neural Machine Translation with Search Engine Based Image Retrievalcited by 4
  20. 2024Adding Multimodal Capabilities to a Text-only Translation Modelcited by 4
  21. 20243AM: An Ambiguity-Aware Multi-Modal Machine Translation Datasetcited by 4
  22. 2024Detecting Frames in News Headlines and Lead Images in U.S. Gun Violence Coveragecited by 4
  23. 2025MultiScript30k: Leveraging Multilingual Embeddings to Extend Cross Script Parallel Datacited by 4
  24. 2021BERTGEN: Multi-task Generation through BERTcited by 3
  25. 2023Multimodal Speech Recognition for Language-Guided Embodied Agentscited by 3
  26. 2023Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairscited by 3
  27. 2023Video-Helpful Multimodal Machine Translationcited by 3
  28. 2024Detecting Concrete Visual Tokens for Multimodal Machine Translationcited by 3
  29. 2024Towards Zero-Shot Multimodal Machine Translationcited by 3
  30. 2021Visual Cues and Error Correction for Translation Robustnesscited by 2
  31. 2023RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-trainingcited by 2
  32. 2023Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languagescited by 2
  33. 2025TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentariescited by 2
  34. 2025Dual-branch Prompting for Multimodal Machine Translationcited by 2
  35. 2023Visual Question Generation in Bengalicited by 1
  36. 2024The Case for Evaluating Multimodal Translation Models on Text Datasetscited by 1
  37. 2024Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasetscited by 1
  38. 2023Generalization algorithm of multimodal pre-training model based on graph-text self-supervised trainingcited by 0

50 most cited of 54 All 54 (JSON)