mlsanityData to 7 Oct 2026

On Vision Features in Multimodal Machine Translation

arXiv 2203.09173 ↗First posted 17 Mar 2022cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

Previous work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is on the quality of vision models. In this work, we investigate the impact of vision models on MMT. Given the fact that Transformer is becoming popular in computer vision, we experiment with various strong models (such as Vision Transformer) and enhanced features (such as object-detection and image captioning). We develop a selective attention model to study the patch-level contribution of an image in MMT. On detailed probing tasks, we find that stronger vision models are helpful for learning translation from the visual modality. Our results also suggest the need of carefully examining MMT models, especially when current benchmarks are small-scale and biased. Our code could be found at urlhttps://github.com/libeineu/fairseqmmt.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

25 of 35 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  10. 2021Instances as Queriescited by 157
  11. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  12. 2019Probing the Need for Visual Context in Multimodal Machine Translationcited by 54
13 more
  1. 2020A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translationcited by 51
  2. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  3. 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
  4. 2017Imagination improves Multimodal Translationcited by 38
  5. 2020Dynamic Context-guided Capsule Network for Multimodal Machine Translationcited by 36
  6. 2020Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translationcited by 34
  7. 2017Attention Strategies for Multi-Source Sequence-to-Sequence Learningcited by 32
  8. 2018The MeMAD Submission to the WMT18 Multimodal Translation Taskcited by 22
  9. 2021Gumbel-Attention for Multi-modal Machine Translationcited by 20
  10. 2016Multimodal Attention for Neural Machine Translationcited by 19
  11. 2021Cross-lingual Visual Pre-training for Multimodal Machine Translationcited by 19
  12. 2020Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Groundingcited by 13
  13. 2017An empirical study on the effectiveness of images in Multimodal Neural Machine Translationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 13; 2024: 7; 2025: 5; 2026: 2.

31 papers.

Sort:

By area:

  1. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  2. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  3. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoningcited by 60
  4. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answeringcited by 59
  5. 2023Exploring Better Text Image Translation with Multimodal Codebookcited by 23
  6. 2022LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translationcited by 21
  7. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  8. 2022Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspectivecited by 10
  9. 2023BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translationcited by 9
  10. 2022Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translationcited by 7
  11. 2023M2C: Towards Automatic Multimodal Manga Complementcited by 7
  12. 2023Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translationcited by 6
19 more
  1. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  2. 2025Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translationcited by 6
  3. 2022Low-resource Neural Machine Translation with Cross-modal Alignmentcited by 5
  4. 2024m3P: Towards Multimodal Multilingual Translation with Multimodal Promptcited by 5
  5. 20243AM: An Ambiguity-Aware Multi-Modal Machine Translation Datasetcited by 4
  6. 2023Visual Information Matters for ASR Error Correctioncited by 3
  7. 2023Improving speech translation by fusing speech and textcited by 3
  8. 2023Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairscited by 3
  9. 2023Video-Helpful Multimodal Machine Translationcited by 3
  10. 2024Towards Zero-Shot Multimodal Machine Translationcited by 3
  11. 2023Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languagescited by 2
  12. 2025TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentariescited by 2
  13. 2025Dual-branch Prompting for Multimodal Machine Translationcited by 2
  14. 2024Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasetscited by 1
  15. 2025Multimodal Machine Translation with Visual Scene Graph Pruningcited by 1
  16. 2024Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Taskscited by 0
  17. 2025Memory Reviving, Continuing Learning and Beyond: Evaluation of Pre-trained Encoders and Decoders for Multimodal Machine Translationcited by 0
  18. 2026Video-guided Machine Translation with Global Video Contextcited by 0
  19. 2026Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weightingcited by 0