mlsanityData to 7 Oct 2026

A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges

arXiv 2405.12669 ↗First posted 21 May 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

In recent years, multi-modal machine translation has attracted significant interest in both academia and industry due to its superior performance. It takes both textual and visual modalities as inputs, leveraging visual context to tackle the ambiguities in source texts. In this paper, we begin by offering an exhaustive overview of 99 prior works, comprehensively summarizing representative studies from the perspectives of dominant models, datasets, and evaluation metrics. Afterwards, we analyze the impact of various factors on model performance and finally discuss the possible research directions for this task in the future. Over time, multi-modal machine translation has developed more types to meet diverse needs. Unlike previous surveys confined to the early stage of multi-modal machine translation, our survey thoroughly concludes these emerging types from different aspects, so as to provide researchers with a better understanding of its current state.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 97 of its 149 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

97 of 149 references.

References · 97

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
38 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  7. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  8. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2017Dynamic Routing Between Capsulescited by 689
  11. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  12. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  13. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  14. 2018An efficient framework for learning sentence representationscited by 197
  15. 2019ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)cited by 147
  16. 2019ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019cited by 138
  17. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  18. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  19. 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
  20. 2017ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)cited by 103
  21. 2023Revisiting Relation Extraction in the era of Large Language Modelscited by 100
  22. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  23. 2019Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methodscited by 55
  24. 2019Probing the Need for Visual Context in Multimodal Machine Translationcited by 54
  25. 2020A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translationcited by 51
  26. 2017Doubly-Attentive Decoder for Multi-modal Neural Machine Translationcited by 46
  27. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  28. 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
  29. 2017Imagination improves Multimodal Translationcited by 38
  30. 2020Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewardscited by 37
  31. 2020Dynamic Context-guided Capsule Network for Multimodal Machine Translationcited by 36
  32. 2017Attention Strategies for Multi-Source Sequence-to-Sequence Learningcited by 32
  33. 2022On Vision Features in Multimodal Machine Translationcited by 31
  34. 2019Distilling Translations with Visual Awarenesscited by 30
  35. 2019Multimodal Machine Translation through Visuals and Speechcited by 29
  36. 2022VALHALLA: Visual Hallucination for Machine Translationcited by 27
  37. 2023Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucinationcited by 27
  38. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  2. 2025ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoningcited by 3
  3. 2025PATIMT-Bench: A Multi-Scenario Benchmark for Position-Aware Text Image Machine Translation in Large Vision-Language Modelscited by 2
  4. 2024ViSymRe: Vision Multimodal Symbolic Regressioncited by 1
  5. 2026Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusioncited by 0
  6. 2026Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Surveycited by 0