mlsanityData to 7 Oct 2026

m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt

arXiv 2403.17556 ↗First posted 26 Mar 2024cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Multilingual translation supports multiple translation directions by projecting all languages in a shared space, but the translation quality is undermined by the difference between languages in the text-only modality, especially when the number of languages is large. To bridge this gap, we introduce visual context as the universal language-independent representation to facilitate multilingual translation. In this paper, we propose a framework to leverage the multimodal prompt to guide the Multimodal Multilingual neural Machine Translation (m3P), which aligns the representations of different languages with the same meaning and generates the conditional vision-language memory for translation. We construct a multilingual multimodal instruction dataset (InstrMulti102) to support 102 languages. Our method aims to minimize the representation distance of different languages by regarding the image as a central language. Experimental results show that m3P outperforms previous text-only baselines and multilingual multimodal methods by a large margin. Furthermore, the probing experiments validate the effectiveness of our method in enhancing translation under the low-resource and massively multilingual scenario.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 52 of its 70 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

52 of 70 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  3. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  6. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  8. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  10. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  11. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  12. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  13. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  14. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  15. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  16. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  17. 2019Massively Multilingual Neural Machine Translationcited by 246
  18. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  19. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  20. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  21. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  22. 2018Universal Neural Machine Translation for Extremely Low Resource Languagescited by 66
  23. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  24. 2020A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translationcited by 51
  25. 2024xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoningcited by 46
  26. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  27. 2023Radiology-Llama2: Best-in-Class Large Language Model for Radiologycited by 42
  28. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
  29. 2021Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Taskcited by 34
  30. 2022On Vision Features in Multimodal Machine Translationcited by 31
  31. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  32. 2020XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoderscited by 24
  33. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  34. 2021Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Modelscited by 21
  35. 2022LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translationcited by 21
  36. 2022GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminatorcited by 20
  37. 2023Deeply Coupled Cross-Modal Prompt Learningcited by 11
  38. 2018LIUM-CVC Submissions for WMT18 Multimodal Translation Taskcited by 8

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 5.

5 papers.

  1. 2024UniCoder: Scaling Code Large Language Model via Universal Codecited by 18
  2. 2024IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Webcited by 16
  3. 2024XFormParser: A Simple and Effective Multimodal Multilingual Semi-structured Form Parsercited by 5
  4. 2024RoNID: New Intent Discovery with Generated-Reliable Labels and Cluster-friendly Representationscited by 2
  5. 2024Towards Real-world Scenario: Imbalanced New Intent Discoverycited by 2