mlsanityData to 7 Oct 2026

LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation

arXiv 2210.15461 ↗First posted 19 Oct 2022cs.CL cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Multimodal Machine Translation (MMT) focuses on enhancing text-only translation with visual features, which has attracted considerable attention from both natural language processing and computer vision communities. Recent advances still struggle to train a separate model for each language pair, which is costly and unaffordable when the number of languages increases in the real world. In other words, the multilingual multimodal machine translation (Multilingual MMT) task has not been investigated, which aims to handle the aforementioned issues by providing a shared semantic space for multiple languages. Besides, the image modality has no language boundaries, which is superior to bridging the semantic gap between languages. To this end, we first propose the Multilingual MMT task by establishing two new Multilingual MMT benchmark datasets covering seven languages. Then, an effective baseline LVP-M3 using visual prompts is proposed to support translations between different languages, which includes three stages (token encoding, language-aware visual prompt generation, and language translation). Extensive experimental results on our constructed benchmark datasets demonstrate the effectiveness of LVP-M3 method for Multilingual MMT.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 47 of its 58 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

47 of 58 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2016Layer Normalizationcited by 5,579
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
35 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  4. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  7. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  8. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  9. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  10. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  11. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  12. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  13. 2016Dynamic Filter Networkscited by 318
  14. 2020Conditional Convolutions for Instance Segmentationcited by 314
  15. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  16. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  17. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  18. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  19. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  20. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  21. 2015Where To Look: Focus Regions for Visual Question Answeringcited by 64
  22. 2020A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translationcited by 51
  23. 2017Doubly-Attentive Decoder for Multi-modal Neural Machine Translationcited by 46
  24. 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
  25. 2017Imagination improves Multimodal Translationcited by 38
  26. 2020Dynamic Context-guided Capsule Network for Multimodal Machine Translationcited by 36
  27. 2017Attention Strategies for Multi-Source Sequence-to-Sequence Learningcited by 32
  28. 2022On Vision Features in Multimodal Machine Translationcited by 31
  29. 2019Distilling Translations with Visual Awarenesscited by 30
  30. 2018A Visual Attention Grounding Neural Model for Multimodal Machine Translationcited by 22
  31. 2020Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivotingcited by 22
  32. 2021Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Modelscited by 21
  33. 2018CUNI System for the WMT18 Multimodal Translation Taskcited by 13
  34. 2022UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translationcited by 10
  35. 2022CoupleFace: Relation Matters for Face Recognition Distillationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 9; 2025: 3; 2026: 3.

21 papers.

Sort:

By area:

  1. 2023OWL: A Large Language Model for IT Operationscited by 43
  2. 2024IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Webcited by 16
  3. 2024LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detectioncited by 15
  4. 2024E^2-LLM: Efficient and Extreme Length Extension of Large Language Modelscited by 11
  5. 2025Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flowcited by 9
  6. 2023M2C: Towards Automatic Multimodal Manga Complementcited by 7
  7. 2023Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translationcited by 6
  8. 2024Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategiescited by 6
  9. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  10. 2024SVIPTR: Fast and Efficient Scene Text Recognition with Vision Permutable Extractorcited by 5
  11. 2024m3P: Towards Multimodal Multilingual Translation with Multimodal Promptcited by 5
  12. 2025MultiScript30k: Leveraging Multilingual Embeddings to Extend Cross Script Parallel Datacited by 4
9 more
  1. 2023Multilingual Entity and Relation Extraction from Unified to Language-specific Trainingcited by 3
  2. 2023MT4CrossOIE: Multi-stage Tuning for Cross-lingual Open Information Extractioncited by 3
  3. 2023Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labelscited by 2
  4. 2024Context-Informed Machine Translation of Manga using Multimodal Large Language Modelscited by 1
  5. 2026VIDA: A Dataset for Visually Dependent Ambiguity in Multimodal Machine Translationcited by 1
  6. 2024MLAD: A Unified Model for Multi-system Log Anomaly Detectioncited by 0
  7. 2025LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translationcited by 0
  8. 2026Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusioncited by 0
  9. 2026Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weightingcited by 0