mlsanityData to 7 Oct 2026

MURAL: Multimodal, Multitask Retrieval Across Languages

arXiv 2109.05125 ↗First posted 10 Sep 2021cs.IR cs.AI cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

Both image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages. We use both types of pairs in MURAL (MUltimodal, MUltitask Representations Across Languages), a dual encoder that solves two tasks: 1) image-text matching and 2) translation pair matching. By incorporating billions of translation pairs, MURAL extends ALIGN (Jia et al. PMLR'21)--a state-of-the-art dual encoder learned from 1.8 billion noisy image-text pairs. When using the same encoders, MURAL's performance matches or exceeds ALIGN's cross-modal retrieval performance on well-resourced languages across several datasets. More importantly, it considerably improves performance on under-resourced languages, showing that text-text learning can overcome a paucity of image-caption examples for these languages. On the Wikipedia Image-Text dataset, for example, MURAL-base improves zero-shot mean recall by 8.1% on average for eight under-resourced languages and by 6.8% on average when fine-tuning. We additionally show that MURAL's text representations cluster not only with respect to genealogical connections but also based on areal linguistics, such as the Balkan Sprachbund.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 35 of its 59 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

35 of 59 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  11. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  12. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
23 more
  1. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  2. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  3. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  4. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  5. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  6. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  7. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  8. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  9. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  10. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 61
  11. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  12. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  13. 2017STAIR Captions: Constructing a Large-Scale Japanese Image Caption Datasetcited by 47
  14. 2020Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCOcited by 46
  15. 2020PMIndia -- A Collection of Parallel Corpora of Languages of Indiacited by 40
  16. 2021Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrievalcited by 27
  17. 2020Towards Zero-shot Cross-lingual Image Retrievalcited by 25
  18. 2019AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learningcited by 22
  19. 2020Learning to Scale Multilingual Representations for Vision-Language Taskscited by 22
  20. 2017Image Pivoting for Learning Multilingual Multimodal Representationscited by 17
  21. 2017JESC: Japanese-English Subtitle Corpuscited by 16
  22. 2019Polyglot Contextual Representations Improve Crosslingual Transfercited by 16
  23. 2020The Ubiqus English-Inuktitut System for WMT20cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 18; 2023: 15; 2024: 3; 2025: 4; 2026: 2.

44 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  7. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  8. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  9. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  10. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  11. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  12. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
32 more
  1. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  2. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 66
  3. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Modelscited by 64
  4. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  5. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  6. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
  7. 2022Simple and Effective Synthesis of Indoor 3D Scenescited by 33
  8. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  9. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  10. 2023Tem-adapter: Adapting Image-Text Pretraining for Video Question Answercited by 22
  11. 2023De-Diffusion Makes Text a Strong Cross-Modal Interfacecited by 14
  12. 2022Large-scale Bilingual Language-Image Contrastive Learningcited by 12
  13. 2023CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfercited by 10
  14. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Taskscited by 9
  15. 2022ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-trainingcited by 8
  16. 2022Image Search with Text Feedback by Additive Attention Compositional Learningcited by 7
  17. 2022ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generationcited by 7
  18. 2023BELT:Bootstrapping Electroencephalography-to-Language Decoding and Zero-Shot Sentiment Classification by Natural Language Supervisioncited by 5
  19. 2023Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Promptscited by 3
  20. 2025Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligningcited by 3
  21. 2022Vision-and-Language Pretrainingcited by 2
  22. 2023Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignmentcited by 2
  23. 2022GPPF: A General Perception Pre-training Framework via Sparsely Activated Multi-Task Learningcited by 1
  24. 2023Making the Most of What You Have: Adapting Pre-trained Visual Language Models in the Low-data Regimecited by 1
  25. 2024Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrievalcited by 1
  26. 2025Multilingual Vision-Language Models, A Surveycited by 1
  27. 2023Multi-Prompt with Depth Partitioned Cross-Modal Learningcited by 0
  28. 2023PRE: Vision-Language Prompt Learning with Reparameterization Encodercited by 0
  29. 2024Foundations of Multisensory Artificial Intelligencecited by 0
  30. 2025Towards Understanding Ambiguity Resolution in Multimodal Inference of Meaningcited by 0
  31. 2026Multilingual-To-Multimodal (M2M): Unlocking New Languages with Monolingual Textcited by 0
  32. 2026BrokenBind: Universal Modality Exploration beyond Dataset Boundariescited by 0