mlsanityData to 7 Oct 2026

UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

arXiv 2104.00332 ↗First posted 1 Apr 2021cs.CVcited by 64

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision-and-language pre-training has achieved impressive success in learning multimodal representations between vision and language. To generalize this success to non-English languages, we introduce UC2, the first machine translation-augmented framework for cross-lingual cross-modal representation learning. To tackle the scarcity problem of multilingual captions for image datasets, we first augment existing English-only datasets with other languages via machine translation (MT). Then we extend the standard Masked Language Modeling and Image-Text Matching training objectives to multilingual setting, where alignment between different languages is captured through shared visual context (i.e, using image as pivot). To facilitate the learning of a joint embedding space of images and all languages of interest, we further propose two novel pre-training tasks, namely Masked Region-to-Token Modeling (MRTM) and Visual Translation Language Modeling (VTLM), leveraging MT-enhanced translated data. Evaluation on multilingual image-text retrieval and multilingual visual question answering benchmarks demonstrates that our proposed framework achieves new state-of-the-art on diverse non-English benchmarks while maintaining comparable performance to monolingual pre-trained models on English tasks.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 1 Apr 2021.

Builds on

42 of 55 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
30 more
  1. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  2. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  3. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  5. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  6. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  7. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  8. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  9. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  10. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  11. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  12. 2019Massively Multilingual Neural Machine Translationcited by 246
  13. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  14. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  15. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  16. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  17. 2017STAIR Captions: Constructing a Large-Scale Japanese Image Caption Datasetcited by 47
  18. 2017Doubly-Attentive Decoder for Multi-modal Neural Machine Translationcited by 46
  19. 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
  20. 2017Imagination improves Multimodal Translationcited by 38
  21. 2020Visual Grounding in Video for Unsupervised Word Translationcited by 27
  22. 2018A Visual Attention Grounding Neural Model for Multimodal Machine Translationcited by 22
  23. 2019MULE: Multimodal Universal Language Embeddingcited by 22
  24. 2020Learning to Scale Multilingual Representations for Vision-Language Taskscited by 22
  25. 2020Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivotingcited by 22
  26. 2015Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learningcited by 18
  27. 2017Image Pivoting for Learning Multilingual Multimodal Representationscited by 17
  28. 2018Doubly Attentive Transformer Machine Translationcited by 5
  29. 2020M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-trainingcited by 4
  30. 2017CUNI System for the WMT17 Multimodal Translation Taskcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 23; 2023: 17; 2024: 8; 2025: 5; 2026: 3.

64 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  3. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  4. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  5. 2021Class-agnostic Object Detection with Multi-modal Transformercited by 85
  6. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  7. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  8. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  9. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  10. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  11. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  12. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
38 more
  1. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44
  2. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  3. 2023CLIP in Medical Imaging: A Surveycited by 40
  4. 2023Transformers in Speech Processing: A Surveycited by 38
  5. 2021ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integrationcited by 31
  6. 2024Parrot: Multilingual Visual Instruction Tuningcited by 30
  7. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  8. 2022Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluationcited by 26
  9. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  10. 2021Understanding Chinese Video and Language via Contrastive Multimodal Pre-Trainingcited by 23
  11. 2022Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignmentcited by 23
  12. 2022Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learningcited by 22
  13. 2022Multilingual Multimodal Learning with Machine Translated Textcited by 16
  14. 2022ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generationcited by 13
  15. 2024Multimodal LLM Enhanced Cross-lingual Cross-modal Retrievalcited by 13
  16. 2023LEMaRT: Label-Efficient Masked Region Transform for Image Harmonizationcited by 12
  17. 2024Semi-Supervised Semantic Segmentation Based on Pseudo-Labels: A Surveycited by 12
  18. 2023Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representationscited by 11
  19. 2021Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-trainingcited by 10
  20. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Taskscited by 9
  21. 2023Translation-Enhanced Multilingual Text-to-Image Generationcited by 9
  22. 2023Dual-view Curricular Optimal Transport for Cross-lingual Cross-modal Retrievalcited by 9
  23. 2022Delving Deeper into Cross-lingual Visual Question Answeringcited by 8
  24. 20223D-TOGO: Towards Text-Guided Cross-Category 3D Object Generationcited by 8
  25. 2022ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generationcited by 7
  26. 2024Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learningcited by 7
  27. 2025WaMo: Wavelet-Enhanced Multi-Frequency Trajectory Analysis for Fine-Grained Text-Motion Retrievalcited by 7
  28. 2022Improving the Cross-Lingual Generalisation in Visual Question Answeringcited by 5
  29. 2022Generalizing Multimodal Pre-training into Multilingual via Language Acquisitioncited by 3
  30. 2022MuMUR : Multilingual Multimodal Universal Retrievalcited by 3
  31. 2023Entity-Level Text-Guided Image Manipulationcited by 3
  32. 2023Meta-learning For Vision-and-language Cross-lingual Transfercited by 3
  33. 2022Vision-and-Language Pretrainingcited by 2
  34. 2023RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-trainingcited by 2
  35. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languagescited by 2
  36. 2024Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answeringcited by 2
  37. 2022cViL: Cross-Lingual Training of Vision-Language Models using Knowledge Distillationcited by 1
  38. 2022Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunitiescited by 1

50 most cited of 64 All 64 (JSON)