mlsanityData to 7 Oct 2026

ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation

arXiv 2211.04861 ↗First posted 9 Nov 2022cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent cross-lingual cross-modal works attempt to extend Vision-Language Pre-training (VLP) models to non-English inputs and achieve impressive performance. However, these models focus only on understanding tasks utilizing encoder-only architecture. In this paper, we propose ERNIE-UniX2, a unified cross-lingual cross-modal pre-training framework for both generation and understanding tasks. ERNIE-UniX2 integrates multiple pre-training paradigms (e.g., contrastive learning and language modeling) based on encoder-decoder architecture and attempts to learn a better joint representation across languages and modalities. Furthermore, ERNIE-UniX2 can be seamlessly fine-tuned for varieties of generation and understanding downstream tasks. Pre-trained on both multilingual text-only and image-text datasets, ERNIE-UniX2 achieves SOTA results on various cross-lingual cross-modal generation and understanding tasks such as multimodal machine translation and multilingual visual question answering.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 46 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

38 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
25 more
  1. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  2. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  6. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  7. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  8. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  9. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  10. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  11. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  12. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  13. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  14. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  15. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 77
  16. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  17. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 74
  18. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  19. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  20. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  21. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  22. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44
  23. 2021ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generationcited by 41
  24. 2020VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generationcited by 34
  25. 2017Baselines and test data for cross-lingual inferencecited by 11

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 2; 2025: 1.

7 papers.

  1. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  2. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  3. 2024ParGo: Bridging Vision-Language with Partial and Global Viewscited by 40
  4. 2023Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labelscited by 2
  5. 2023Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languagescited by 2
  6. 2025Multimodal Machine Translation with Visual Scene Graph Pruningcited by 1
  7. 2023Few-shot Multimodal Multitask Multilingual Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Improving the Cross-Lingual Generalisation in Visual Question Answering
  2. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training
  3. 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
  4. 2022Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning
  5. 2022cViL: Cross-Lingual Training of Vision-Language Models using Knowledge Distillation
  6. 2024CROME: Cross-Modal Adapters for Efficient Multimodal LLM
  7. 2022Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training
  8. 2025Multilingual Vision-Language Models, A Survey