mlsanityData to 14 Oct 2026

Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark

arXiv 2202.06767 ↗First posted 14 Feb 2022cs.CV cs.LGcited by 114

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Pre-training (VLP) models have shown remarkable performance on various downstream tasks. Their success heavily relies on the scale of pre-trained cross-modal datasets. However, the lack of large-scale datasets and benchmarks in Chinese hinders the development of Chinese VLP models and broader multilingual applications. In this work, we release a large-scale Chinese cross-modal dataset named Wukong, which contains 100 million Chinese image-text pairs collected from the web. Wukong aims to benchmark different multi-modal pre-training methods to facilitate the VLP research and community development. Furthermore, we release a group of models pre-trained with various image encoders (ViT-B/ViT-L/SwinT) and also apply advanced pre-training techniques into VLP such as locked-image text tuning, token-wise similarity in contrastive learning, and reduced-token interaction. Extensive experiments and a benchmarking of different downstream tasks including a new largest human-verified image-text test dataset are also provided. Experiments show that Wukong can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods. For the zero-shot image classification task on 10 datasets, WukongViT-L achieves an average accuracy of 73.03%. For the image-text retrieval task, it achieves a mean recall of 71.6% on AIC-ICC which is 12.9% higher than WenLan 2.0. Also, our Wukong models are benchmarked on downstream tasks with other variants on multiple datasets, e.g., Flickr8K-CN, Flickr-30K-CN, COCO-CN, et al. More information can be referred to: https://wukong-dataset.github.io/wukong-dataset/.

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 72 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

40 of 72 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
27 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  8. 2021Scaling Vision Transformerscited by 999
  9. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  12. 2019How multilingual is Multilingual BERT?cited by 645
  13. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  14. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  15. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  16. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  17. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  18. 2021M6: A Chinese Multimodal Pretrainercited by 91
  19. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  20. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  21. 2017AI Challenger : A Large-scale Dataset for Going Deeper in Image Understandingcited by 73
  22. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  23. 2020Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCOcited by 46
  24. 2021LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrievalcited by 44
  25. 2021Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretrainingcited by 42
  26. 2017Fluency-Guided Cross-Lingual Image Captioningcited by 33
  27. 2017Image Pivoting for Learning Multilingual Multimodal Representationscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 18; 2024: 40; 2025: 41; 2026: 13.

114 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  3. 2023A Survey on Multimodal Large Language Modelscited by 909
  4. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  5. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  6. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  7. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  8. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  9. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  10. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 183
  11. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 180
  12. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
38 more
  1. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  2. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  3. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  4. 2025Ovis2.5 Technical Reportcited by 108
  5. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  6. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  7. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  8. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  9. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
  10. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  11. 2025Ovis-U1 Technical Reportcited by 67
  12. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 66
  13. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 54
  14. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
  15. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 47
  16. 2023Natural Language-Assisted Sign Language Recognitioncited by 46
  17. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editingcited by 46
  18. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  19. 2023Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Studycited by 38
  20. 2023BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decodingcited by 36
  21. 2024RIGID: A Training-free and Model-Agnostic Framework for Robust AI-Generated Image Detectioncited by 36
  22. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  23. 2023Edit Everything: A Text-Guided Generative System for Images Editingcited by 33
  24. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  25. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
  26. 2024WildFake: A Large-scale Challenging Dataset for AI-Generated Images Detectioncited by 31
  27. 2024GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Modelscited by 29
  28. 2026Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scalecited by 29
  29. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  30. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughnesscited by 25
  31. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  32. 2024The Evolution of Multimodal Model Architecturescited by 21
  33. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  34. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scalecited by 18
  35. 2024POINTS: Improving Your Vision-language Model with Affordable Strategiescited by 17
  36. 2025LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesiscited by 15
  37. 2024Multilingual Diversity Improves Vision-Language Representationscited by 14
  38. 2024Do Language Models Understand Time?cited by 14

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022CCMB: A Large-scale Chinese Cross-modal Benchmark
  2. 2022VLP: A Survey on Vision-Language Pre-training
  3. 2022WuDaoMM: A large-scale Multi-Modal Dataset for Pre-training models
  4. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  5. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese
  6. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate
  7. 2021Contrastive Vision-Language Pre-training with Limited Resources
  8. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training