mlsanityData to 7 Oct 2026

Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

arXiv 2211.01335 ↗First posted 2 Nov 2022cs.CV cs.CLcited by 108

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). We have released our codes, models, and demos in https://github.com/OFA-Sys/Chinese-CLIP

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 81 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

57 of 81 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  16. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  17. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  18. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  19. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  20. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  21. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  22. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  23. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  24. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  25. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  26. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  27. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  28. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  29. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  30. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  31. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  32. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  33. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  34. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  35. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  36. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  37. 2021M6: A Chinese Multimodal Pretrainercited by 91
  38. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 84

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 28; 2024: 34; 2025: 23; 2026: 22.

108 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2025Qwen-Image Technical Reportcited by 973
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  5. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  6. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  7. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  8. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  9. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  10. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  11. 2023Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarkscited by 33
  12. 2024Parrot: Multilingual Visual Instruction Tuningcited by 30
38 more
  1. 2024RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reportscited by 29
  2. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  3. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  4. 2023Movie101: A New Movie Understanding Benchmarkcited by 25
  5. 2023NLLB-CLIP -- train performant multilingual image retrieval model on a budgetcited by 24
  6. 2024LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Applicationcited by 21
  7. 2025VLM as Policy: Common-Law Content Moderation Framework for Short Video Platformcited by 19
  8. 2023TextPainter: Multimodal Text Image Generation with Visual-harmony and Text-comprehension for Poster Designcited by 18
  9. 2024From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approachcited by 15
  10. 2024Multilingual Diversity Improves Vision-Language Representationscited by 14
  11. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  12. 2023Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representationscited by 11
  13. 2023Side4Video: Spatial-Temporal Side Network for Memory-Efficient Image-to-Video Transfer Learningcited by 11
  14. 2024An Empirical Study and Analysis of Text-to-Image Generation Using Large Language Model-Powered Textual Representationcited by 11
  15. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 11
  16. 2024Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Supportcited by 10
  17. 2024MATE: Meet At The Embedding -- Connecting Images with Long Textscited by 10
  18. 2026PosterVerse: A Full-Workflow Framework for Commercial-Grade Poster Generation with HTML-Based Scalable Typographycited by 8
  19. 2026LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understandingcited by 8
  20. 2023TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real Worldcited by 7
  21. 2023CAPIVARA: Cost-Efficient Approach for Improving Multilingual CLIP Performance on Low-Resource Languagescited by 7
  22. 2024ViLReF: An Expert Knowledge Enabled Vision-Language Retinal Foundation Modelcited by 7
  23. 2023Sticker820K: Empowering Interactive Retrieval with Stickerscited by 6
  24. 2023Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communitiescited by 6
  25. 2025MUSE: A Simple Yet Effective Multimodal Search-Based Framework for Lifelong User Interest Modelingcited by 6
  26. 2023PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generationcited by 5
  27. 2025SeriesBench: A Benchmark for Narrative-Driven Drama Series Understandingcited by 5
  28. 2023Efficient Cross-Lingual Transfer for Chinese Stable Diffusion with Images as Pivotscited by 4
  29. 2023iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Designcited by 4
  30. 2024SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignmentcited by 4
  31. 2024M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretrainingcited by 4
  32. 2024A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generationcited by 4
  33. 2024VisionCLIP: An Med-AIGC based Ethical Language-Image Foundation Model for Generalizable Retina Image Analysiscited by 4
  34. 2024A Survey on Large Language Models from Concept to Implementationcited by 4
  35. 2024Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Losscited by 4
  36. 2025Towards Cross-modal Retrieval in Chinese Cultural Heritage Documents: Dataset and Solutioncited by 4
  37. 2023Edit As You Wish: Video Caption Editing with Multi-grained User Controlcited by 3
  38. 2023Cross-view Semantic Alignment for Livestreaming Product Recognitioncited by 3

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Meta CLIP 2: A Worldwide Scaling Recipe
  2. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
  3. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining
  4. 2023Improved baselines for vision-language pre-training
  5. 2024Contrastive Localized Language-Image Pre-Training
  6. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-training
  7. 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Study
  8. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark