# Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

arXiv: https://arxiv.org/abs/2211.01335 · HTML: https://mlsanity.com/paper/2211.01335/

## Key facts

- Cited by 109 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 57 of its 81 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 57)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)

## Built on by (most cited first, 30 of 109)

- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://mlsanity.com/paper/2309.15112/) (2023)
- [Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer](https://mlsanity.com/paper/2511.22699/) (2025)
- [Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model](https://mlsanity.com/paper/2502.10248/) (2025)
- [ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities](https://mlsanity.com/paper/2305.11172/) (2023)
- [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://mlsanity.com/paper/2211.06679/) (2022)
- [Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages](https://mlsanity.com/paper/2308.12038/) (2023)
- [Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks](https://mlsanity.com/paper/2306.04362/) (2023)
- [RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports](https://mlsanity.com/paper/2405.14137/) (2024)
- [Parrot: Multilingual Visual Instruction Tuning](https://mlsanity.com/paper/2406.02539/) (2024)
- [Few-shot Adaptation of Multi-modal Foundation Models: A Survey](https://mlsanity.com/paper/2401.01736/) (2024)
- [Heterogeneous Contrastive Learning for Foundation Models and Beyond](https://mlsanity.com/paper/2404.00225/) (2024)
- [Movie101: A New Movie Understanding Benchmark](https://mlsanity.com/paper/2305.12140/) (2023)
- [NLLB-CLIP -- train performant multilingual image retrieval model on a budget](https://mlsanity.com/paper/2309.01859/) (2023)
- [LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application](https://mlsanity.com/paper/2405.03988/) (2024)
- [VLM as Policy: Common-Law Content Moderation Framework for Short Video Platform](https://mlsanity.com/paper/2504.14904/) (2025)
- [TextPainter: Multimodal Text Image Generation with Visual-harmony and Text-comprehension for Poster Design](https://mlsanity.com/paper/2308.04733/) (2023)
- [From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approach](https://mlsanity.com/paper/2412.11892/) (2024)
- [Multilingual Diversity Improves Vision-Language Representations](https://mlsanity.com/paper/2405.16915/) (2024)
- [FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Model](https://mlsanity.com/paper/2510.10921/) (2025)
- [FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training](https://mlsanity.com/paper/2411.11927/) (2024)
- [Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations](https://mlsanity.com/paper/2306.08658/) (2023)
- [Side4Video: Spatial-Temporal Side Network for Memory-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2311.15769/) (2023)
- [Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support](https://mlsanity.com/paper/2401.14688/) (2024)
- [An Empirical Study and Analysis of Text-to-Image Generation Using Large Language Model-Powered Textual Representation](https://mlsanity.com/paper/2405.12914/) (2024)
- [MATE: Meet At The Embedding -- Connecting Images with Long Texts](https://mlsanity.com/paper/2407.09541/) (2024)
- [PosterVerse: A Full-Workflow Framework for Commercial-Grade Poster Generation with HTML-Based Scalable Typography](https://mlsanity.com/paper/2601.03993/) (2026)

## Abstract (arXiv)

The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). We have released our codes, models, and demos in https://github.com/OFA-Sys/Chinese-CLIP