mlsanityData to 7 Oct 2026

CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet

arXiv 2212.06138 ↗First posted 12 Dec 2022cs.CV cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Recent studies have shown that CLIP has achieved remarkable success in performing zero-shot inference while its fine-tuning performance is not satisfactory. In this paper, we identify that fine-tuning performance is significantly impacted by hyper-parameter choices. We examine various key hyper-parameters and empirically evaluate their impact in fine-tuning CLIP for classification tasks through a comprehensive study. We find that the fine-tuning performance of CLIP is substantially underestimated. Equipped with hyper-parameter refinement, we demonstrate CLIP itself is better or at least competitive in fine-tuning compared with large-scale supervised pre-training approaches or latest works that use CLIP as prediction targets in Masked Image Modeling. Specifically, CLIP ViT-Base/16 and CLIP ViT-Large/14 can achieve 85.7%,88.0% finetuning Top-1 accuracy on the ImageNet-1K dataset . These observations challenge the conventional conclusion that CLIP is not suitable for fine-tuning, and motivate us to rethink recently proposed improvements based on CLIP. We will release our code publicly at urlhttps://github.com/LightDXY/FT-CLIP.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 30 of its 30 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

30 of 30 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  10. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  11. 2021Scaling Vision Transformerscited by 993
  12. 2016Deep Networks with Stochastic Depthcited by 973
18 more
  1. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  2. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  4. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  5. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  6. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  7. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  8. 2022DeiT III: Revenge of the ViTcited by 417
  9. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  10. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  11. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  12. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  13. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  14. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  15. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  16. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  17. 2022Bootstrapped Masked Autoencoders for Vision BERT Pretrainingcited by 60
  18. 2022CAE v2: Context Autoencoder with CLIP Targetcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 10; 2025: 10; 2026: 4.

35 papers.

Sort:

By area:

  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  3. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  4. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  5. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  6. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  7. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  8. 2023Contrastive Feature Masking Open-Vocabulary Vision Transformercited by 31
  9. 2023Layer-wise Linear Mode Connectivitycited by 22
  10. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  11. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillationcited by 18
  12. 2024Unsupervised Domain Adaption Harnessing Vision-Language Pre-trainingcited by 12
23 more
  1. 2024Anchor-based Robust Finetuning of Vision-Language Modelscited by 10
  2. 2024Classification Done Right for Vision-Language Pre-Trainingcited by 10
  3. 2023Multi-scale Diffusion Denoised Smoothingcited by 9
  4. 2024Anytime Continual Learning for Open Vocabulary Classificationcited by 8
  5. 2024Multilingual Vision-Language Pre-training for the Remote Sensing Domaincited by 7
  6. 2023Masking meets Supervision: A Strong Learning Alliancecited by 3
  7. 2023RetailKLIP : Finetuning OpenCLIP backbone using metric learning on a single GPU for Zero-shot retail product image classificationcited by 3
  8. 2025VLMT: Vision-Language Multimodal Transformer for Multimodal Multi-hop Question Answeringcited by 3
  9. 2025Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directionscited by 3
  10. 2025Differentiable Hierarchical Visual Tokenizationcited by 3
  11. 2025AME: Aligned Manifold Entropy for Robust Vision-Language Distillationcited by 2
  12. 2025Universal Embedding Function for Traffic Classification via QUIC Domain Recognition Pretraining: A Transfer Learning Successcited by 1
  13. 2025Frugal Incremental Generative Modeling using Variational Autoencoderscited by 1
  14. 2025Mitigating Data Exfiltration Attacks through Layer-Wise Learning Rate Decay Fine-Tuningcited by 1
  15. 2026Breaking the Limits of Open-Weight CLIP: An Optimization Framework for Self-supervised Fine-tuning of CLIPcited by 1
  16. 2026jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillationcited by 1
  17. 2024Enhancing Weakly Supervised Semantic Segmentation with Multi-modal Foundation Models: An End-to-End Approachcited by 0
  18. 2024Adapting Pretrained ViTs with Convolution Injector for Visuo-Motor Controlcited by 0
  19. 2025Exploring the Use of Contrastive Language-Image Pre-Training for Human Posture Classification: Insights from Yoga Pose Analysiscited by 0
  20. 2025Feature Space Analysis by Guided Diffusion Modelcited by 0
  21. 2025Dual Randomized Smoothing: Beyond Global Noise Variancecited by 0
  22. 2026Toward Clinically Ready Foundation Models in Medical Image Analysis: Adaptation Mechanisms and Deployment Trade-offscited by 0
  23. 2026FDIR: Harmonizing Fidelity and Human-Machine Preference in Lossy Compression Image Restorationcited by 0