mlsanityData to 7 Oct 2026

Exploring Vision-Language Models for Imbalanced Learning

arXiv 2304.01457 ↗First posted 4 Apr 2023cs.AI cs.CV cs.LGcited by 35

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-Language models (VLMs) that use contrastive language-image pre-training have shown promising zero-shot classification performance. However, their performance on imbalanced dataset is relatively poor, where the distribution of classes in the training dataset is skewed, leading to poor performance in predicting minority classes. For instance, CLIP achieved only 5% accuracy on the iNaturalist18 dataset. We propose to add a lightweight decoder to VLMs to avoid OOM (out of memory) problem caused by large number of classes and capture nuanced features for tail classes. Then, we explore improvements of VLMs using prompt tuning, fine-tuning, and incorporating imbalanced algorithms such as Focal Loss, Balanced SoftMax and Distribution Alignment. Experiments demonstrate that the performance of VLMs can be further boosted when used with decoder and imbalanced methods. Specifically, our improved VLMs significantly outperforms zero-shot classification by an average accuracy of 6.58%, 69.82%, and 6.17%, on ImageNet-LT, iNaturalist18, and Places-LT, respectively. We further analyze the influence of pre-training data size, backbones, and training cost. Our study highlights the significance of imbalanced learning algorithms in face of VLMs pre-trained by huge data. We release our code at https://github.com/Imbalance-VLM/Imbalance-VLM.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 52 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

36 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  9. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  10. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  11. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  12. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
22 more
  1. 2020Long-tail learning via logit adjustmentcited by 605
  2. 2021Image Segmentation Using Text and Image Promptscited by 509
  3. 2019BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognitioncited by 403
  4. 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 397
  5. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
  6. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  7. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 250
  8. 2020Rethinking the Value of Labels for Improving Class-Imbalanced Learningcited by 216
  9. 2021Distribution Alignment: A Unified Framework for Long-tail Visual Recognitioncited by 198
  10. 2020Disentangling Label Distribution for Long-tailed Visual Recognitioncited by 172
  11. 2015Cost Sensitive Learning of Deep Feature Representations from Imbalanced Datacited by 167
  12. 2020Seesaw Loss for Long-Tailed Instance Segmentationcited by 146
  13. 2020Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspectivecited by 143
  14. 2021Contrastive Learning based Hybrid Networks for Long-Tailed Image Classificationcited by 130
  15. 2022A Survey on Long-Tailed Visual Recognitioncited by 75
  16. 2021VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognitioncited by 65
  17. 2021RSG: A Simple but Effective Module for Learning Imbalanced Datasetscited by 61
  18. 2021A Simple Long-Tailed Recognition Baseline via Vision-Language Modelcited by 49
  19. 2022Open-Sampling: Exploring Out-of-Distribution data for Re-balancing Long-tailed datasetscited by 25
  20. 2021Margin Calibration for Long-Tailed Visual Recognitioncited by 17
  21. 2022An Embarrassingly Simple Baseline for Imbalanced Semi-Supervised Learningcited by 12
  22. 2023Rethink Long-tailed Recognition with Vision Transformerscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 13; 2025: 13; 2026: 3.

35 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  3. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  4. 2023Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurtscited by 58
  5. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 51
  6. 2023Understanding and Mitigating the Label Noise in Pre-training on Downstream Taskscited by 27
  7. 2023A Unified Perspective for Loss-Oriented Imbalanced Learning via Localizationcited by 24
  8. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  9. 2024Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwardscited by 19
  10. 2024Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognitioncited by 15
  11. 2025Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classificationcited by 14
  12. 2025Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysiscited by 10
23 more
  1. 2023ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learningcited by 9
  2. 2024Learning from Neighbors: Category Extrapolation for Long-Tail Learningcited by 9
  3. 2025Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderationcited by 9
  4. 2024Impact of Noisy Supervision in Foundation Model Learningcited by 8
  5. 2024EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotypingcited by 6
  6. 2024ChimpVLM: Ethogram-Enhanced Chimpanzee Behaviour Recognitioncited by 5
  7. 2025LIFT+: Lightweight Fine-Tuning for Long-Tail Learningcited by 5
  8. 2025SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoderscited by 5
  9. 2024An Individual Identity-Driven Framework for Animal Re-Identificationcited by 3
  10. 2024VLLaVO: Mitigating Visual Gap through LLMscited by 2
  11. 2024Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Modelcited by 2
  12. 2025Enhancing Features in Long-tailed Data Using Large Vision Modelcited by 2
  13. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  14. 2025Rethinking the Bias of Foundation Model under Long-tailed Distributioncited by 1
  15. 2025LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributionscited by 1
  16. 2025LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soupscited by 1
  17. 2024Categorical Knowledge Fused Recognition: Fusing Hierarchical Knowledge with Image Classification through Aligning and Deep Metric Learningcited by 0
  18. 2025PI-H2T: Enhancing Long-Tailed Visual Recognition with Permutation-Invariant and Head-to-Tail Feature Fusioncited by 0
  19. 2025Semantic-guided Fine-tuning of Foundation Model for Long-tailed Visual Recognitioncited by 0
  20. 2025Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistencycited by 0
  21. 2025Efficient Long-Tail Learning in Latent Space by sampling Synthetic Datacited by 0
  22. 2026CUE: Concept-Aware Multi-Label Expansion to Mitigate Concept Confusion in Long-Tailed Learningcited by 0
  23. 2026Pay More Attention To Text In High-Resolution MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Image Recognition with Vision and Language Embeddings of VLMs
  2. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selection
  3. 2025Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin
  4. 2024Why are Visually-Grounded Language Models Bad at Image Classification?
  5. 2024The Neglected Tails in Vision-Language Models
  6. 2023Vision-Language Models for Vision Tasks: A Survey
  7. 2025Difference Vector Equalization for Robust Fine-tuning of Vision-Language Models
  8. 2025Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts