mlsanityData to 7 Oct 2026

Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts

arXiv 2309.10019 ↗First posted 18 Sep 2023cs.CV cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

The fine-tuning paradigm in addressing long-tail learning tasks has sparked significant interest since the emergence of foundation models. Nonetheless, how fine-tuning impacts performance in long-tail learning was not explicitly quantified. In this paper, we disclose that heavy fine-tuning may even lead to non-negligible performance deterioration on tail classes, and lightweight fine-tuning is more effective. The reason is attributed to inconsistent class conditions caused by heavy fine-tuning. With the observation above, we develop a low-complexity and accurate long-tail learning algorithms LIFT with the goal of facilitating fast prediction and compact models by adaptive lightweight fine-tuning. Experiments clearly verify that both the training time and the learned parameters are significantly reduced with more accurate predictive performance compared with state-of-the-art approaches. The implementation code is available at https://github.com/shijxcs/LIFT.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 53 of its 71 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

53 of 71 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2016Layer Normalizationcited by 5,579
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2022Visual Prompt Tuningcited by 1,674
  10. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  11. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
38 more
  1. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  2. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  3. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  4. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  5. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  6. 2020Long-tail learning via logit adjustmentcited by 605
  7. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  8. 2019BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognitioncited by 403
  9. 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 397
  10. 2022Visual Classification via Description from Large Language Modelscited by 331
  11. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  12. 2020Long-tailed Recognition by Routing Diverse Distribution-Aware Expertscited by 270
  13. 2021Parametric Contrastive Learningcited by 217
  14. 2020Rethinking the Value of Labels for Improving Class-Imbalanced Learningcited by 216
  15. 2021Improving Calibration for Long-Tailed Recognitioncited by 212
  16. 2021Distribution Alignment: A Unified Framework for Long-tail Visual Recognitioncited by 198
  17. 2020Disentangling Label Distribution for Long-tailed Visual Recognitioncited by 172
  18. 2022Balanced Contrastive Learning for Long-Tailed Visual Recognitioncited by 143
  19. 2021Contrastive Learning based Hybrid Networks for Long-Tailed Image Classificationcited by 130
  20. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  21. 2021The Majority Can Help The Minority: Context-rich Minority Oversampling for Long-tailed Classificationcited by 100
  22. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  23. 2021Distributional Robustness Loss for Long-tail Learningcited by 84
  24. 2023Long-tailed Visual Recognition via Gaussian Clouded Logit Adjustmentcited by 75
  25. 2021Distilling Virtual Examples for Long-tailed Recognitioncited by 72
  26. 2022LPT: Long-tailed Prompt Tuning for Image Classificationcited by 69
  27. 2021VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognitioncited by 65
  28. 2022Nested Collaborative Learning for Long-Tailed Visual Recognitioncited by 63
  29. 2021A Simple Long-Tailed Recognition Baseline via Vision-Language Modelcited by 49
  30. 2023Exploring Vision-Language Models for Imbalanced Learningcited by 35
  31. 2022Learning Imbalanced Data with Vision Transformerscited by 34
  32. 2023Visual Tuningcited by 34
  33. 2021Adaptive Logit Adjustment Loss for Long-Tailed Visual Recognitioncited by 33
  34. 2023CUDA: Curriculum of Data Augmentation for Long-Tailed Recognitioncited by 30
  35. 2023Improving Image Recognition by Retrieving from Web-Scale Image-Text Datacited by 20
  36. 2023Long-Tailed Recognition by Mutual Information Maximization between Latent Features and Ground-Truth Labelscited by 16
  37. 2023LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-tailed Multi-Label Visual Recognitioncited by 15
  38. 2023Decoupled Training for Long-Tailed Classification With Stochastic Representationscited by 13

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 32; 2026: 12.

58 papers.

Sort:

By area:

  1. 2023A Unified Perspective for Loss-Oriented Imbalanced Learning via Localizationcited by 24
  2. 2024Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learningcited by 19
  3. 2024Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwardscited by 19
  4. 2024Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognitioncited by 15
  5. 2025Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Datacited by 15
  6. 2024DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detectioncited by 14
  7. 2025Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classificationcited by 14
  8. 2025LADA: Scalable Label-Specific CLIP Adapter for Continual Learningcited by 11
  9. 2024Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognitioncited by 10
  10. 2025LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answeringcited by 10
  11. 2025Long-tailed Recognition with Model Rebalancingcited by 10
  12. 2024Learning from Neighbors: Category Extrapolation for Long-Tail Learningcited by 9
38 more
  1. 2024Difficulty-aware Balancing Margin Loss for Long-tailed Recognitioncited by 9
  2. 2025Improved Balanced Classification with Theoretically Grounded Loss Functionscited by 9
  3. 2024Adaptive Adapter Routing for Long-Tailed Class-Incremental Learningcited by 8
  4. 2025LIFT+: Lightweight Fine-Tuning for Long-Tail Learningcited by 5
  5. 2024Long-Tailed Out-of-Distribution Detection via Normalized Outlier Distribution Adaptationcited by 4
  6. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Modelscited by 4
  7. 2024DiffuLT: How to Make Diffusion Model Useful for Long-tail Recognitioncited by 3
  8. 2025Unlabeled Data vs. Pre-trained Knowledge: Rethinking SSL in the Era of Large Modelscited by 3
  9. 2025Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imagingcited by 3
  10. 2024CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learningcited by 2
  11. 2025Enhancing Features in Long-tailed Data Using Large Vision Modelcited by 2
  12. 2025DeepAndes: A Self-Supervised Vision Foundation Model for Multi-Spectral Remote Sensing Imagery of the Andescited by 2
  13. 2025Tuning Language Models for Robust Prediction of Diverse User Behaviorscited by 2
  14. 2025AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Promptingcited by 2
  15. 2025Tuning the Right Foundation Models is What you Need for Partial Label Learningcited by 2
  16. 2025Dual-View Alignment Learning with Hierarchical-Prompt for Class-Imbalance Multi-Label Classificationcited by 2
  17. 2026Optimized Deferral for Imbalanced Settingscited by 2
  18. 2024LPT++: Efficient Training on Mixture of Long-tailed Expertscited by 1
  19. 2025Rethinking the Bias of Foundation Model under Long-tailed Distributioncited by 1
  20. 2025Multi-label classification for multi-temporal, multi-spatial coral reef condition monitoring using vision foundation model with adapter learningcited by 1
  21. 2025Subtyping Breast Lesions via Generative Augmentation based Long-tailed Recognition in Ultrasoundcited by 1
  22. 2025LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soupscited by 1
  23. 2025Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognitioncited by 1
  24. 2026Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discoverycited by 1
  25. 2026CARE: Class-Adaptive Expert Consensus for Reliable Learning with Long-Tailed Noisy Labelscited by 1
  26. 2024I-MPN: Inductive Message Passing Network for Efficient Human-in-the-Loop Annotation of Mobile Eye Tracking Datacited by 0
  27. 2025ULFine: Unbiased Lightweight Fine-tuning for Foundation-Model-Assisted Long-Tailed Semi-Supervised Learningcited by 0
  28. 2025PI-H2T: Enhancing Long-Tailed Visual Recognition with Permutation-Invariant and Head-to-Tail Feature Fusioncited by 0
  29. 2025Long-Tailed Learning for Generalized Category Discoverycited by 0
  30. 2025Unsupervised Contrastive Learning Using Out-Of-Distribution Data for Long-Tailed Datasetcited by 0
  31. 2025Semantic-guided Fine-tuning of Foundation Model for Long-tailed Visual Recognitioncited by 0
  32. 2025ViRN: Variational Inference and Distribution Trilateration for Long-Tailed Continual Representation Learningcited by 0
  33. 2025Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysiscited by 0
  34. 2025LoFT: Parameter-Efficient Fine-Tuning for Long-tailed Semi-Supervised Learning in Open-World Scenarioscited by 0
  35. 2025Efficient Long-Tail Learning in Latent Space by sampling Synthetic Datacited by 0
  36. 2025Revisiting Mixout: An Overlooked Path to Robust Finetuningcited by 0
  37. 2025How (Mis)calibrated is your Federated CLIP and what to do about it?cited by 0
  38. 2026Semantics-Aware Generative Latent Data Augmentation for Learning in Low-Resource Domainscited by 0

50 most cited of 58 All 58 (JSON)