mlsanityData to 14 Oct 2026

AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning

arXiv 2404.08958 ↗First posted 13 Apr 2024cs.CV cs.CL cs.LGcited by 22

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recently, pre-trained vision-language models (e.g., CLIP) have shown great potential in few-shot learning and attracted a lot of research interest. Although efforts have been made to improve few-shot ability of CLIP, key factors on the effectiveness of existing methods have not been well studied, limiting further exploration of CLIP's potential in few-shot learning. In this paper, we first introduce a unified formulation to analyze CLIP-based few-shot learning methods from a perspective of logit bias, which encourages us to learn an effective logit bias for further improving performance of CLIP-based few-shot learning methods. To this end, we disassemble three key components involved in computation of logit bias (i.e., logit features, logit predictor, and logit fusion) and empirically analyze the effect on performance of few-shot classification. Based on analysis of key components, this paper proposes a novel AMU-Tuning method to learn effective logit bias for CLIP-based few-shot classification. Specifically, our AMU-Tuning predicts logit bias by exploiting the appropriate underlineAuxiliary features, which are fed into an efficient feature-initialized linear classifier with underlineMulti-branch training. Finally, an underlineUncertainty-based fusion is developed to incorporate logit bias into CLIP for few-shot classification. The experiments are conducted on several widely used benchmarks, and the results show AMU-Tuning clearly outperforms its counterparts while achieving state-of-the-art performance of CLIP-based few-shot learning without bells and whistles.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 68 references that are in the map. First posted on arXiv 13 Apr 2024.

Builds on

53 of 68 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  2. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  3. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts4 Dec
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
34 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  12. 2019Natural Adversarial Examplescited by 1,408
  13. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  14. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  15. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  16. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  17. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  18. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  19. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  20. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  21. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  22. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  23. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  24. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  25. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  26. 2019Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processescited by 131
  27. 2019Improved Few-Shot Visual Classificationcited by 115
  28. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  29. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  30. 2023Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modelingcited by 94
  31. 2020Embedding Propagation: Smoother Manifold for Few-Shot Classificationcited by 80
  32. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  33. 2020Enhancing Few-Shot Image Classification with Unlabelled Examplescited by 32
  34. 2022Confidence estimation of classification based on the distribution of the neural network output layercited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 5.

22 papers.

Sort:

By area:

  1. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuningcited by 15
  2. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classificationcited by 13
  3. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
  4. 2026Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learningcited by 5
  5. 2026Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learningcited by 4
  6. 2026Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignmentcited by 4
  7. 2024HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adaptercited by 3
  8. 2024COBRA: COmBinatorial Retrieval Augmentation for Few-Shot Adaptationcited by 2
  9. 2025Causal Disentanglement and Cross-Modal Alignment for Enhanced Few-Shot Learningcited by 2
  10. 2024CLIP's Visual Embedding Projector is a Few-shot Cornucopiacited by 1
  11. 2024Cross-Modal Mapping: Mitigating the Modality Gap for Few-Shot Image Classificationcited by 1
  12. 2025LoFT: LoRA-fused Training Dataset Generation with Few-shot Guidancecited by 1
10 more
  1. 2025Beyond Graph Model: Reliable VLM Fine-Tuning via Random Graph Adaptercited by 1
  2. 2026What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Featurescited by 1
  3. 2024Enhancing Few-Shot Out-of-Distribution Detection with Gradient Aligned Context Optimizationcited by 0
  4. 2024CRoF: CLIP-based Robust Few-shot Learning on Noisy Labelscited by 0
  5. 2025AutoExpert: Automating 3D LiDAR Annotation from Expert-Crafted Guidelinescited by 0
  6. 2025Integrated Structural Prompt Learning for Vision-Language Modelscited by 0
  7. 2025Shallow Deep Learning Can Still Excel in Fine-Grained Few-Shot Learningcited by 0
  8. 2025Color as the Impetus: Transforming Few-Shot Learnercited by 0
  9. 2025Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learningcited by 0
  10. 2026Proximity-CLIP: Text-Guided Semantic Proximity Learning for Zero-Shot Anomaly Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting
  2. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
  3. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  4. 2023Proto-CLIP: Vision-Language Prototypical Network for Few-Shot Learning
  5. 2024Near OOD Detection for Vision-Language Prompt Learning with Contrastive Logit Score
  6. 2024What Makes Good Few-shot Examples for Vision-Language Models?
  7. 2025Generalizable Prompt Learning of CLIP: A Brief Overview
  8. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models