mlsanityData to 7 Oct 2026

Debiased Fine-Tuning for Vision-language Models by Prompt Regularization

arXiv 2301.12429 ↗First posted 29 Jan 2023cs.CVcited by 20

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

We present a new paradigm for fine-tuning large-scale visionlanguage pre-trained models on downstream task, dubbed Prompt Regularization (ProReg). Different from traditional fine-tuning which easily overfits to the downstream task data, ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning. The motivation is: by prompting the large model "a photo of a [CLASS]", the fil-lin answer is only dependent on the pretraining encyclopedic knowledge while independent of the task data distribution, which is usually biased. Specifically, given a training sample prediction during fine-tuning, we first calculate its KullbackLeibler loss of the prompt prediction and Cross-Entropy loss of the ground-truth label, and then combine them with a proposed sample-wise adaptive trade-off weight, which automatically adjusts the transfer between the pretrained and downstream domains. On various out-of-distribution benchmarks, we show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 45 references that are in the map. First posted on arXiv 29 Jan 2023.

Builds on

38 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  2. 2022Prompt Distribution Learning6 May
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models24 Sep
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017Mask R-CNNcited by 7,259
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
20 more
  1. 2017Adversarial Discriminative Domain Adaptationcited by 1,309
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  4. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2021GPT Understands, Toocited by 715
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  8. 2020Long-tail learning via logit adjustmentcited by 605
  9. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  10. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  11. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  12. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  13. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  14. 2018Characterizing and Avoiding Negative Transfercited by 226
  15. 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 183
  16. 2020Interventional Few-Shot Learningcited by 112
  17. 2021Causal Attention for Vision-Language Taskscited by 111
  18. 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 104
  19. 2020StyleGAN2 Distillation for Feed-forward Image Manipulationcited by 78
  20. 2019Self-Critical Reasoning for Robust Visual Question Answeringcited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 10; 2025: 3; 2026: 3.

20 papers.

Sort:

By area:

  1. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  2. 2024Balancing Act: Distribution-Guided Debiasing in Diffusion Modelscited by 50
  3. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  4. 2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
  5. 2024Identifying Implicit Social Biases in Vision-Language Modelscited by 27
  6. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
  7. 2024Selective Vision-Language Subspace Projection for Few-shot CLIPcited by 18
  8. 2024BendVLM: Test-Time Debiasing of Vision-Language Embeddingscited by 17
  9. 2024Joint Vision-Language Social Bias Removal for CLIPcited by 17
  10. 2023Compositional Zero-shot Learning via Progressive Language-based Observationscited by 12
  11. 2023Improved Visual Fine-tuning with Natural Language Supervisioncited by 7
  12. 2024Robust Fine-tuning of Zero-shot Models via Variance Reductioncited by 6
8 more
  1. 2025Generalized Kullback-Leibler Divergence Losscited by 6
  2. 2025Mitigating Confounding in Speech-Based Dementia Detection through Weight Maskingcited by 3
  3. 2026Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theorycited by 2
  4. 2024Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancementcited by 1
  5. 2024Generalizable Prompt Tuning for Vision-Language Modelscited by 1
  6. 2025Debiased Prompt Tuning in Vision-Language Model without Annotationscited by 1
  7. 2026Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Modelscited by 0
  8. 2026Test-Time Hallucination Control in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompt Learning via Meta-Regularization
  2. 2024Revisiting Prompt Pretraining of Vision-Language Models
  3. 2025Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts
  4. 2025GLAD: Generalizable Tuning for Vision-Language Models
  5. 2025Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
  6. 2026Towards Calibrating Prompt Tuning of Vision-Language Models
  7. 2023Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models
  8. 2023Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability