mlsanityData to 14 Oct 2026

GroupCoOp: Group-robust Fine-tuning via Group Prompt Learning

arXiv 2509.23781 ↗First posted 28 Sep 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) of vision-language models (VLMs) excels in various vision tasks thanks to the rich knowledge and generalization ability of VLMs. However, recent studies revealed that such fine-tuned VLMs are vulnerable to spurious correlations stemming from the subgroup imbalance in the fine-tuning datasets. To resolve this issue, we propose Group Context Optimization (GroupCoOp), a simple and effective debiased fine-tuning algorithm that enhances the group robustness of fine-tuned VLMs. Its key idea is to employ group-specific text prompts as group representatives serving as multiple classifiers for their target class. The rich semantic knowledge of the text encoder of VLM enables the discovery of effective group prompts even for groups with a small number of training samples. Leveraging the group prompts for each class addresses the issues caused by the group-imbalanced training set, such as the neglect of minority groups and the scattered distribution of each class in the embedding space. GroupCoOp achieved the best results on five benchmarks across five CLIP architectures and occasionally outperformed prior methods that fine-tune the entire network, despite training only 0.016% of the network's parameters.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 62 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

57 of 62 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs22 Mar
  2. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations12 Mar
  3. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  3. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  5. 2019Invariant Risk Minimizationcited by 1,854
  6. 2022Visual Prompt Tuningcited by 1,681
  7. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  8. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  9. 2021Learning to Prompt for Continual Learningcited by 944
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  11. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  12. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  13. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  14. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  15. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  16. 2017Learning multiple visual domains with residual adapterscited by 557
  17. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  18. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  19. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  20. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  21. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  22. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  23. 2019Fixing the train-test resolution discrepancycited by 209
  24. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  25. 2019Learning De-biased Representations with Biased Representationscited by 199
  26. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 172
  27. 2023Debiasing Vision-Language Models via Biased Promptscited by 149
  28. 2020Learning from Failure: Training Debiased Classifier from Biased Classifiercited by 133
  29. 2021Learning Debiased Representation via Disentangled Feature Augmentationcited by 116
  30. 2021EnD: Entangling and Disentangling deep representations for bias correctioncited by 105
  31. 2022Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimationcited by 88
  32. 2022MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflictscited by 81
  33. 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73
  34. 2023Discover and Cure: Concept-aware Mitigation of Spurious Correlationcited by 71
  35. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
  36. 2022Learning Debiased Classifier with Biased Committeecited by 51
  37. 2023Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningcited by 47
  38. 2021Learning Fair Classifiers with Partially Annotated Group Labelscited by 40

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models
  2. 2025Debiased Prompt Tuning in Vision-Language Model without Annotations
  3. 2024Retaining and Enhancing Pre-trained Knowledge in Vision-Language Models with Prompt Ensembling
  4. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  5. 2026Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models
  6. 2026Fine-tuning Pre-trained Vision-Language Models in a Human-Annotation-Free Manner
  7. 2023Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability
  8. 2023Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models