mlsanityData to 7 Oct 2026

Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance

arXiv 2404.00860 ↗First posted 1 Apr 2024cs.LG cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large-scale contrastive vision-language pre-trained models provide the zero-shot model achieving competitive performance across a range of image classification tasks without requiring training on downstream data. Recent works have confirmed that while additional fine-tuning of the zero-shot model on the reference data results in enhanced downstream performance, it compromises the model's robustness against distribution shifts. Our investigation begins by examining the conditions required to achieve the goals of robust fine-tuning, employing descriptions based on feature distortion theory and joint energy-based models. Subsequently, we propose a novel robust fine-tuning algorithm, Lipsum-FT, that effectively utilizes the language modeling aspect of the vision-language pre-trained models. Extensive experiments conducted on distribution shift scenarios in DomainNet and ImageNet confirm the superiority of our proposed Lipsum-FT approach over existing robust fine-tuning methods.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 70 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

56 of 70 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Prompts14 Jul
  2. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
38 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  3. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2022Visual Prompt Tuningcited by 1,674
  6. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  12. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  13. 2014How transferable are features in deep neural networks?cited by 1,216
  14. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  15. 2021Ethical and social risks of harm from Language Modelscited by 957
  16. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  17. 2021Robust fine-tuning of zero-shot modelscited by 785
  18. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  19. 2013Domain Generalization via Invariant Feature Representationcited by 622
  20. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  21. 2015Towards Open Set Deep Networkscited by 533
  22. 2013Do Deep Nets Really Need to be Deep?cited by 466
  23. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  24. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  25. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  26. 2017Feature Generating Networks for Zero-Shot Learningcited by 257
  27. 2014Towards Open World Recognitioncited by 226
  28. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
  29. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  30. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  31. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  32. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  33. 2016A Theory of Generative ConvNetcited by 174
  34. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  35. 2019DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networkscited by 80
  36. 2020Rethinking the Hyperparameters for Fine-tuningcited by 69
  37. 2020Distance-Based Regularisation of Deep Networks for Fine-Tuningcited by 47
  38. 2023Trainable Projected Gradient Method for Robust Fine-tuningcited by 37

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 7.

14 papers.

Sort:

By area:

  1. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  2. 2024Robust Fine-tuning of Zero-shot Models via Variance Reductioncited by 6
  3. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  4. 2024Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Modelscited by 2
  5. 2025StarFT: Robust Fine-tuning of Zero-shot Models via Spuriosity Alignmentcited by 1
  6. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Modelscited by 1
  7. 2025Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representationscited by 0
  8. 2025Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performancecited by 0
  9. 2026Model soups need only one ingredientcited by 0
  10. 2026The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Modelscited by 0
  11. 2026TRACER: Persistent Regularization for Robust Multimodal Finetuningcited by 0
  12. 2026Domain Generalization via Text-Anchored Information Bottleneckcited by 0
2 more
  1. 2026Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMscited by 0
  2. 2026Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  2. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models
  3. 2026Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
  4. 2025Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models
  5. 2025Difference Vector Equalization for Robust Fine-tuning of Vision-Language Models
  6. 2023Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?
  7. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
  8. 2026Hierarchically Robust Zero-shot Vision-language Models