mlsanityData to 7 Oct 2026

Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models

arXiv 2306.02080 ↗First posted 3 Jun 2023cs.CV cs.CL cs.LGcited by 30

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Various adaptation methods, such as LoRA, prompts, and adapters, have been proposed to enhance the performance of pre-trained vision-language models in specific domains. The robustness of these adaptation methods against distribution shifts have not been studied. In this study, we assess the robustness of 11 widely-used adaptation methods across 4 vision-language datasets under multimodal corruptions. Concretely, we introduce 7 benchmark datasets, including 96 visual and 87 textual corruptions, to investigate the robustness of different adaptation methods, the impact of available adaptation examples, and the influence of trainable parameter size during adaptation. Our analysis reveals that: 1) Adaptation methods are more sensitive to text corruptions than visual corruptions. 2) Full fine-tuning does not consistently provide the highest robustness; instead, adapters can achieve better robustness with comparable clean performance. 3) Contrary to expectations, our findings indicate that increasing the number of adaptation data and parameters does not guarantee enhanced robustness; instead it results in even lower robustness. We hope this study could benefit future research in the development of robust multimodal adaptation methods. The benchmark, code, and dataset used in this study can be accessed at https://adarobustness.github.io .

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 80 references that are in the map. First posted on arXiv 3 Jun 2023.

Builds on

65 of 80 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022A Survey of Vision-Language Pre-Trained Models18 Feb
  3. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov

Other references · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2015Fast R-CNNcited by 2,930
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  11. 2019Natural Adversarial Examplescited by 1,401
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  13. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  14. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  15. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  17. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  18. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  19. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  20. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  21. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  22. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  23. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  24. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  25. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  26. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
  27. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  28. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  29. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  30. 2018Generalisation in humans and deep neural networkscited by 237
  31. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  32. 2021Vision Transformers are Robust Learnerscited by 208
  33. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  34. 2019Benchmarking the Robustness of Semantic Segmentation Modelscited by 111
  35. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  36. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  37. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  38. 2022SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustnesscited by 74

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 13; 2026: 5.

30 papers.

Sort:

By area:

  1. 2023A Survey on Transferability of Adversarial Examples across Deep Neural Networkscited by 43
  2. 2024Perturbation-Restrained Sequential Model Editingcited by 27
  3. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructionscited by 26
  4. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarioscited by 19
  5. 2024Which Model Generated This Image? A Model-Agnostic Approach for Origin Attributioncited by 15
  6. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  7. 2025Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Expertscited by 13
  8. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  9. 2025FAA-CLIP: Federated Adversarial Adaptation of CLIPcited by 5
  10. 2025FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answeringcited by 5
  11. 2024Evaluating Durability: Benchmark Insights into Multimodal Watermarkingcited by 3
  12. 2025Serial Low-rank Adaptation of Vision Transformercited by 3
18 more
  1. 2025PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Modelscited by 3
  2. 2024Automatic benchmarking of large multimodal models via iterative experiment programmingcited by 2
  3. 2025textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Timecited by 2
  4. 2025ColorBlindnessEval: Can Vision-Language Models Pass Color Blindness Tests?cited by 2
  5. 2025Deep Research Agents Brings Deeper Harmcited by 2
  6. 2025VLM-SlideEval: Evaluating VLMs on Structured Comprehension and Perturbation Sensitivity in PPTcited by 2
  7. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observationcited by 2
  8. 2023Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attackscited by 1
  9. 2024An Empirical Study of Parameter Efficient Fine-tuning on Vision-Language Pre-train Modelcited by 1
  10. 2024Benchmarking Vision-Language Contrastive Methods for Medical Representation Learningcited by 1
  11. 2024SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Taskscited by 1
  12. 2025A Novel Trustworthy Video Summarization Algorithm Through a Mixture of LoRA Expertscited by 0
  13. 2025SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Taskscited by 0
  14. 2025Privacy Preserving Ordinal-Meta Learning with VLMs for Fine-Grained Fruit Quality Predictioncited by 0
  15. 2026Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Methodcited by 0
  16. 2026Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Modelscited by 0
  17. 2026Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferabilitycited by 0
  18. 2026Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Taskcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditions
  2. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
  3. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  4. 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Models
  5. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptation
  6. 2025Realistic Test-Time Adaptation of Vision-Language Models
  7. 2026To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models
  8. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models