mlsanityData to 7 Oct 2026

Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation

arXiv 2404.13425 ↗First posted 20 Apr 2024cs.CV cs.AIcited by 8

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-Language Models (VLMs) play a crucial role in the advancement of Artificial General Intelligence (AGI). As AGI rapidly evolves, addressing security concerns has emerged as one of the most significant challenges for VLMs. In this paper, we present extensive experiments that expose the vulnerabilities of conventional adaptation methods for VLMs, highlighting significant security risks. Moreover, as VLMs grow in size, the application of traditional adversarial adaptation techniques incurs substantial computational costs. To address these issues, we propose a parameter-efficient adversarial adaptation method called textbftextitAdvLoRA based on Low-Rank Adaptation. We investigate and reveal the inherent low-rank properties involved in adversarial adaptation for VLMs. Different from LoRA, we enhance the efficiency and robustness of adversarial adaptation by introducing a novel reparameterization method that leverages parameter clustering and alignment. Additionally, we propose an adaptive parameter update strategy to further bolster robustness. These innovations enable our AdvLoRA to mitigate issues related to model security and resource wastage. Extensive experiments confirm the effectiveness and efficiency of AdvLoRA.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 67 references that are in the map. First posted on arXiv 20 Apr 2024.

Builds on

33 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 32

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2022Visual Prompt Tuningcited by 1,674
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  9. 2016Adversarial Machine Learning at Scalecited by 934
  10. 2021Robust fine-tuning of zero-shot modelscited by 785
  11. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  12. 2017Localizing Moments in Video with Natural Languagecited by 660
20 more
  1. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  2. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  3. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  4. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  5. 2017On Detecting Adversarial Perturbationscited by 232
  6. 2020Bag of Tricks for Adversarial Trainingcited by 199
  7. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  8. 2019Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systemscited by 151
  9. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  10. 2020Adversarial Training for Large Neural Language Modelscited by 109
  11. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  12. 2022On Adversarial Robustness of Trajectory Prediction for Autonomous Vehiclescited by 89
  13. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  14. 2024Multi-LoRA Composition for Image Generationcited by 80
  15. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  16. 2018Sparse Adversarial Perturbations for Videoscited by 56
  17. 2019Growing a Brain: Fine-Tuning by Increasing Model Capacitycited by 43
  18. 2024LoRA Meets Dropout under a Unified Frameworkcited by 17
  19. 2024FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformerscited by 9
  20. 2021Re-identification of Individuals in Genomic Datasets Using Public Face Imagescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 1.

8 papers.

  1. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 92
  2. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  3. 2025A Survey on Efficient Vision-Language Modelscited by 19
  4. 2025Scaling Up AI-Generated Image Detection with Generator-Aware Prototypescited by 8
  5. 2025Confidence Elicitation: A New Attack Vector for Large Language Modelscited by 4
  6. 2025Vision-EKIPL: External Knowledge-Infused Policy Learning for Visual Reasoningcited by 4
  7. 2025DAC-LoRA: Dynamic Adversarial Curriculum for Efficient and Robust Few-Shot Adaptationcited by 1
  8. 2026SARFusion: Scene-Aware Routing Fusion for Robust Camera-LiDAR 3D Object Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
  2. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  3. 2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models
  4. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning
  5. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  6. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
  7. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  8. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models