mlsanityData to 7 Oct 2026

Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models

arXiv 2407.02716 ↗First posted 2 Jul 2024cs.CV cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Fine-tuning pre-trained Vision-Language Models (VLMs) has shown remarkable capabilities in medical image and textual depiction synergy. Nevertheless, many pre-training datasets are restricted by patient privacy concerns, potentially containing noise that can adversely affect downstream performance. Moreover, the growing reliance on multi-modal generation exacerbates this issue because of its susceptibility to adversarial attacks. To investigate how VLMs trained on adversarial noisy data perform on downstream medical tasks, we first craft noisy upstream datasets using multi-modal adversarial attacks. Through our comprehensive analysis, we unveil that moderate noise enhances model robustness and transferability, but increasing noise levels negatively impact downstream task performance. To mitigate this issue, we propose rectify adversarial noise (RAN) framework, a recipe designed to effectively defend adversarial attacks and rectify the influence of upstream noise during fine-tuning.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 62 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

40 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective30 Apr
  2. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks6 Dec

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  8. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  9. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  10. 2019Adversarial Training for Free!cited by 638
  11. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  12. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
26 more
  1. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
  2. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  3. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  4. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  5. 2023Multimodal Deep Learningcited by 257
  6. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  7. 2019Are Anchor Points Really Indispensable in Label-Noise Learning?cited by 208
  8. 2020Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learningcited by 142
  9. 2020To be Robust or to be Fair: Towards Fairness in Adversarial Trainingcited by 137
  10. 2023Machine Learning for Synthetic Data Generation: A Reviewcited by 118
  11. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 111
  12. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  13. 2022Guided Diffusion Model for Adversarial Purificationcited by 90
  14. 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71
  15. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  16. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  17. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
  18. 2024Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Imagescited by 41
  19. 2023Understanding and Mitigating the Label Noise in Pre-training on Downstream Taskscited by 27
  20. 2022Robust Medical Image Classification from Noisy Labeled Data with Global and Local Representation Guided Co-trainingcited by 21
  21. 2024Understanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thoughtcited by 19
  22. 2018Hyperspectral Image Classification in the Presence of Noisy Labelscited by 12
  23. 2024Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imagingcited by 9
  24. 2022Unsupervised Domain Adaptation for Segmentation with Black-box Source Modelcited by 7
  25. 2021Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Searchcited by 3
  26. 2023Generating Medical Prescriptions with Conditional Transformercited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024Progressive Compositionality in Text-to-Image Generative Modelscited by 11
  2. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  3. 2025Mixture-of-Shape-Experts (MoSE): End-to-End Shape Dictionary Framework to Prompt SAM for Generalizable Medical Segmentationcited by 4
  4. 2025VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026A Mechanistic Analysis of Adversarial Fine-tuning of Vision Transformers
  2. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks
  3. 2025Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation
  4. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models
  5. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
  6. 2025Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models
  7. 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learning
  8. 2025A Framework for Evaluating Vision-Language Model Safety: Building Trust in AI for Public Sector Applications