mlsanityData to 14 Oct 2026

On the Reliability of Vision-Language Models Under Adversarial Frequency-Domain Perturbations

arXiv 2507.22398 ↗First posted 30 Jul 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-Language Models (VLMs) are increasingly used as perceptual modules for visual content reasoning, including through captioning and DeepFake detection. In this work, we expose a critical vulnerability of VLMs when exposed to subtle, structured perturbations in the frequency domain. Specifically, we highlight how these feature transformations undermine authenticity/DeepFake detection and automated image captioning tasks. We design targeted image transformations, operating in the frequency domain to systematically adjust VLM outputs when exposed to frequency-perturbed real and synthetic images. We demonstrate that the perturbation injection method generalizes across five state-of-the-art VLMs which includes different-parameter Qwen2/2.5 and BLIP models. Experimenting across ten real and generated image datasets reveals that VLM judgments are sensitive to frequency-based cues and may not wholly align with semantic content. Crucially, we show that visually-imperceptible spatial frequency transformations expose the fragility of VLMs deployed for automated image captioning and authenticity detection tasks. Our findings under realistic, black-box constraints challenge the reliability of VLMs, underscoring the need for robust multimodal perception systems.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 74 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

51 of 74 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models16 Apr
  2. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructions14 Mar
  3. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2020Denoising Diffusion Implicit Modelscited by 9,781
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  11. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
36 more
  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  13. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  14. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  15. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,101
  16. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  17. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  18. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  19. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  20. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  21. 2020Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributionscited by 280
  22. 2020Media Forensics and DeepFakes: an overviewcited by 209
  23. 2018Learning Rich Features for Image Manipulation Detectioncited by 154
  24. 2019Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolutioncited by 140
  25. 2023CIFAKE: Image Classification and Explainable Identification of AI-Generated Synthetic Imagescited by 84
  26. 2022Frequency-driven Imperceptible Adversarial Attack on Semantic Similaritycited by 77
  27. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 75
  28. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 72
  29. 2023AdvDiff: Generating Unrestricted Adversarial Examples using Diffusion Modelscited by 46
  30. 2025Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024cited by 45
  31. 2023Mutual-modality Adversarial Attack with Semantic Perturbationcited by 14
  32. 2023Overload: Latency Attacks on Object Detection for Edge Devicescited by 13
  33. 2023Detecting Images Generated by Diffuserscited by 12
  34. 2024FACL-Attack: Frequency-Aware Contrastive Learning for Transferable Adversarial Attackscited by 8
  35. 2025SITA: Structurally Imperceptible and Transferable Adversarial Attacks for Stylized Image Generationcited by 6
  36. 2025LLMs Are Not Yet Ready for Deepfake Image Detectioncited by 4

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Visually Prompted Benchmarks Are Surprisingly Fragilecited by 7
  2. 2025When Harmful Content Gets Camouflaged: Unveiling Perception Failure of LVLMs with CamHarmTIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  2. 2026Same Answer, Different Representations: Hidden instability in VLMs
  3. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
  4. 2026Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
  5. 2026It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at ε≤ 4/255
  6. 2024Harnessing the Power of Large Vision Language Models for Synthetic Image Detection
  7. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
  8. 2024MVTamperBench: Evaluating Robustness of Vision-Language Models