mlsanityData to 14 Oct 2026

Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models

arXiv 2505.24227 ↗First posted 30 May 2025cs.CV cs.CRcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

While adversarial attacks on vision-and-language pretraining (VLP) models have been explored, generating natural adversarial samples crafted through realistic and semantically meaningful perturbations remains an open challenge. Existing methods, primarily designed for classification tasks, struggle when adapted to VLP models due to their restricted optimization spaces, leading to ineffective attacks or unnatural artifacts. To address this, we propose LightD, a novel framework that generates natural adversarial samples for VLP models via semantically guided relighting. Specifically, LightD leverages ChatGPT to propose context-aware initial lighting parameters and integrates a pretrained relighting model (IC-light) to enable diverse lighting adjustments. LightD expands the optimization space while ensuring perturbations align with scene semantics. Additionally, gradient-based optimization is applied to the reference lighting image to further enhance attack effectiveness while maintaining visual naturalness. The effectiveness and superiority of the proposed LightD have been demonstrated across various VLP models in tasks such as image captioning and visual question answering.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 66 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

56 of 66 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models30 May
  2. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory19 Mar
  3. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  4. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization7 Dec
  5. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  6. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
38 more
  1. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  2. 2021Intriguing Properties of Vision Transformerscited by 446
  3. 2018Feature Denoising for Improving Adversarial Robustnesscited by 314
  4. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  5. 2017Adversarial Examples for Semantic Segmentation and Object Detectioncited by 275
  6. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  7. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  8. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  9. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  10. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  11. 2014From Captions to Visual Concepts and Backcited by 109
  12. 2019ColorFool: Semantic Adversarial Colorizationcited by 89
  13. 2018Semantic Adversarial Examplescited by 86
  14. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
  15. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 75
  16. 2015Where To Look: Focus Regions for Visual Question Answeringcited by 64
  17. 2018Motivating the Rules of the Game for Adversarial Example Researchcited by 57
  18. 2019Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifierscited by 57
  19. 2021Human-Adversarial Visual Question Answeringcited by 57
  20. 2016Visual Question Answering: Datasets, Algorithms, and Future Challengescited by 56
  21. 2015Ask Your Neurons: A Neural-based Approach to Answering Questions about Imagescited by 51
  22. 2021Towards General Purpose Vision Systemscited by 46
  23. 2023A Survey on Transferability of Adversarial Examples across Deep Neural Networkscited by 44
  24. 2023Improving Fast Adversarial Training with Prior-Guided Knowledgecited by 35
  25. 2020Improving Query Efficiency of Black-box Adversarial Attackcited by 34
  26. 2019Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variablescited by 26
  27. 2021Multimodal Integration of Human-Like Attention in Visual Question Answeringcited by 20
  28. 2021Adversarial Relighting Against Face Recognitioncited by 16
  29. 2019Adaptively Aligned Image Captioning via Adaptive Attention Timecited by 15
  30. 2022TASA: Deceiving Question Answering Models by Twin Answer Sentences Attackcited by 15
  31. 2020Can You Spot the Chameleon? Adversarially Camouflaging Images from Co-Salient Object Detectioncited by 14
  32. 2019EdgeFool: An Adversarial Image Enhancement Filtercited by 10
  33. 2024Hard-label based Small Query Black-box Adversarial Attackcited by 10
  34. 2020Adversarial Image Color Transformations in Explicit Color Filter Spacecited by 9
  35. 2021Controlled Caption Generation for Images Through Adversarial Attackscited by 9
  36. 2023Adversarial Attacks on Video Object Segmentation with Hard Region Discoverycited by 9
  37. 2020Semantically Adversarial Learnable Filterscited by 5
  38. 2024AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When Lighting Deceives: Exposing Vision-Language Models' Illumination Vulnerability Through Illumination Transformation Attack
  2. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  3. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
  4. 2026Lights, Camera, Malfunction: When Illumination Robustness Leaves VLA Models Blind to Color
  5. 2026LAMP: Learning Universal Adversarial Perturbations for Multi-Image Tasks via Pre-trained Models
  6. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  7. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models
  8. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach