mlsanityData to 7 Oct 2026

ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agents

arXiv 2509.16645 ↗First posted 20 Sep 2025cs.CVcited by 19

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Vision-Language Models (VLMs), with their strong reasoning and planning capabilities, are widely used in embodied decision-making (EDM) tasks in embodied agents, such as autonomous driving and robotic manipulation. Recent research has increasingly explored adversarial attacks on VLMs to reveal their vulnerabilities. However, these attacks either rely on overly strong assumptions, requiring full knowledge of the victim VLM, which is impractical for attacking VLM-based agents, or exhibit limited effectiveness. The latter stems from disrupting most semantic information in the image, which leads to a misalignment between the perception and the task context defined by system prompts. This inconsistency interrupts the VLM's reasoning process, resulting in invalid outputs that fail to affect interactions in the physical world. To this end, we propose a fine-grained adversarial attack framework, ADVEDM, which modifies the VLM's perception of only a few key objects while preserving the semantics of the remaining regions. This attack effectively reduces conflicts with the task context, making VLMs output valid but incorrect decisions and affecting the actions of agents, thus posing a more substantial safety threat in the physical world. We design two variants of based on this framework, ADVEDM-R and ADVEDM-A, which respectively remove the semantics of a specific object from the image and add the semantics of a new object into the image. The experimental results in both general scenarios and EDM tasks demonstrate fine-grained control and excellent attack performance.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 52 of its 66 references that are in the map. First posted on arXiv 20 Sep 2025.

Builds on

52 of 66 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2024GPT-4o System Cardcited by 4,467
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2022Planning-oriented Autonomous Drivingcited by 1,068
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  4. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  5. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  6. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  7. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  8. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  9. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  10. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  11. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  12. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  13. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  14. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  15. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
  16. 2022Frequency Domain Model Augmentation for Adversarial Attackcited by 128
  17. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  18. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  19. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attackscited by 89
  20. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 88
  21. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  22. 2022Generated Faces in the Wild: Quantitative Comparison of Stable Diffusion, Midjourney and DALL-E 2cited by 55
  23. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  24. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 40
  25. 2023Downstream-agnostic Adversarial Examplescited by 32
  26. 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 32
  27. 2024DarkSAM: Fooling Segment Anything Model to Segment Nothingcited by 32
  28. 2024Exploring the Robustness of Decision-Level Through Adversarial Attacks on LLM-Based Embodied Modelscited by 31
  29. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Modelscited by 27
  30. 2024Transferable Adversarial Facial Images for Privacy Protectioncited by 26
  31. 2024Adversarial Robustness for Visual Grounding of Multimodal Large Language Modelscited by 23
  32. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  33. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 22
  34. 2024Securely Fine-tuning Pre-trained Encoders Against Adversarial Examplescited by 21
  35. 2024NumbOD: A Spatial-Frequency Fusion Attack Against Object Detectorscited by 17
  36. 2024PB-UAP: Hybrid Universal Adversarial Attack For Image Segmentationcited by 14
  37. 2025Test-Time Backdoor Detection for Object Detection Modelscited by 14
  38. 2024Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Featurecited by 13

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 15.

19 papers.

Sort:

By area:

  1. 2025Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2cited by 11
  2. 2026How Adversarial Environments Mislead Agentic AI?cited by 4
  3. 2025Adversarial Confusion Attack: Disrupting Multimodal Large Language Modelscited by 3
  4. 2026Erosion Attack for Adversarial Training to Enhance Semantic Segmentation Robustnesscited by 3
  5. 2026VLA-Hijack: A Transferable Patch Attack against Vision-Language-Action Models via Visual Proprioception Hijackingcited by 3
  6. 2025UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Modelscited by 2
  7. 2026Transferable Physical-World Adversarial Patches Against Pedestrian Detection Modelscited by 2
  8. 2026ZAYA1-VL-8B Technical Reportcited by 2
  9. 2025Dual-View Inference Attack: Machine Unlearning Amplifies Privacy Exposurecited by 1
  10. 2026Transferable Physical-World Adversarial Patches Against Object Detection in Autonomous Drivingcited by 1
  11. 2026PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysiscited by 1
  12. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
7 more
  1. 2026Revealing Interpretable Failure Modes of VLMscited by 0
  2. 2026Zamba2-VL Technical Reportcited by 0
  3. 2026VFACamou: View-Fused Adversarial Camouflage for Environment-Adaptive Physical Evasioncited by 0
  4. 2026REALM: A Unified Red-Teaming Benchmark for Physical-World VLMscited by 0
  5. 2026SafeBranch: Branch-Pair Safety Alignment for Embodied Agentscited by 0
  6. 2026Security and Privacy in Large-Model-Driven Embodied Agents: Attacks, Defenses, and Future Directionscited by 0
  7. 2026It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at ε≤ 4/255cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models
  2. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Driving
  3. 2025Adversarial Attacks on Robotic Vision Language Action Models
  4. 2025Model-agnostic Adversarial Attack and Defense for Vision-Language-Action Models
  5. 2026Exploiting Vulnerabilities: Universal Adversarial Attacks on Vision-Language-Action Models in Robotics
  6. 2026Hijacking Robots with a Piece of Paper: A Systematic Study of Physical Prompt Injection in VLM-Controlled Robots
  7. 2026Breaking the weakest link to evade vision language models
  8. 2025AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models