mlsanityData to 14 Oct 2026

Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusion

arXiv 2505.23266 ↗First posted 29 May 2025cs.CR cs.AI cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

We present Adversarial Object Fusion (AdvOF), a novel attack framework targeting vision-and-language navigation (VLN) agents in service-oriented environments by generating adversarial 3D objects. While foundational models like Large Language Models (LLMs) and Vision Language Models (VLMs) have enhanced service-oriented navigation systems through improved perception and decision-making, their integration introduces vulnerabilities in mission-critical service workflows. Existing adversarial attacks fail to address service computing contexts, where reliability and quality-of-service (QoS) are paramount. We utilize AdvOF to investigate and explore the impact of adversarial environments on the VLM-based perception module of VLN agents. In particular, AdvOF first precisely aggregates and aligns the victim object positions in both 2D and 3D space, defining and rendering adversarial objects. Then, we collaboratively optimize the adversarial object with regularization between the adversarial and victim object across physical properties and VLM perceptions. Through assigning importance weights to varying views, the optimization is processed stably and multi-viewedly by iterative fusions from local updates and justifications. Our extensive evaluations demonstrate AdvOF can effectively degrade agent performance under adversarial conditions while maintaining minimal interference with normal navigation tasks. This work advances the understanding of service security in VLM-powered navigation systems, providing computational foundations for robust service composition in physical-world deployments.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 72 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

60 of 72 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  2. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2023Segment Anythingcited by 8,234
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2017Mask R-CNNcited by 7,277
  11. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  12. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
38 more
  1. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  5. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  8. 2022Language-driven Semantic Segmentationcited by 717
  9. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  10. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  11. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  12. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  13. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  14. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  15. 2022Visual Language Maps for Robot Navigationcited by 365
  16. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  17. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  18. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 288
  19. 2023Large Language Model Alignment: A Surveycited by 243
  20. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  21. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  22. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  23. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  24. 2020Adversarial Camouflage: Hiding Physical-World Attacks with Natural Stylescited by 126
  25. 2018PointCloud Saliency Mapscited by 112
  26. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
  27. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 89
  28. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  29. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 73
  30. 2022Physical Adversarial Attack meets Computer Vision: A Decade Surveycited by 67
  31. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  32. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  33. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  34. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 56
  35. 2020LG-GAN: Label Guided Adversarial Network for Flexible Targeted Attack of Point Cloud-based Deep Networkscited by 55
  36. 2022DTA: Physical Camouflage Attacks using Differentiable Transformation Networkcited by 51
  37. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 48
  38. 2024Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Samplingcited by 37

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025From Static to Adaptive Defense: Federated Multi-Agent Deep Reinforcement Learning-Driven Moving Target Defense Against DoS Attacks in UAV Swarm Networkscited by 1
  2. 2025Hierarchical Task Offloading and Trajectory Optimization in Low-Altitude Intelligent Networks Via Auction and Diffusion-based MARLcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Adversarial attacks against Modern Vision-Language Models
  2. 2026Partially Observable Adversarial Patch Attacks on Vision-Language-Action Models in Robotics
  3. 2026Exploiting Vulnerabilities: Universal Adversarial Attacks on Vision-Language-Action Models in Robotics
  4. 2024Malicious Path Manipulations via Exploitation of Representation Vulnerabilities of Vision-Language Navigation Systems
  5. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Driving
  6. 2025When Data Manipulation Meets Attack Goals: An In-depth Survey of Attacks for VLMs
  7. 2025Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack
  8. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors