mlsanityData to 14 Oct 2026

On the Adversarial Robustness of 3D Large Vision-Language Models

arXiv 2601.06464 ↗First posted 10 Jan 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

3D Vision-Language Models (VLMs), such as PointLLM and GPT4Point, have shown strong reasoning and generalization abilities in 3D understanding tasks. However, their adversarial robustness remains largely unexplored. Prior work in 2D VLMs has shown that the integration of visual inputs significantly increases vulnerability to adversarial attacks, making these models easier to manipulate into generating toxic or misleading outputs. In this paper, we investigate whether incorporating 3D vision similarly compromises the robustness of 3D VLMs. To this end, we present the first systematic study of adversarial robustness in point-based 3D VLMs. We propose two complementary attack strategies: textitVision Attack, which perturbs the visual token features produced by the 3D encoder and projector to assess the robustness of vision-language alignment; and textitCaption Attack, which directly manipulates output token sequences to evaluate end-to-end system robustness. Each attack includes both untargeted and targeted variants to measure general vulnerability and susceptibility to controlled manipulation. Our experiments reveal that 3D VLMs exhibit significant adversarial vulnerabilities under untargeted attacks, while demonstrating greater resilience against targeted attacks aimed at forcing specific harmful outputs, compared to their 2D counterparts. These findings highlight the importance of improving the adversarial robustness of 3D VLMs, especially as they are deployed in safety-critical applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 44 references that are in the map. First posted on arXiv 10 Jan 2026.

Builds on

34 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models23 May
  2. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models3 Feb
  3. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models9 Oct
  4. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective30 Apr
  5. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  7. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 979
  8. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  10. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  11. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  12. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
17 more
  1. 2018PU-Net: Point Cloud Upsampling Networkcited by 209
  2. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  3. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  4. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  5. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  6. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  7. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  8. 2018Generating 3D Adversarial Point Cloudscited by 144
  9. 2019AdvPC: Transferable Adversarial Perturbations on 3D Point Cloudscited by 84
  10. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 73
  11. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  12. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 68
  13. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  14. 2021Imperceptible Transfer Attack and Defense on 3D Point Cloud Classificationcited by 36
  15. 2024Hide in Thicket: Generating Imperceptible and Rational Adversarial Perturbations on 3D Point Cloudscited by 24
  16. 2020ShapeAdv: Generating Shape-Aware Adversarial 3D Point Cloudscited by 20
  17. 2023Adversarial Examples in the Physical World: A Surveycited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  2. 2026Breaking the weakest link to evade vision language models
  3. 2024AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?
  4. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  5. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
  6. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  7. 2024Can 3D Vision-Language Models Truly Understand Natural Language?
  8. 2025Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack