mlsanityData to 14 Oct 2026

Misusing Tools in Large Language Models With Visual Adversarial Examples

arXiv 2310.03185 ↗First posted 4 Oct 2023cs.CR cs.AIcited by 42

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are being enhanced with the ability to use tools and to process multiple modalities. These new capabilities bring new benefits and also new security risks. In this work, we show that an attacker can use visual adversarial examples to cause attacker-desired tool usage. For example, the attacker could cause a victim LLM to delete calendar events, leak private conversations and book hotels. Different from prior work, our attacks can affect the confidentiality and integrity of user resources connected to the LLM while being stealthy and generalizable to multiple input prompts. We construct these attacks using gradient-based adversarial training and characterize performance along multiple dimensions. We find that our adversarial images can manipulate the LLM to invoke tools following real-world syntax almost always (~98%) while maintaining high similarity to clean images (~0.9 SSIM). Furthermore, using human scoring and automated metrics, we find that the attacks do not noticeably affect the conversation (and its semantics) between the user and the LLM.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

18 of 34 references.

References · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
6 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  3. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  4. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 124
  5. 2023On the Exploitability of Instruction Tuningcited by 116
  6. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 13; 2025: 16; 2026: 10.

42 papers.

Sort:

By area:

  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  2. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  3. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
  4. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  6. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 111
  7. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 99
  8. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  9. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  10. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 82
  11. 2024Best-of-N Jailbreakingcited by 76
  12. 2024Safety of Multimodal Large Language Models on Images and Textscited by 75
30 more
  1. 2025Design Patterns for Securing LLM Agents against Prompt Injectionscited by 73
  2. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  3. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 68
  4. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 68
  5. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  6. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attackscited by 39
  7. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  8. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 33
  9. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 30
  10. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 28
  11. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Toolscited by 24
  12. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 23
  13. 2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agentscited by 23
  14. 2026From Storage to Steering: Memory Control Flow Attacks on LLM Agentscited by 19
  15. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interfacecited by 17
  16. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 17
  17. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  18. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
  19. 2025On the Robustness of GUI Grounding Models Against Image Attackscited by 14
  20. 2025Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correctioncited by 14
  21. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  22. 2025Transferability of Adversarial Attacks in Video-based MLLMs: A Cross-modal Image-to-Video Approachcited by 7
  23. 2025Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Modelscited by 7
  24. 2026TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systemscited by 5
  25. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 4
  26. 2025Phi: Preference Hijacking in Multi-modal Large Language Models at Inference Timecited by 2
  27. 2026A Marketplace for AI-Generated Adult Content and Deepfakescited by 1
  28. 2026The Persistent Vulnerability of Aligned AI Systemscited by 0
  29. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0
  30. 2026AwarenessBench: Assessing Cognitive Capabilities of Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
  2. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  3. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  4. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models
  5. 2024Coercing LLMs to do and reveal (almost) anything
  6. 2025Attack and defense techniques in large language models: A survey and new perspectives
  7. 2025Safeguarding Efficacy in Large Language Models: Evaluating Resistance to Human-Written and Algorithmic Adversarial Prompts
  8. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research