mlsanityData to 7 Oct 2026

Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety

arXiv 2502.05206 ↗First posted 2 Feb 2025cs.CR cs.AI cs.CLcited by 63

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

The rapid advancement of large models, driven by their exceptional abilities in learning and generalization through large-scale pre-training, has reshaped the landscape of Artificial Intelligence (AI). These models are now foundational to a wide range of applications, including conversational AI, recommendation systems, autonomous driving, content generation, medical diagnostics, and scientific discovery. However, their widespread deployment also exposes them to significant safety risks, raising concerns about robustness, reliability, and ethical implications. This survey provides a systematic review of current safety research on large models, covering Vision Foundation Models (VFMs), Large Language Models (LLMs), Vision-Language Pre-training (VLP) models, Vision-Language Models (VLMs), Diffusion Models (DMs), and large-model-powered Agents. Our contributions are summarized as follows: (1) We present a comprehensive taxonomy of safety threats to these models, including adversarial attacks, data poisoning, backdoor attacks, jailbreak and prompt injection attacks, energy-latency attacks, data and model extraction attacks, and emerging agent-specific threats. (2) We review defense strategies proposed for each type of attacks if available and summarize the commonly used datasets and benchmarks for safety research. (3) Building on this, we identify and discuss the open challenges in large model safety, emphasizing the need for comprehensive safety evaluations, scalable and effective defense mechanisms, and sustainable data practices. More importantly, we highlight the necessity of collective efforts from the research community and international collaboration. Our work can serve as a useful reference for researchers and practitioners, fostering the ongoing development of comprehensive defense systems and platforms to safeguard AI models.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 257 of its 706 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

257 of 706 references.

Closest ancestors · 24

Same direction, newest first

  1. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models30 Jan
  2. 2024Adversarial Prompt Distillation for Vision-Language Models22 Nov
  3. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey31 Oct
  4. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks28 Oct
  5. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models7 Oct
  6. 2024Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Models20 Sep
  7. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends10 Jul
  8. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift27 Jun
  9. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models26 Jun
  10. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Models13 Jun
  11. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models8 Jun
  12. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors17 May
12 more
  1. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective30 Apr
  2. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models16 Apr
  3. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructions14 Mar
  4. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image5 Mar
  5. 2024Test-Time Backdoor Attacks on Multimodal Large Language Models13 Feb
  6. 2024Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks1 Feb
  7. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  8. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization7 Dec
  9. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models4 Dec
  10. 2023Can Language Models be Instructed to Protect Personal Information?3 Oct
  11. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning24 Aug
  12. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs19 Jul

Other references · 233

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2023Mistral 7Bcited by 3,065
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2024Mixtral of Expertscited by 1,785
38 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  8. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  9. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  10. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  11. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  12. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  13. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  14. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  15. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  16. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  17. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  18. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  19. 2024Alignment faking in large language modelscited by 316
  20. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  21. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 257
  22. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  23. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  24. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  25. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  26. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  27. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  28. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  29. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  30. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  31. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  32. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 181
  33. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  34. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  35. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  36. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  37. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 144
  38. 2023A Recipe for Watermarking Diffusion Modelscited by 136

50 most cited of 233 All 233 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 48.

63 papers.

Sort:

By area:

  1. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 120
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2025Measuring Agents in Productioncited by 35
  4. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  5. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
  6. 2026A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypescited by 19
  7. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMscited by 18
  8. 2025Attack the Messages, Not the Agents: A Multi-round Adaptive Stealthy Tampering Framework for LLM-MAScited by 17
  9. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
  10. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  11. 2026MemMark: State-Evolution Attribution Watermarking for Agent Long-Term Memory Systemscited by 12
  12. 2025Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Cipherscited by 8
38 more
  1. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Modelscited by 8
  2. 2026AIR: Improving Agent Safety through Incident Responsecited by 8
  3. 2026Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verificationcited by 6
  4. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Modelscited by 5
  5. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  6. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  7. 2026Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safetycited by 4
  8. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Modelscited by 4
  9. 2026Safeguarding LLM Agents from Misalignment through Provenance Analysiscited by 4
  10. 2025Thought Purity: A Defense Framework For Chain-of-Thought Attackcited by 3
  11. 2025PaperAsk: A Benchmark for Reliability Evaluation of LLMs in Paper Search and Readingcited by 3
  12. 2026DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoningcited by 3
  13. 2026Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMscited by 3
  14. 2026MCPShield: A Security Cognition Layer for Adaptive Trust Calibration in Model Context Protocol Agentscited by 3
  15. 2026TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systemscited by 3
  16. 2026Safety, Security, and Cognitive Risks in World Modelscited by 3
  17. 2026DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Modelcited by 3
  18. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  19. 2025Reading Smiles: Proxy Bias in Foundation Models for Facial Emotion Recognitioncited by 1
  20. 2025ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIPcited by 1
  21. 2026Automated Safety Benchmarking: A Multi-agent Pipeline for LVLMscited by 1
  22. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Modelscited by 1
  23. 2026Do Prompts Guarantee Safety? Mitigating Toxicity from LLM Generations through Subspace Interventioncited by 1
  24. 2026The Struggle Between Continuation and Refusal: A Mechanistic Analysis of the Continuation-Triggered Jailbreak in LLMscited by 1
  25. 2026From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directionscited by 1
  26. 2026Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agentscited by 1
  27. 2026When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agentscited by 1
  28. 2026Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agentscited by 1
  29. 2026Governance by Construction for Generalist Agentscited by 1
  30. 2026ShareLock: A Stealthy Multi-Tool Threshold Poisoning Attack Against MCPcited by 1
  31. 2025TestDG: Test-time Domain Generalization for Continual Test-time Adaptationcited by 0
  32. 2025RedTopic: Toward Topic-Diverse Red Teaming of Large Language Modelscited by 0
  33. 2026SafeCRS: Personalized Safety Alignment for LLM-Based Conversational Recommender Systemscited by 0
  34. 2026OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequencescited by 0
  35. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  36. 2026Adjustable Text-Guided Backdoor Attacks with Natural-Word Triggers on Multimodal Pretrained Modelscited by 0
  37. 2026Adaptive Equilibrium: Dynamic Weighting Framework for Generalized Interruption of DeepFake Modelscited by 0
  38. 2026Disentangling Intent from Role: Adversarial Self-Play for Persona-Invariant Safety Alignmentcited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Model Safety: A Holistic Survey
  2. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations
  3. 2026Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety
  4. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1
  5. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  6. 2026A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5
  7. 2025Noise Injection Systemically Degrades Large Language Model Safety Guardrails
  8. 2025DeepKnown-Guard: A Proprietary Model-Based Safety Response Framework for AI Agents