mlsanityData to 14 Oct 2026

From AI-Generated Content to Agentic Action: Security and Safety Threats in Generative AI

arXiv 2605.16471 ↗First posted 15 May 2026cs.CRcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Generative AI systems are increasingly used not only to produce content but also to retrieve data, invoke tools, and execute actions. This work examines the security and safety implications of that shift across content-level, model-level, and agentic threats. We analyze how attacker access requirements, system autonomy, and the scope of potential harm change as models move from generating artifacts to executing operations through tool chains and external APIs. We then assess technical countermeasures including detection, watermarking, alignment, and emerging agentic safeguards, and show that several depend on forms of institutional coordination that current governance arrangements do not yet provide. Across the cases examined, capability deployment and attack-surface expansion repeatedly outpace defensive responses as systems move from generating content to executing real-world actions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 80 of its 133 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

80 of 133 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers19 Aug
  2. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents24 Mar
  3. 2025Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories5 Mar
  4. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  5. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  6. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan

Other references · 74

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
38 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,101
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  4. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  6. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  7. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  9. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  10. 2023A Watermark for Large Language Modelscited by 725
  11. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  12. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  13. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  14. 2023Towards Universal Fake Image Detectors that Generalize Across Generative Modelscited by 490
  15. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  16. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  17. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  18. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  19. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  20. 2023Can AI-Generated Text be Reliably Detected?cited by 386
  21. 2024A StrongREJECT for Empty Jailbreakscited by 385
  22. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  23. 2023GenImage: A Million-Scale Benchmark for Detecting AI-Generated Imagecited by 321
  24. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  25. 2023The Stable Signature: Rooting Watermarks in Latent Diffusion Modelscited by 304
  26. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  27. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  28. 2019StegaStamp: Invisible Hyperlinks in Physical Photographscited by 265
  29. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  30. 2024Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Textcited by 263
  31. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  32. 2023RADAR: Robust AI-Text Detection via Adversarial Learningcited by 190
  33. 2023Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robustcited by 187
  34. 2024RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectorscited by 163
  35. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 128
  36. 2024AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Errorcited by 125
  37. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 122
  38. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 110

50 most cited of 74 All 74 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agents
  2. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomy
  3. 2025A Safety and Security Framework for Real-World Agentic Systems
  4. 2025Securing Generative AI Agentic Workflows: Risks, Mitigation, and a Proposed Firewall Architecture
  5. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
  6. 2026Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges
  7. 2026From Secure Agentic AI to Secure Agentic Web: Challenges, Threats, and Future Directions
  8. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents