mlsanityData to 7 Oct 2026

JailGuard: A Universal Detection Framework for LLM Prompt-based Attacks

arXiv 2312.10766 ↗First posted 17 Dec 2023cs.CRcited by 46

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The systems and software powered by Large Language Models (LLMs) and Multi-Modal LLMs (MLLMs) have played a critical role in numerous scenarios. However, current LLM systems are vulnerable to prompt-based attacks, with jailbreaking attacks enabling the LLM system to generate harmful content, while hijacking attacks manipulate the LLM system to perform attacker-desired tasks, underscoring the necessity for detection tools. Unfortunately, existing detecting approaches are usually tailored to specific attacks, resulting in poor generalization in detecting various attacks across different modalities. To address it, we propose JailGuard, a universal detection framework deployed on top of LLM systems for prompt-based attacks across text and image modalities. JailGuard operates on the principle that attacks are inherently less robust than benign ones. Specifically, JailGuard mutates untrusted inputs to generate variants and leverages the discrepancy of the variants' responses on the target model to distinguish attack samples from benign samples. We implement 18 mutators for text and image inputs and design a mutator combination policy to further improve detection generalization. The evaluation on the dataset containing 15 known attack types suggests that JailGuard achieves the best detection accuracy of 86.14%/82.90% on text and image inputs, outperforming state-of-the-art methods by 11.81%-25.73% and 12.20%-21.40%.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. First posted on arXiv 17 Dec 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 22; 2026: 10.

46 papers.

Sort:

By area:

  1. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  4. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  5. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  6. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  7. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  8. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  9. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 61
  10. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  11. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 42
  12. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
34 more
  1. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  2. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
  3. 2024Safety Alignment for Vision Language Modelscited by 27
  4. 2025WAInjectBench: Benchmarking Prompt Injection Detections for Web Agentscited by 21
  5. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Modelscited by 19
  6. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17
  7. 2025FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowchartscited by 15
  8. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  9. 2024Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroringcited by 10
  10. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Modelcited by 10
  11. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  12. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  13. 2024Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vectorcited by 8
  14. 2024Defending LVLMs Against Vision Attacks through Partial-Perception Supervisioncited by 7
  15. 2024Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignmentcited by 5
  16. 2025AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenderscited by 5
  17. 2025PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreakingcited by 3
  18. 2026Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Modelscited by 3
  19. 2026WebAgentGuard: A Reasoning-Driven Guard Model for Detecting Prompt Injection Attacks in Web Agentscited by 3
  20. 2025CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensorscited by 2
  21. 2025Contextual Image Attack: How Visual Context Exposes Multimodal Safety Vulnerabilitiescited by 2
  22. 2026Safe responses matter: Output-aware safety guardrail mitigate over-refusal in MLLMscited by 2
  23. 2025Topological Signatures of Adversaries in Multimodal Alignmentscited by 1
  24. 2025How Jailbreak Defenses Work and Ensemble? A Mechanistic Investigationcited by 1
  25. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  26. 2025Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractorscited by 1
  27. 2026VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defensecited by 1
  28. 2026Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projectioncited by 1
  29. 2025Multilingual Collaborative Defense for Large Language Modelscited by 0
  30. 2026FENCE: A Financial and Multimodal Jailbreak Detection Datasetcited by 0
  31. 2026VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Promptscited by 0
  32. 2026Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreakscited by 0
  33. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Modelscited by 0
  34. 2026SafeRI: Recognition and Intervention for Token-Level Safety Intervention in Large Vision Language Modelscited by 0