mlsanityData to 13 Oct 2026

Preventing Jailbreak Prompts as Malicious Tools for Cybercriminals: A Cyber Defense Perspective

arXiv 2411.16642 ↗First posted 25 Nov 2024cs.CR cs.CLcited by 3

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreak prompts pose a significant threat in AI and cybersecurity, as they are crafted to bypass ethical safeguards in large language models, potentially enabling misuse by cybercriminals. This paper analyzes jailbreak prompts from a cyber defense perspective, exploring techniques like prompt injection and context manipulation that allow harmful content generation, content filter evasion, and sensitive information extraction. We assess the impact of successful jailbreaks, from misinformation and automated social engineering to hazardous content creation, including bioweapons and explosives. To address these threats, we propose strategies involving advanced prompt analysis, dynamic safety protocols, and continuous model fine-tuning to strengthen AI resilience. Additionally, we highlight the need for collaboration among AI researchers, cybersecurity experts, and policymakers to set standards for protecting AI systems. Through case studies, we illustrate these cyber defense approaches, promoting responsible AI practices to maintain system integrity and public trust. textbfcolorredWarning: This paper contains content which the reader may find offensive.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 29 of its 43 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

29 of 43 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks2 Nov
  2. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation11 Oct
  3. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet27 Aug
  4. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks18 Aug
  5. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey5 Jul
  6. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings19 Jun
  7. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs13 Jun
  8. 2024AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lens6 Jun
  9. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks30 May
  10. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response22 May
  11. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge8 Apr
  12. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
10 more
  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  2. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
  3. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  4. 2024Break the Breakout: Reinventing LM Defense Against Jailbreak Attacks with Self-Refinement23 Feb
  5. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attacks14 Feb
  6. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  8. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  9. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  10. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 7

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 647
  3. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  4. 2023From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacycited by 143
  5. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 95
  6. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 37
  7. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 34

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  2. 2025TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agentcited by 9
  3. 2026FENCE: A Financial and Multimodal Jailbreak Detection Datasetcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
  2. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility
  3. 2025Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
  4. 2025Defending Large Language Models Against Jailbreak Exploits with Responsible AI Considerations
  5. 2025Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models
  6. 2025CySecBench: Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models
  7. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  8. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models