mlsanityData to 13 Oct 2026

JailbreakHunter: A Visual Analytics Approach for Jailbreak Prompts Discovery from Large-Scale Human-LLM Conversational Datasets

arXiv 2407.03045 ↗First posted 3 Jul 2024cs.HC cs.CL cs.LGcited by 2

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) have gained significant attention but also raised concerns due to the risk of misuse. Jailbreak prompts, a popular type of adversarial attack towards LLMs, have appeared and constantly evolved to breach the safety protocols of LLMs. To address this issue, LLMs are regularly updated with safety patches based on reported jailbreak prompts. However, malicious users often keep their successful jailbreak prompts private to exploit LLMs. To uncover these private jailbreak prompts, extensive analysis of large-scale conversational datasets is necessary to identify prompts that still manage to bypass the system's defenses. This task is highly challenging due to the immense volume of conversation data, diverse characteristics of jailbreak prompts, and their presence in complex multi-turn conversations. To tackle these challenges, we introduce JailbreakHunter, a visual analytics approach for identifying jailbreak prompts in large-scale human-LLM conversational datasets. We have designed a workflow with three analysis levels: group-level, conversation-level, and turn-level. Group-level analysis enables users to grasp the distribution of conversations and identify suspicious conversations using multiple criteria, such as similarity with reported jailbreak prompts in previous research and attack success rates. Conversation-level analysis facilitates the understanding of the progress of conversations and helps discover jailbreak prompts within their conversation contexts. Turn-level analysis allows users to explore the semantic similarity and token overlap between a singleturn prompt and the reported jailbreak prompts, aiding in the identification of new jailbreak strategies. The effectiveness and usability of the system were verified through multiple case studies and expert interviews.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 47 of its 69 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

47 of 69 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  3. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  4. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models17 Jul
  7. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  8. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  5. 2023Visual Instruction Tuningcited by 7,827
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  7. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,870
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 817
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 723
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 529
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 458
  12. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 421
27 more
  1. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  2. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 354
  3. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  4. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  5. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 284
  6. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 277
  7. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 216
  8. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  9. 2023Auditing large language models: a three-layered approachcited by 136
  10. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 134
  11. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 132
  12. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 98
  13. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 76
  14. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  15. 2018Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Modelscited by 69
  16. 2022Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Modelscited by 51
  17. 2023AttentionViz: A Global View of Transformer Attentioncited by 50
  18. 2020Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Modelscited by 49
  19. 2017Understanding Hidden Memories of Recurrent Neural Networkscited by 44
  20. 2024LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Modelscited by 29
  21. 2023WizMap: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddingscited by 28
  22. 2019SANVis: Visual Analytics for Understanding Self-Attention Networkscited by 21
  23. 2023PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Modelscited by 19
  24. 2022A Unified Understanding of Deep NLP Models for Text Classificationcited by 18
  25. 2022ShortcutLens: A Visual Analytics Approach for Exploring Shortcuts in Natural Language Understanding Datasetcited by 6
  26. 2015Visualization-Aware Sampling for Very Large Databasescited by 5
  27. 2021ConVIScope: Visual Analytics for Exploring Patient Conversationscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defensecited by 10
  2. 2025NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Internal Layers of LLMs Reveal Patterns for Jailbreak Detection?
  2. 2026Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Models
  3. 2024JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
  5. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
  6. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  7. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  8. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models