mlsanityData to 7 Oct 2026

WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models

arXiv 2406.18510 ↗First posted 26 Jun 2024cs.CLcited by 206

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

We introduce WildTeaming, an automatic LLM safety red-teaming framework that mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of novel jailbreak tactics, and then composes multiple tactics for systematic exploration of novel jailbreaks. Compared to prior work that performed red-teaming via recruited human workers, gradient-based optimization, or iterative revision with LLMs, our work investigates jailbreaks from chatbot users who were not specifically instructed to break the system. WildTeaming reveals previously unidentified vulnerabilities of frontier LLMs, resulting in up to 4.6x more diverse and successful adversarial attacks compared to state-of-the-art jailbreak methods. While many datasets exist for jailbreak evaluation, very few open-source datasets exist for jailbreak training, as safety training data has been closed even when model weights are open. With WildTeaming we create WildJailbreak, a large-scale open-source synthetic safety dataset with 262K vanilla (direct request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate exaggerated safety behaviors, WildJailbreak provides two contrastive types of queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that resemble harmful queries in form but contain no harm. As WildJailbreak considerably upgrades the quality and scale of existing safety resources, it uniquely enables us to examine the scaling effects of data and the interplay of data properties and model capabilities during safety training. Through extensive experiments, we identify the training properties that enable an ideal balance of safety behaviors: appropriate safeguarding without over-refusal, effective handling of vanilla and adversarial queries, and minimal, if any, decrease in general capabilities. All components of WildJailbeak contribute to achieving balanced safety behaviors of models.

Key facts. Cited by 206 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 11 of its 91 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

11 of 91 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 9

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  4. 2023Instruction Tuning with GPT-4cited by 676
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  6. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  7. 2023An Overview of Catastrophic AI Riskscited by 214
  8. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attackscited by 30
  9. 2024Corrective Machine Unlearningcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 98; 2026: 91.

206 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  3. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  4. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  5. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  6. 2025Qwen3Guard Technical Reportcited by 125
  7. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  8. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  9. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  10. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  11. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  12. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
38 more
  1. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  2. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 41
  3. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
  4. 2025Large-Scale Data Selection for Instruction Tuningcited by 34
  5. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  6. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  7. 2025Representation Bending for Large Language Model Safetycited by 28
  8. 2025OpenAI's Approach to External Red Teaming for AI Models and Systemscited by 26
  9. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  10. 2025Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Modelscited by 25
  11. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 25
  12. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 25
  13. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 24
  14. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
  15. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  16. 2025To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Modelscited by 22
  17. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  18. 2025BingoGuard: LLM Content Moderation Tools with Risk Levelscited by 20
  19. 2025Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilitiescited by 20
  20. 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 20
  21. 2025Enhancing LLM Safety Through a Theoretical Minimax Game Lenscited by 18
  22. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 18
  23. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  24. 2025NeuroStrike: Neuron-Level Attacks on Aligned LLMscited by 18
  25. 2024SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Taskscited by 17
  26. 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 17
  27. 2025PurpCode: Reasoning for Safer Code Generationcited by 17
  28. 2025Tempest: Autonomous Multi-Turn Jailbreaking of Large Language Models with Tree Searchcited by 15
  29. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  30. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 15
  31. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  32. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 14
  33. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 13
  34. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  35. 2025Consistency Training Helps Stop Sycophancy and Jailbreakscited by 13
  36. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 12
  37. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compressioncited by 12
  38. 2025DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Modelscited by 11

50 most cited of 206 All 206 (JSON)