mlsanityData to 7 Oct 2026

LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shifts

arXiv 2410.10700 ↗First posted 14 Oct 2024cs.CL cs.AIcited by 80

Not on the map. Why?

Cited by 64 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Safety concerns in large language models (LLMs) have gained significant attention due to their exposure to potentially harmful data during pre-training. In this paper, we identify a new safety vulnerability in LLMs: their susceptibility to textitnatural distribution shifts between attack prompts and original toxic prompts, where seemingly benign prompts, semantically related to harmful content, can bypass safety mechanisms. To explore this issue, we introduce a novel attack method, textitActorBreaker, which identifies actors related to toxic prompts within pre-training distribution to craft multi-turn prompts that gradually lead LLMs to reveal unsafe content. ActorBreaker is grounded in Latour's actor-network theory, encompassing both human and non-human actors to capture a broader range of vulnerabilities. Our experimental results demonstrate that ActorBreaker outperforms existing attack methods in terms of diversity, effectiveness, and efficiency across aligned LLMs. To address this vulnerability, we propose expanding safety training to cover a broader semantic space of toxic content. We thus construct a multi-turn safety dataset using ActorBreaker. Fine-tuning models on our dataset shows significant improvements in robustness, though with some trade-offs in utility. Code is available at https://github.com/AI45Lab/ActorAttack.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 59 of its 80 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

59 of 80 references.

References · 59

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
38 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  13. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  14. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  15. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  16. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  17. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  18. 2023Detecting Language Model Attacks with Perplexitycited by 411
  19. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  20. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  21. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  22. 2023Faithful Chain-of-Thought Reasoningcited by 285
  23. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  24. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  25. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  26. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  27. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  28. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  29. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  30. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  31. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  32. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  33. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  34. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 79
  35. 2024Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLMcited by 73
  36. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  37. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  38. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 33; 2026: 47.

80 papers.

Sort:

By area:

  1. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 94
  4. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 61
  5. 2025Safety in Large Reasoning Models: A Surveycited by 58
  6. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  7. 2025MAGPIE: A benchmark for Multi-AGent contextual PrIvacy Evaluationcited by 27
  8. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  9. 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attackscited by 23
  10. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  11. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMscited by 18
  12. 2025Tempest: Autonomous Multi-Turn Jailbreaking of Large Language Models with Tree Searchcited by 15
38 more
  1. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  2. 2025Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoningcited by 11
  3. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMscited by 10
  4. 2026SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attackscited by 10
  5. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  6. 2025Jailbreaking to Jailbreakcited by 8
  7. 2025MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attackscited by 8
  8. 2025Does Chain-of-Thought Reasoning Really Reduce Harmfulness from Jailbreaking?cited by 8
  9. 2025Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attackscited by 8
  10. 2026Quantifying Trust: Financial Risk Management for Trustworthy AI Agentscited by 8
  11. 2025SafeTy Reasoning Elicitation Alignment for Multi-Turn Dialoguescited by 7
  12. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safetycited by 7
  13. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replaycited by 6
  14. 2025Strategize Globally, Adapt Locally: A Multi-Turn Red Teaming Agent with Dual-Level Learningcited by 5
  15. 2025Chain-of-Lure: A Universal Jailbreak Attack Framework using Unconstrained Synthetic Narrativescited by 5
  16. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
  17. 2026ICON: Intent-Context Coupling for Efficient Multi-Turn Jailbreak Attackcited by 5
  18. 2026Evaluating Answer Leakage Robustness of LLM Tutors against Adversarial Student Attackscited by 5
  19. 2025Eliciting Behaviors in Multi-Turn Conversationscited by 4
  20. 2026Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counselingcited by 4
  21. 2025ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreakscited by 3
  22. 2026The Echo Chamber Multi-Turn LLM Jailbreakcited by 3
  23. 2026MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attackscited by 3
  24. 2025Effective and Efficient Jailbreaks of Black-Box LLMs with Cross-Behavior Attackscited by 2
  25. 2025The Illusionist's Prompt: Exposing the Factual Vulnerabilities of Large Language Models with Linguistic Nuancescited by 2
  26. 2025PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploitscited by 2
  27. 2025Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judgescited by 2
  28. 2025Efficient LLM Safety Evaluation through Multi-Agent Debatecited by 2
  29. 2026Multi-turn Jailbreaking Attack in Multi-Modal Large Language Modelscited by 2
  30. 2026Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmeticcited by 2
  31. 2026Helpful to a Fault: Measuring Illicit Assistance in Multi-Turn, Multilingual LLM Agentscited by 2
  32. 2026TrajGuard: Streaming Hidden-state Trajectory Detection for Decoding-time Jailbreak Defensecited by 2
  33. 2026SafeDream: Safety World Model for Proactive Early Jailbreak Detectioncited by 2
  34. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  35. 2026Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimizationcited by 2
  36. 2026Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMscited by 2
  37. 2025Activation-Guided Local Editing for Jailbreaking Attackscited by 1
  38. 2025Bidirectional Intention Inference Enhances LLMs' Defense Against Multi-Turn Jailbreak Attackscited by 1

50 most cited of 80 All 80 (JSON)