mlsanityData to 14 Oct 2026

Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM

arXiv 2508.05775 ↗First posted 7 Aug 2025cs.CL cs.CYcited by 9

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have revolutionized content creation across digital platforms, offering unprecedented capabilities in natural language generation and understanding. Meanwhile, they pose risks by inadvertently producing toxic, offensive, or biased content. This dual role of LLMs, both as powerful tools for text generation and as potential sources of harmful language, presents a pressing sociotechnical challenge. In this survey, we systematically review recent studies encompassing unintentional toxicity, adversarial jailbreak attacks, and comprehensive mitigation strategies. We explore LLMs' dual role as both generators of harm and enablers of safety through detection, classification, content moderation, and prevention. We propose a unified taxonomy of LLM-related harms and defenses, analyze emerging multimodal and LLM-assisted jailbreak strategies, and assess mitigation efforts, including reinforcement learning with human feedback (RLHF), prompt engineering, and safety alignment. Our review highlights the evolving landscape of LLM safety and identifies limitations in current evaluation methodologies. Ultimately, our review outlines future research directions to guide the development of robust and ethically aligned language technologies.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 149 of its 199 references that are in the map. First posted on arXiv 7 Aug 2025.

Builds on

149 of 199 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Model-Editing-Based Jailbreak against Safety-aligned Large Language Models11 Dec
  2. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  3. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing11 Jul
  4. 2024LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Models3 Jul
  5. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languages23 Jun
  6. 2024RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs13 Jun
  7. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States9 Jun
  8. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  9. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  10. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  12. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 137

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  12. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
38 more
  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  2. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  3. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  4. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  5. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  6. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  7. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  8. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  9. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 207
  10. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  11. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  12. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 184
  13. 2022OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and conceptscited by 177
  14. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 172
  15. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  16. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  17. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 114
  18. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  19. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 107
  20. 2023Text Classification via Large Language Modelscited by 106
  21. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
  22. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 86
  23. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
  24. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 76
  25. 2023Improving Activation Steering in Language Models with Mean-Centringcited by 68
  26. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 68
  27. 2024Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialoguecited by 66
  28. 2022Handling Bias in Toxic Speech Detection: A Surveycited by 61
  29. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 58
  30. 2023"I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generationcited by 56
  31. 2023Self-Guard: Empower the LLM to Safeguard Itselfcited by 54
  32. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  33. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Railscited by 52
  34. 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
  35. 2023Unveiling the Implicit Toxicity in Large Language Modelscited by 51
  36. 2024Evaluating Gender Bias in Large Language Models via Chain-of-Thought Promptingcited by 50
  37. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 49
  38. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspectivecited by 48

50 most cited of 137 All 137 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2026Do Coding Agents Understand Least-Privilege Authorization?cited by 6
  2. 2025Countermind: A Multi-Layered Security Architecture for Large Language Modelscited by 2
  3. 2026Large Language Models in the Abuse Detection Pipelinecited by 1
  4. 2025Systematizing LLM Persona Design: A Four-Quadrant Technical Taxonomy for AI Companion Applicationscited by 0
  5. 2026Safety Is Not Universal: The Selective Safety Trap in LLM Alignmentcited by 0
  6. 2026BarrierSteer: LLM Safety via Learning Barrier Steeringcited by 0
  7. 2026Why Do Large Language Models Generate Harmful Content?cited by 0
  8. 2026One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intentcited by 0
  9. 2026Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Responsible LLMs: Inherent Risk, Malicious Use, and Mitigation Strategy
  2. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks
  3. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
  4. 2025Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?
  5. 2026No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios
  6. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy
  7. 2024Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLM
  8. 2025Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses