mlsanityData to 7 Oct 2026

Safety alignment and jailbreak attacks

687 papers7 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: safety, alignment, safety alignment, harmful, llms

Key facts. Safety alignment and jailbreak attacks: 687 papers on arXiv since 2019 (data through 7 Oct 2026). 203 of its 687 mapped papers (30%) cite “Universal and Transferable Adversarial Attacks on Aligned Language Models” (2023, arXiv 2307.15043). 127 of its 687 mapped papers (18%) cite “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” (2023, arXiv 2310.03693). 30 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

30 new papers, 17 Sep – 7 Oct; 7 this week.

  1. This week · 7
  2. How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defense
  3. Safe at One Loop, Risky at Another: Aligning Safety Across Recurrent Depths in Looped Language Models
  4. SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routing
  5. SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models
  6. The Model Plants the Trigger: Answer-Side Backdoor Attacks in Multi-Turn Large Language Models
  7. Reward Stealing Attack on Large Language Models also fits: Reward hacking and misspecification
  8. Visual Grounding Safety in Vision-Language Models
  9. Earlier · 23
  10. Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment
  11. Safe Context Switching for Agents in the Wild: Mitigating Subspace Interference via Orthogonal Adaptation
  12. EmoRSS: Mitigating Emotion-Induced Over-Refusal in Large Language Models
All 30
  1. Target-free Latent Safety Alignment
  2. DNAlign: Dynamic Null-Space Safe Alignment for LLMs
  3. TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safety
  4. CodeMimicry: Exploiting Safety Generalization Lag in Large Language Models via Structured Code Completion
  5. ShieldCLIP: Selective Safety Alignment for Harmful Content Mitigation in Multimodal Foundation Models
  6. Faithful Dual-constrained Erasure for Robust LLM Safety Alignment
  7. ACTR: Aligning Thoughts and Responses for Multilingual Safety in Reasoning LLMs
  8. Refusal Localizes, the Damage Relocates: Safety Layers Under Few-Sample Fine-Tuning
  9. The Geometry of Harmfulness in Multi-Turn Attacks
  10. Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignment
  11. Controlled Decoding Attacks on Black-Box LLMs
  12. Safer Content or Firmer Refusals? A Hybrid Perturbation Defense for Alignment under Harmful Fine-tuning
  13. Does the Unsafe Gradient Survive a Conversation? On the Fragility of Gradient-Based Jailbreak Detection in Multi-Turn Dialogue
  14. Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models
  15. DeShortcut-Align: Decoupling Spurious Shortcuts for Robust Safety Alignment in Large Reasoning Models
  16. When Do Model Internals Help? Exploring the Role of Representation Engineering in LLM Safety
  17. EOPSA: Efficient On-Policy Self-Distilled Safety Alignment
  18. AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
  19. SafeMol: Dual-Modality Safety Alignment for Molecular Multimodal Models
  20. Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment

Foundations

Most cited by papers here

  1. Universal and Transferable Adversarial Attacks on Aligned Language Models2023cited by 203 here
  2. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!2023cited by 127 here
  3. BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset2023cited by 98 here
  4. Safety Alignment Should Be Made More Than Just a Few Tokens Deep2024cited by 87 here
  5. Safe RLHF: Safe Reinforcement Learning from Human Feedback2023cited by 75 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2017Deep reinforcement learning from human preferencescited by 65 here
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 36 here
  3. 2020Learning to summarize from human feedbackcited by 35 here
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 37 here
  5. 2022Training language models to follow instructions with human feedbackcited by 252 here
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 134 here
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 203 here
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 127 here
  9. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 38 here
  10. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 87 here
14 more it builds on, not in this chain
  1. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 36 here
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 98 here
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 208 here
  4. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 36 here
  5. 2023Are aligned neural networks adversarially aligned?cited by 29 here
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 98 here
  7. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 57 here
  8. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 51 here
  9. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 32 here
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 75 here
  11. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 43 here
  12. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 29 here
  13. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 41 here
  14. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 31 here

The 24 most cited of 692.

Browse

30 of 687, ranked 6–35

  1. 6AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models202357
  2. 7Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models202351
  3. 8FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts202343
  4. 9PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference202441
  5. 10Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications202438
  6. 11Visual Adversarial Examples Jailbreak Aligned Large Language Models202336
  7. 12Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned202236
  8. 13Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations202332
  9. 14Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable202531
  10. 15OR-Bench: An Over-Refusal Benchmark for Large Language Models202429
All 30
  1. 16Are aligned neural networks adversarially aligned?202329
  2. 17SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal202428
  3. 18Baseline Defenses for Adversarial Attacks Against Aligned Language Models202328
  4. 19STAIR: Improving Safety Alignment with Introspective Reasoning202525
  5. 20GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher202325
  6. 21Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models202421
  7. 22Safety Layers in Aligned Large Language Models: The Key to LLM Security202421
  8. 23Removing RLHF Protections in GPT-4 via Fine-Tuning202321
  9. 24SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model202419
  10. 25Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic202418
  11. 26LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B202317
  12. 27Universal Jailbreak Backdoors from Poisoned Human Feedback202317
  13. 28Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack202416
  14. 29Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack202416
  15. 30Rule Based Rewards for Language Model Safety202416
  16. 31InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance202415
  17. 32SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety202514
  18. 33A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity202414
  19. 34Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning202413
  20. 35Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety202512