mlsanityData to 7 Oct 2026

Safety alignment, refusal and guardrails

1,071 papers14 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: safety, alignment, llms, harmful, language

Key facts. Safety alignment, refusal and guardrails: 1,071 papers on arXiv since 2022 (data through 7 Oct 2026). 244 of its 1,071 mapped papers (23%) cite “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” (2023, arXiv 2310.03693). 204 of its 1,071 mapped papers (19%) cite “Refusal in Language Models Is Mediated by a Single Direction” (2024, arXiv 2406.11717). 44 new papers 24 Sep–7 Oct.

What’s new

24 Sep – 7 Oct 2026

44 new papers, 24 Sep – 7 Oct; 14 this week.

  1. This week · 14
  2. How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defense
  3. Safe at One Loop, Risky at Another: Aligning Safety Across Recurrent Depths in Looped Language Models
  4. How to train your model organism
  5. SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models
  6. Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs
  7. AdaGuard: Enhancing Safety and Policy Compliance with Reasoning-Enabled LLM-As-A-Judge Guardrails also fits: Prompt injection and agent security
  8. Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements
  9. SIGMA: Self-Improving Alignment Generalization from a Model Spec
  10. The Model Plants the Trigger: Answer-Side Backdoor Attacks in Multi-Turn Large Language Models also fits: Jailbreaking
  11. Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge also fits: Prompt injection and agent security
All 44
  1. Dynamic Budget Allocation for LLM Evaluation under Hard Resource Constraints also fits: Prompt injection and agent security
  2. SAFESHIELD: A Decision-Organization Framework for Deployment-Time Safety of Small Language Models
  3. TrustMI: Causally controlling how assistants trust their users also fits: Prompt injection and agent security
  4. Visual Grounding Safety in Vision-Language Models
  5. Earlier · 30
  6. Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment
  7. Don't Judge an LLM Only by Its Activations: Discovering Suppressed Safety Features via Counterfactual Activation Potential
  8. Safe Context Switching for Agents in the Wild: Mitigating Subspace Interference via Orthogonal Adaptation
  9. EmoRSS: Mitigating Emotion-Induced Over-Refusal in Large Language Models
  10. On the Steering Dimensionality of Refusal in Language Models
  11. Bounded Reachability & Jailbreak Detection via Contraction-Constrained State Space Models also fits: Jailbreaking
  12. DNAlign: Dynamic Null-Space Safe Alignment for LLMs
  13. TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safety
  14. Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding
  15. Faithful Dual-constrained Erasure for Robust LLM Safety Alignment also fits: Machine unlearning
  16. Refusal Localizes, the Damage Relocates: Safety Layers Under Few-Sample Fine-Tuning
  17. The Geometry of Harmfulness in Multi-Turn Attacks
  18. Evaluating Language Model Safety Across Long Adversarial Conversations
  19. ACTR: Aligning Thoughts and Responses for Multilingual Safety in Reasoning LLMs
  20. Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models
  21. Less Sycophancy, Stronger Refusal? Lessons for AI Safety from Mechanistic Interpretability
  22. Don't Inoculate Everything: Stratified Inoculation Prompting Narrows Backdoor Triggers and Preserves Desired Traits also fits: Machine unlearning
  23. Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Models
  24. DeShortcut-Align: Decoupling Spurious Shortcuts for Robust Safety Alignment in Large Reasoning Models
  25. When Do Model Internals Help? Exploring the Role of Representation Engineering in LLM Safety
  26. Jailbreak Context Lingers: Divergent Safety Routing and Its Cross-Task Predictability in Tool Agents also fits: Jailbreaking
  27. EOPSA: Efficient On-Policy Self-Distilled Safety Alignment
  28. How to Tame a Multi-Headed Hydra? Adaptive Multi-Category Safety Steering for Large Language Models
  29. COGNIT-Guard: Calibrated Standalone Direct-Decision Guardrails with Heterogeneous CPU-NPU Confidence Cascading under Explicit Latency and False-Positive Constraints
  30. AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents also fits: Prompt injection and agent security
  31. SafeMol: Dual-Modality Safety Alignment for Molecular Multimodal Models
  32. Safety Reconstructed: Generative Modeling via Masked Diffusion Builds Strong Safety Guardrails
  33. CG-Probes: Recovering Guardrail Directions from Patient Query Embeddings
  34. Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment
  35. Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

Foundations

Most cited by papers here

  1. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!2023cited by 244 here
  2. Refusal in Language Models Is Mediated by a Single Direction2024cited by 204 here
  3. Safety Alignment Should Be Made More Than Just a Few Tokens Deep2024cited by 160 here
  4. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs2024cited by 149 here
  5. Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications2024cited by 86 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2022Red Teaming Language Models with Language Modelscited by 100 here
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 84 here
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 228 here
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 419 here
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 244 here
  6. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 85 here
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 208 here
  8. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 89 here
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 286 here
  10. 2024A StrongREJECT for Empty Jailbreakscited by 99 here
  11. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 117 here
  12. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 94 here
  13. 2024Improving Alignment and Robustness with Circuit Breakerscited by 86 here
  14. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 96 here
10 more it builds on, not in this chain
  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 68 here
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 62 here
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 119 here
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 84 here
  5. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 86 here
  6. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 76 here
  7. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 160 here
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 204 here
  9. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 62 here
  10. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 149 here

The 24 most cited of 1,516.

Browse

30 of 1,071, ranked 6–35

  1. 6Improving Alignment and Robustness with Circuit Breakers202486
  2. 7OR-Bench: An Over-Refusal Benchmark for Large Language Models202476
  3. 8PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference202462
  4. 9Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails202558
  5. 10LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B202358
  6. 11What is in Your Safe Data? Identifying Benign Data that Breaks Safety202443
  7. 12Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack202443
  8. 13Representation Noising: A Defence Mechanism Against Harmful Finetuning202438
  9. 14Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs202537
  10. 15The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence202535
All 30
  1. 16Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning202435
  2. 17The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1202533
  3. 18Qwen3Guard Technical Report202533
  4. 19LLMs Encode Harmfulness and Refusal Separately202531
  5. 20ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming202431
  6. 21Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey202430
  7. 22GuardReasoner: Towards Reasoning-based LLM Safeguards202529
  8. 23On Prompt-Driven Safeguarding for Large Language Models202427
  9. 24The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning202426
  10. 25RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability202525
  11. 26OpenAI o1 System Card202424
  12. 27STAIR: Improving Safety Alignment with Introspective Reasoning202522
  13. 28H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking202522
  14. 29FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning202522
  15. 30A safety realignment framework via subspace-oriented model fusion for large language models202422
  16. 31Building Guardrails for Large Language Models202421
  17. 32Robustifying Safety-Aligned Large Language Models through Clean Data Curation202421
  18. 33Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models202421
  19. 34SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering202421
  20. 35Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation202421