mlsanityData to 14 Oct 2026

Suan: Rectifying Direct Preference Safety Alignment in Large Language Models

arXiv 2609.08634 ↗First posted 8 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 64 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

52 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Safety Alignment of LMs via Non-cooperative Games23 Dec
  2. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety26 May
  3. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2020Learning to summarize from human feedbackcited by 2,588
34 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  9. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  10. 2024A Survey on Large Language Models for Code Generationcited by 658
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  12. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  13. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  14. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  15. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  16. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  17. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  18. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  19. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  20. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  21. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  22. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  23. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 81
  24. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  25. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 66
  26. 2025CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentationcited by 57
  27. 2023Spear Phishing With Large Language Modelscited by 52
  28. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  29. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 29
  30. 2024Understanding Reference Policies in Direct Preference Optimizationcited by 22
  31. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Modelscited by 19
  32. 2024Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimizationcited by 18
  33. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 14
  34. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimizationcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Cat-DPO: Category-Adaptive Safety Alignment
  2. 2026Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraints
  3. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
  4. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  5. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  6. 2026Towards Context-Invariant Safety Alignment for Large Language Models
  7. 2025Safe at the Margins: A General Approach to Safety Alignment in Low-Resource English Languages -- A Singlish Case Study
  8. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualization