mlsanityData to 14 Oct 2026

LLM Safety Alignment is Divergence Estimation in Disguise

arXiv 2502.00657 ↗First posted 2 Feb 2025cs.LG cs.AI cs.CYcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

We present a theoretical framework showing that popular LLM alignment methods, including RLHF and its variants, can be understood as divergence estimators between aligned (safe or preferred) and unaligned (harmful or less preferred) distributions. This perspective explains the emergence of separation in the latent space between safe and harmful prompts after alignment. As an application of our general divergence framework, we propose KLDO, a novel KL divergence-based alignment method, and empirically validate its effectiveness. We further show that using compliance-refusal datasets, rather than standard preference-based datasets, leads to stronger separation and improved safety alignment. Finally, to quantify the separation effect, we propose a distance-based metric in the prompt representation space, which also acts as a statistically significant indicator for model safety.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 54 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

47 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024SafeAligner: Safety Alignment against Jailbreak Attacks via Response Disparity Guidance26 Jun
  2. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  5. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 42

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023Mistral 7Bcited by 3,164
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
30 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  4. 2023Instruction Tuning with GPT-4cited by 686
  5. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  6. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  7. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  9. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  10. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  11. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  12. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  13. 2024Direct Preference Optimization with an Offsetcited by 122
  14. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  15. 2023Is RLHF More Difficult than Standard RL?cited by 96
  16. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  17. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  18. 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysiscited by 64
  19. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  20. 2023Spear Phishing With Large Language Modelscited by 52
  21. 2024Binary Classifier Optimization for Large Language Model Alignmentcited by 45
  22. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 44
  23. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  24. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  25. 2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 20
  26. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  27. 2024BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignmentcited by 20
  28. 2024Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labelscited by 19
  29. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
  30. 2024Accelerated Preference Optimization for Large Language Model Alignmentcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual contextcited by 6
  2. 2026Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingcited by 5
  3. 2026LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safetycited by 3
  4. 2026An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shiftcited by 2
  5. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustnesscited by 1
  6. 2026f-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  2. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
  3. 2024Alignment with Preference Optimization Is All You Need for LLM Safety
  4. 2026Why Is RLHF Alignment Shallow? A Gradient Analysis
  5. 2026Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs
  6. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates
  7. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Control
  8. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning