mlsanityData to 7 Oct 2026

Alignment with Preference Optimization Is All You Need for LLM Safety

arXiv 2409.07772 ↗First posted 12 Sep 2024cs.LGcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

We demonstrate that preference optimization methods can effectively enhance LLM safety. Applying various alignment techniques to the Falcon 11B model using safety datasets, we achieve a significant boost in global safety score (from 57.64% to 99.90%) as measured by LlamaGuard 3 8B, competing with state-of-the-art models. On toxicity benchmarks, average scores in adversarial settings dropped from over 0.6 to less than 0.07. However, this safety improvement comes at the cost of reduced general capabilities, particularly in math, suggesting a trade-off. We identify noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and performance. Our study ultimately shows that alignment techniques can be sufficient for building safe and robust models.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 25 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

18 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  7. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  8. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  9. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  10. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  11. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  12. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
4 more
  1. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  2. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  3. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  4. 2024Automated Progressive Red Teamingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Positioncited by 11
  3. 2024Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theorycited by 6
  4. 2025AdaptiveGuard: Towards Adaptive Runtime Safety for LLM-Powered Softwarecited by 2
  5. 2026PromptCD: Test-Time Behavior Enhancement via Polarity-Prompt Contrastive Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Safety Alignment is Divergence Estimation in Disguise
  2. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
  3. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Control
  4. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  5. 2026CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment
  6. 2025More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
  7. 2026BSO: Safety Alignment Is Density Ratio Matching
  8. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMs