mlsanityData to 14 Oct 2026

Curriculum Learning for Safety Alignment

arXiv 2605.26315 ↗First posted 25 May 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Direct Preference Optimisation (DPO) is widely used for safety alignment in large language models. However, prior work shows it is brittle and exhibits poor out-of-distribution (OOD) generalisation. In this paper, we investigate whether Curriculum Learning can improve the robustness of DPO-based safety alignment. We propose Staged-Competence, a curriculum-based framework that organises preference data by difficulty, employs competence-based sampling, and progressively updates the reference model during training. Averaged across three model families, Staged-Competence reduces OOD harmful response rates by 16% and jailbreak attack success rates by 20%, while preserving general capabilities with near-zero over-refusal. We further show that Staged-Competence (1) matches baseline safety with only 75% of the training data and (2) yields better separation between safe and unsafe responses. Staged-Competence is agnostic to the policy optimisation loss and can extend to other DPO variants and alignment domains. Our code and data are available at https://github.com/Sandeep5500/curriculum-learning-for-safety.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 29 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

26 of 29 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety26 May
  2. 2025Improving LLM Safety Alignment with Dual-Objective Optimization5 Mar
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun
  4. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal20 Jun
  5. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  6. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 18

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  8. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  9. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  10. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  11. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 286
  12. 2021Curriculum Learning: A Surveycited by 259
6 more
  1. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 202
  2. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspectivecited by 69
  3. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferencescited by 39
  4. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 27
  5. 2026Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamicscited by 6
  6. 2025Teaching According to Talents! Instruction Tuning LLMs with Competence-Aware Curriculum Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Safety Alignment via Balanced Direct Preference Optimization
  2. 2025More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
  3. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
  4. 2026Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment
  5. 2026SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
  6. 2026PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models
  7. 2024Superficial Safety Alignment Hypothesis
  8. 2025Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling