mlsanityData to 14 Oct 2026

Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback

arXiv 2505.20075 ↗First posted 26 May 2025cs.AIcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models trained through Reinforcement Learning from AI Feedback (RLAIF) methods frequently suffer from limited generalizability, which hinders the alignment performance of policy models. This challenge stems from various issues, including distribution shift, preference label noise, and mismatch of overly challenging samples with model capacity. In this paper, we aim to enhance the generalizability of reward models through a data-centric approach, driven by the insight that these issues are inherently intertwined from a uniform perspective of data difficulty. Accordingly, we propose a novel framework, Curriculum-RLAIF, which constructs preference pairs with varying difficulty levels and then produces a specific curriculum for reward model training. Comprehensive experimental results suggest that reward models trained with Curriculum-RLAIF achieve improved generalizability, boosting the alignment performance of policy models by a significant margin without incurring additional inference costs compared to various existing non-curriculum baselines. Further analysis and comparison with alternative strategies highlight the superiority of Curriculum-RLAIF in simplicity, efficiency, and effectiveness.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 65 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

26 of 65 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2023Internally Rewarded Reinforcement Learning1 Feb
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 93
  11. 2018Data augmentation instead of explicit regularizationcited by 63
  12. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 43
6 more
  1. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  2. 2023Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logiccited by 35
  3. 2025OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preferencecited by 28
  4. 2025Towards User-level Private Reinforcement Learning with Human Feedbackcited by 14
  5. 2024Dissecting Representation Misalignment in Contrastive Learning via Influence Functioncited by 8
  6. 2024REAL: Response Embedding-based Alignment for LLMscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 20.

32 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  2. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  3. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 52
  4. 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 50
  5. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
  6. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 27
  7. 2026ExpSeek: Self-Triggered Experience Seeking for Web Agentscited by 23
  8. 2026Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrievalcited by 18
  9. 2026ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrievalcited by 17
  10. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  11. 2026TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrievalcited by 15
  12. 2025Understanding LLM Reasoning for Abstractive Summarizationcited by 12
20 more
  1. 2025Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Naturecited by 11
  2. 2025What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoningcited by 11
  3. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  4. 2026FoE: Forest of Errors Makes the First Solution the Best in Large Reasoning Modelscited by 8
  5. 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modelingcited by 8
  6. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinementcited by 6
  7. 2026Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judgescited by 4
  8. 2026From Internal Diagnosis to External Auditing: A VLM-Driven Paradigm for Data-Free Online Backdoor Defensecited by 3
  9. 2026Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Modelscited by 3
  10. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  11. 2026AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learningcited by 3
  12. 2025PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learningcited by 2
  13. 2025Understanding Reinforcement Learning for Model Training, and future directions with GRAPEcited by 2
  14. 2026TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesiscited by 1
  15. 2026Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generationcited by 1
  16. 2026ANCHOR: Abductive Network Construction with Hierarchical Orchestration for Reliable Probability Inference in Large Language Modelscited by 1
  17. 2026Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learningcited by 0
  18. 2026SOLAR: A Self-Optimizing Open-Ended Autonomous Agent for Lifelong Learning and Continual Adaptationcited by 0
  19. 2026What do Reward Models Memorize?cited by 0
  20. 2026LightAIR: Lightweight Action Inversion and Riemannian Rectification for Text-based Person Anomaly Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  2. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
  3. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment
  4. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  5. 2026Reinforcement Learning Towards Broadly and Persistently Beneficial Models
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2026To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization
  8. 2024Multi-objective Reinforcement learning from AI Feedback