mlsanityData to 7 Oct 2026

CREAM: Consistency Regularized Self-Rewarding Language Models

arXiv 2410.12735 ↗First posted 16 Oct 2024cs.LG cs.CLcited by 35

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent self-rewarding large language models (LLM) have successfully applied LLM-as-a-Judge to iteratively improve the alignment performance without the need of human annotations for preference data. These methods commonly utilize the same LLM to act as both the policy model (which generates responses) and the reward model (which scores and ranks those responses). The ranked responses are then used as preference pairs to train the LLM via direct alignment technologies (e.g. DPO). However, it is noteworthy that throughout this process, there is no guarantee of accuracy in the rewarding and ranking, which is critical for ensuring accurate rewards and high-quality preference data. Empirical results from relatively small LLMs (e.g., 7B parameters) also indicate that improvements from self-rewarding may diminish after several iterations in certain situations, which we hypothesize is due to accumulated bias in the reward system. This bias can lead to unreliable preference data for training the LLM. To address this issue, we first formulate and analyze the generalized iterative preference fine-tuning framework for self-rewarding language model. We then introduce the regularization to this generalized framework to mitigate the overconfident preference labeling in the self-rewarding process. Based on this theoretical insight, we propose a Consistency Regularized sElf-rewarding lAnguage Model (CREAM) that leverages the consistency of rewards across different iterations to regularize the self-rewarding training, helping the model to learn from more reliable preference data. With this explicit regularization, our empirical results demonstrate the superiority of CREAM in improving both reward consistency and alignment performance. The code is publicly available at https://github.com/Raibows/CREAM.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 52 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

40 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
25 more
  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  6. 2022Large Language Models Can Self-Improvecited by 656
  7. 2024Self-Rewarding Language Modelscited by 654
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  9. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  11. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  13. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  14. 2019Confidence Regularized Self-Trainingcited by 418
  15. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  16. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  17. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  18. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  19. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  20. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  21. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  22. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  23. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  24. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  25. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemblecited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 21; 2026: 12.

35 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  3. 2025Self-Adapting Language Modelscited by 71
  4. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 43
  5. 2025A Survey of Direct Preference Optimizationcited by 35
  6. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  7. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  8. 2025Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Modelscited by 13
  9. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 11
  10. 2025Anyprefer: An Agentic Framework for Preference Data Synthesiscited by 10
  11. 2025Large Language Models Have Intrinsic Meta-Cognition, but Need a Good Lenscited by 10
  12. 2025Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoningcited by 10
23 more
  1. 2025Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?cited by 9
  2. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  3. 2025Investigating CoT Monitorability in Large Reasoning Modelscited by 8
  4. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectorscited by 6
  5. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 6
  6. 2025TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoningcited by 6
  7. 2025Representation Consistency for Accurate and Coherent LLM Answer Aggregationcited by 5
  8. 2025Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Futurecited by 5
  9. 2025DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learningcited by 5
  10. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
  11. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  12. 2025Reward Model Routing in Alignmentcited by 3
  13. 2026DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinationscited by 3
  14. 2026When Context Returns: Toward Robust Internalization in On-Policy Distillationcited by 3
  15. 2026Can LLMs Learn to Reason Robustly under Noisy Supervision?cited by 2
  16. 2026R2V Agent: Teaching SLMs When to Ask for Helpcited by 2
  17. 2025Value-Free Policy Optimization via Reward Partitioningcited by 1
  18. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 1
  19. 2025Token-Level Inference-Time Alignment for Vision-Language Modelscited by 0
  20. 2026Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMscited by 0
  21. 2026Unsupervised Post-Training of Foundation Models: A Surveycited by 0
  22. 2026J-Zero: Unified Challenger--Solver--Judge Self-Evolution from Zero Datacited by 0
  23. 2026Multi-LLM Collaborative Alignment via Stackelberg Gamescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
  3. 2026Steerable Cultural Preference Optimization of Reward Models
  4. 2024Bootstrapping Language Models with DPO Implicit Rewards
  5. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  6. 2024Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation
  7. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  8. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment