mlsanityData to 7 Oct 2026

Self-Evolved Reward Learning for LLMs

arXiv 2411.00418 ↗First posted 1 Nov 2024cs.CL cs.AIcited by 16

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is a crucial technique for aligning language models with human preferences, playing a pivotal role in the success of conversational models like GPT-4, ChatGPT, and Llama 2. A core challenge in employing RLHF lies in training a reliable reward model (RM), which relies on high-quality labels typically provided by human experts or advanced AI system. These methods can be costly and may introduce biases that affect the language model's responses. As language models improve, human input may become less effective in further enhancing their performance. In this paper, we propose Self-Evolved Reward Learning (SER), a novel approach where the RM generates additional training data to iteratively improve itself. We conducted extensive experiments on multiple datasets such as HH-RLHF and UltraFeedback, using models like Mistral and Llama 3, and compare SER against various baselines. Our results demonstrate that even with limited human-annotated data, learning from self-feedback can robustly enhance RM performance, thereby boosting the capabilities of large language models (LLMs). Resources of this paper can be found at https://aka.ms/ser

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 42 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

23 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  2. 2024Self-Rewarding Language Models18 Jan
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  4. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023Mistral 7Bcited by 3,065
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  7. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  8. 2023Instruction Tuning with GPT-4cited by 676
  9. 2022Large Language Models Can Self-Improvecited by 656
  10. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  11. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  12. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
4 more
  1. 2017Deep Reinforcement Learning: An Overviewcited by 327
  2. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  3. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 80
  4. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 4.

16 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
  3. 2025IPO: Your Language Model is Secretly a Preference Classifiercited by 17
  4. 2025CEC-Zero: Zero-Supervision Character Error Correction with Self-Generated Rewardscited by 10
  5. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9
  6. 2025RLTHF: Targeted Human Feedback for LLM Alignmentcited by 7
  7. 2025Generate, Discriminate, Evolve: Enhancing Context Faithfulness via Fine-Grained Sentence-Level Self-Evolutioncited by 7
  8. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  9. 2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
  10. 2025Self-Improving Model Steeringcited by 3
  11. 2025Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learningcited by 3
  12. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesiscited by 2
4 more
  1. 2026Towards AGI A Pragmatic Approach Towards Self Evolving Agentcited by 2
  2. 2026Self-evolving Embodied AIcited by 2
  3. 2026Cleansing the Artificial Mind: A Self-Reflective Detoxification Framework for Large Language Modelscited by 1
  4. 2026How Can Recommendation Feedback Evolve Agent Memory?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  2. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  3. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  4. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  5. 2024How to Evaluate Reward Models for RLHF
  6. 2024Reward-Robust RLHF in LLMs
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling