mlsanityData to 7 Oct 2026

Self-Rewarding Language Models

arXiv 2401.10020 ↗First posted 18 Jan 2024cs.CL cs.AIcited by 654

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

We posit that to achieve superhuman agents, future models require superhuman feedback in order to provide an adequate training signal. Current approaches commonly train reward models from human preferences, which may then be bottlenecked by human performance level, and secondly these separate frozen reward models cannot then learn to improve during LLM training. In this work, we study Self-Rewarding Language Models, where the language model itself is used via LLM-as-a-Judge prompting to provide its own rewards during training. We show that during Iterative DPO training that not only does instruction following ability improve, but also the ability to provide high-quality rewards to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard, including Claude 2, Gemini Pro, and GPT-4 0613. While there is much left still to explore, this work opens the door to the possibility of models that can continually improve in both axes.

Key facts. Cited by 654 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 51 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

33 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
15 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  7. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  8. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  9. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  10. 2023Self-Alignment with Instruction Backtranslationcited by 149
  11. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  12. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  13. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  14. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
  15. 2022The CRINGE Loss: Learning what language not to modelcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 229; 2025: 202; 2026: 223.

654 papers.

Sort:

By area:

  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  4. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  5. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  6. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  7. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  8. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  10. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  11. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  12. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
38 more
  1. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  2. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  3. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  4. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  5. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  6. 2024Iterative Reasoning Preference Optimizationcited by 244
  7. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  8. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  9. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  10. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  11. 2025Learning to Reason without External Rewardscited by 199
  12. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  13. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  14. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  15. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  16. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  17. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  18. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  19. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  20. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  21. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  22. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  23. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  24. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  25. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  26. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  27. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  28. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Callscited by 121
  29. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
  30. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  31. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  32. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  33. 2024Critique-out-Loud Reward Modelscited by 105
  34. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  35. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  36. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  37. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  38. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94

50 most cited of 654 All 654 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  2. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignment
  3. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  4. 2025A Systematic Analysis of Base Model Choice for Reward Modeling
  5. 2025Generalist Reward Models: Found Inside Large Language Models
  6. 2025Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping
  7. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  8. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems