mlsanityData to 7 Oct 2026

Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge

arXiv 2407.19594 ↗First posted 28 Jul 2024cs.CL cs.AIcited by 202

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model's ability to judge em and follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision.

Key facts. Cited by 202 papers in the map (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 41 of its 53 references that are in the map. First posted on arXiv 28 Jul 2024.

Builds on

41 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  2. 2024Self-Rewarding Language Models18 Jan

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2020Learning to summarize from human feedbackcited by 2,562
27 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  4. 2022Red Teaming Language Models with Language Modelscited by 983
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  9. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  11. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  12. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  13. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  14. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  15. 2022Self-critiquing models for assisting human evaluatorscited by 372
  16. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  17. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  18. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  19. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  20. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  21. 2023Self-Alignment with Instruction Backtranslationcited by 149
  22. 2024LLM Critics Help Catch LLM Bugscited by 138
  23. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  24. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  25. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  26. 2024Following Length Constraints in Instructionscited by 45
  27. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 38; 2025: 103; 2026: 61.

202 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  6. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
  7. 2024Generative Reward Modelscited by 148
  8. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  9. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  10. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  11. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  12. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 70
38 more
  1. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  2. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  3. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  4. 2025Reinforcement Learning from Human Feedbackcited by 53
  5. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  6. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  7. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  8. 2025Reward Reasoning Modelcited by 47
  9. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  10. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 43
  11. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  12. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  13. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  14. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30
  15. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 30
  16. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 30
  17. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  18. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 29
  19. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  20. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  21. 2024Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learningcited by 27
  22. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  23. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  24. 2024Self-rationalization improves LLM as a fine-grained judgecited by 25
  25. 2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
  26. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 25
  27. 2024SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Modelscited by 23
  28. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Taskscited by 22
  29. 2025Process-based Self-Rewarding Language Modelscited by 21
  30. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  31. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  32. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  33. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  34. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  35. 2024Better Instruction-Following Through Minimum Bayes Riskcited by 19
  36. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 19
  37. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  38. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17

50 most cited of 202 All 202 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  2. 2023Reasons to Reject? Aligning Language Models with Judgments
  3. 2024CREAM: Consistency Regularized Self-Rewarding Language Models
  4. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Do
  5. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
  6. 2024Self-Judge: Selective Instruction Following with Alignment Self-Evaluation
  7. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
  8. 2026PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards