mlsanityData to 7 Oct 2026

Self-Taught Evaluators

arXiv 2408.02666 ↗First posted 5 Aug 2024cs.CL cs.AIcited by 70

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve. In this work, we present an approach that aims to im-prove evaluators without human annotations, using synthetic training data only. Starting from unlabeled instructions, our iterative self-improvement scheme generates contrasting model outputs and trains an LLM-as-a-Judge to produce reasoning traces and final judgments, repeating this training at each new iteration using the improved predictions. Without any labeled preference data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct) from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms commonly used LLM judges such as GPT-4 and matches the performance of the top-performing reward models trained with labeled examples.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 47 references that are in the map. First posted on arXiv 5 Aug 2024.

Builds on

34 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Models12 Oct
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
19 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  5. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  6. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  7. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  8. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  9. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  10. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  11. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  12. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  13. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  14. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  15. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  16. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  17. 2023Learning to Reason and Memorize with Self-Notescited by 30
  18. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Modelscited by 27
  19. 2017Synthetic Data for Social Goodcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 36; 2026: 18.

70 papers.

Sort:

By area:

  1. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  5. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  6. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  7. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  8. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  9. 2025Process Reward Models That Thinkcited by 118
  10. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  11. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  12. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
38 more
  1. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  2. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  3. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  4. 2024Self-Consistency Preference Optimizationcited by 50
  5. 2025Reward Reasoning Modelcited by 47
  6. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  7. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  8. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  9. 2024Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoningcited by 30
  10. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  11. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  12. 2025Learning to Reason via Mixture-of-Thought for Logical Reasoningcited by 20
  13. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
  14. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  15. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 18
  16. 2024Training Language Models to Critique With Multi-agent Feedbackcited by 16
  17. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 16
  18. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  19. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  20. 2025R.I.P.: Better Models by Survival of the Fittest Promptscited by 13
  21. 2026ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agentscited by 13
  22. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 13
  23. 2024Montessori-Instruct: Generate Influential Training Data Tailored for Student Learningcited by 12
  24. 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Modelscited by 11
  25. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
  26. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
  27. 2025Rethinking Reward Models for Multi-Domain Test-Time Scalingcited by 10
  28. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  29. 2025ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Modelscited by 8
  30. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  31. 2025AI & Human Co-Improvement for Safer Co-Superintelligencecited by 8
  32. 2025Think Twice: Branch-and-Rethink Reasoning Reward Modelcited by 7
  33. 2026WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agentscited by 7
  34. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safetycited by 7
  35. 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
  36. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertaintycited by 6
  37. 2025KnowRL: Teaching Language Models to Know What They Knowcited by 5
  38. 2025PRInTS: Reward Modeling for Long-Horizon Information Seekingcited by 5

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-rationalization improves LLM as a fine-grained judge
  2. 2026PERSONAJUDGE: Simulating Individual Human Preference Judgments with Evaluator-Specific Demonstration Data
  3. 2024LLM Evaluators Recognize and Favor Their Own Generations
  4. 2025Do LLM Evaluators Prefer Themselves for a Reason?
  5. 2025Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
  6. 2024Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
  7. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
  8. 2026On Cost-Effective LLM-as-a-Judge Improvement Techniques