mlsanityData to 14 Oct 2026

West-of-N: Synthetic Preferences for Self-Improving Reward Models

arXiv 2401.12086 ↗First posted 22 Jan 2024cs.CL cs.AI cs.LGcited by 28

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

The success of reinforcement learning from human feedback (RLHF) in language model alignment is strongly dependent on the quality of the underlying reward model. In this paper, we present a novel approach to improve reward model quality by generating synthetic preference data, thereby augmenting the training dataset with on-policy, high-quality preference pairs. Motivated by the promising results of Best-of-N sampling strategies in language model training, we extend their application to reward model training. This results in a self-training strategy to generate preference pairs by selecting the best and worst candidates in a pool of responses to a given query. Empirically, we find that this approach improves the performance of any reward model, with an effect comparable to the addition of a similar quantity of human preference data. This work opens up new avenues of research for improving RLHF for language model alignment, by offering synthetic preference generation as a solution to reward modeling challenges.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 34 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

25 of 34 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  8. 2023PaLM 2 Technical Reportcited by 1,162
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  10. 2020Rethinking Pre-training and Self-trainingcited by 352
  11. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  12. 2019Revisiting Self-Training for Neural Sequence Generationcited by 155
5 more
  1. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 127
  2. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  3. 2022Self-Training: A Surveycited by 86
  4. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  5. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 13; 2026: 5.

28 papers.

Sort:

By area:

  1. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  3. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  4. 2024Self-Taught Evaluatorscited by 72
  5. 2025Reinforcement Learning from Human Feedbackcited by 58
  6. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  7. 2024Self-rationalization improves LLM as a fine-grained judgecited by 27
  8. 2025Robust Reward Modeling via Causal Rubricscited by 19
  9. 2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
  10. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  11. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 15
  12. 2024Speculative Coreset Selection for Task-Specific Fine-tuningcited by 10
16 more
  1. 2025Frictional Agent Alignment Framework: Slow Down and Don't Break Thingscited by 10
  2. 2025Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomescited by 7
  3. 2025Expert Preference-based Evaluation of Automated Related Work Generationcited by 6
  4. 2025Debiasing Reward Models by Representation Learning with Guaranteescited by 6
  5. 2025Enterprise Large Language Model Evaluation Benchmarkcited by 5
  6. 2024MetaRM: Shifted Distributions Alignment via Meta-Learningcited by 4
  7. 2024TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Modelscited by 4
  8. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  9. 2025Mutual-Taught for Co-adapting Policy and Reward Modelscited by 3
  10. 2024ToW: Thoughts of Words Improve Reasoning in Large Language Modelscited by 2
  11. 2025Retrieval-Augmented Fine-Tuning With Preference Optimization For Visual Program Generationcited by 2
  12. 2026Fork-and-Flush: Escaping Idea Basins in Autoresearch Agentscited by 1
  13. 2026ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignmentcited by 0
  14. 2026MARS: Margin and Semantic-Aware Data Augmentation for Reward Modelingcited by 0
  15. 2026MindfulAgents: Personalizing Mindfulness Meditation via an Expert-Aligned Multi-Agent Systemcited by 0
  16. 2026ABC-Align: Prediction-Powered Alignment with Adaptive Bias Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Fine-Tuning Language Models with Reward Learning on Policy
  2. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  4. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
  5. 2026The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
  6. 2025Online Preference Alignment for Language Models via Count-based Exploration
  7. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  8. 2024Towards Comprehensive Preference Data Collection for Reward Modeling