mlsanityData to 14 Oct 2026

Nash Learning from Human Feedback

arXiv 2312.00886 ↗First posted 1 Dec 2023stat.ML cs.AI cs.GTcited by 224

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human preferences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.

Key facts. Cited by 224 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 87 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

46 of 87 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  3. 2023Is RLHF More Difficult than Standard RL?25 Jun
  4. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  7. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  8. 2014Convex Optimization: Algorithms and Complexitycited by 575
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  10. 2022Teaching language models to support answers with verified quotescited by 292
  11. 2019A Theory of Regularized Markov Decision Processescited by 286
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
24 more
  1. 2013Optimization, Learning, and Games with Predictable Sequencescited by 188
  2. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  3. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  4. 2015Fast Convergence of Regularized Learning in Gamescited by 150
  5. 2019A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approachcited by 143
  6. 2018Optimistic mirror descent in saddle-point problems: Going the extra (gradient) milecited by 136
  7. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  8. 2017Cycles in adversarial regularized learningcited by 124
  9. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  10. 2018Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimizationcited by 98
  11. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  12. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  13. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  14. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 61
  15. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  16. 2020Reinforcement Learning with Trajectory Feedbackcited by 40
  17. 2023The Consensus Game: Language Model Generation via Equilibrium Searchcited by 36
  18. 2019Optimistic Regret Minimization for Extensive-Form Games via Dilated Distance-Generating Functionscited by 26
  19. 2023How to Scale Your EMAcited by 24
  20. 2018Faster Rates for Convex-Concave Gamescited by 23
  21. 2022On the Limitations of Elo: Real-World Games, are Transitive, not Additivecited by 23
  22. 2016Frank-Wolfe Algorithms for Saddle Point Problemscited by 18
  23. 2018Let's be Honest: An Optimal No-Regret Framework for Zero-Sum Gamescited by 11
  24. 2023Regularized Conventions: Equilibrium Computation as a Model of Pragmatic Reasoningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 90; 2025: 78; 2026: 56.

224 papers.

Sort:

By area:

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  2. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  3. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  4. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  5. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  6. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  8. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  10. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  11. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  12. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
38 more
  1. 2024Critique-out-Loud Reward Modelscited by 106
  2. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  3. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  4. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  5. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  6. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  7. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  8. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  9. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  10. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  11. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  12. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  13. 2024Decoding-time Realignment of Language Modelscited by 67
  14. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 67
  15. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  16. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  17. 2024Learn Your Reference Model for Real Good Alignmentcited by 57
  18. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  19. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  20. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  21. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 51
  22. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 49
  23. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  24. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  25. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  26. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  27. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 44
  28. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 43
  29. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  30. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  31. 2025A Survey of Direct Preference Optimizationcited by 36
  32. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  33. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  34. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  35. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reusecited by 29
  36. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  37. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 27
  38. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 27

50 most cited of 224 All 224 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  6. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  7. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
  8. 2023A Survey of Reinforcement Learning from Human Feedback