mlsanityData to 7 Oct 2026

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

arXiv 2404.03715 ↗First posted 4 Apr 2024cs.LG cs.AI cs.CLcited by 176

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

This paper studies post-training large language models (LLMs) using preference feedback from a powerful oracle to help a model iteratively improve over itself. The typical approach for post-training LLMs involves Reinforcement Learning from Human Feedback (RLHF), which traditionally separates reward learning and subsequent policy optimization. However, such a reward maximization approach is limited by the nature of "point-wise" rewards (such as Bradley-Terry model), which fails to express complex intransitive or cyclic preference relations. While advances on RLHF show reward learning and policy optimization can be merged into a single contrastive objective for stability, they yet still remain tethered to the reward maximization framework. Recently, a new wave of research sidesteps the reward maximization presumptions in favor of directly optimizing over "pair-wise" or general preferences. In this paper, we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm that marries the simplicity and stability of contrastive learning with theoretical generality from optimizing general preferences. Because DNO is a batched on-policy algorithm using a regression-based objective, its implementation is straightforward and efficient. Moreover, DNO enjoys monotonic improvement across iterations that help it improve even over a strong teacher (such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 model aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of 33% on AlpacaEval 2.0 (even after controlling for response length), an absolute gain of 26% (7% to 33%) over the initializing model. It outperforms models with far more parameters, including Mistral Large, Self-Rewarding LM (70B parameters), and older versions of GPT-4.

Key facts. Cited by 176 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 58 of its 94 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

58 of 94 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Self-Rewarding Language Models18 Jan
  2. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  4. 2023Axiomatic Preference Modeling for Longform Question Answering2 Dec
  5. 2023Nash Learning from Human Feedback1 Dec
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  7. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  8. 2023Is RLHF More Difficult than Standard RL?25 Jun
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
1 more
  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2015Trust Region Policy Optimizationcited by 2,503
33 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2019Introduction to Online Convex Optimizationcited by 793
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2014Convex Optimization: Algorithms and Complexitycited by 569
  9. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  10. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  11. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  12. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  13. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  14. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  15. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  16. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  17. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  18. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  19. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  20. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  21. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  22. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  23. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  24. 2015Contextual Dueling Banditscited by 101
  25. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  26. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  27. 2022The Role of Coverage in Online Reinforcement Learningcited by 82
  28. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  29. 2021Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discriminationcited by 53
  30. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
  31. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  32. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Gamecited by 27
  33. 2022On the Limitations of Elo: Real-World Games, are Transitive, not Additivecited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 81; 2025: 65; 2026: 30.

176 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2025Process Reinforcement through Implicit Rewardscited by 478
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  5. 2024Iterative Reasoning Preference Optimizationcited by 244
  6. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  7. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  8. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  9. 2024Nemotron-4 340B Technical Reportcited by 143
  10. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  12. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
38 more
  1. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  2. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  3. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 88
  4. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  5. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  6. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  7. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  8. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  9. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  10. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  11. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  12. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  13. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  14. 2024Learn Your Reference Model for Real Good Alignmentcited by 56
  15. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  16. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
  17. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  18. 2025Reinforcement Learning from Human Feedbackcited by 53
  19. 2024Self-Consistency Preference Optimizationcited by 50
  20. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  21. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  22. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  23. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  24. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  25. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  26. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  27. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  28. 2025A Survey of Direct Preference Optimizationcited by 35
  29. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  30. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  31. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignmentcited by 27
  32. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  33. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 26
  34. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
  35. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24
  36. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  37. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  38. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumcited by 22

50 most cited of 176 All 176 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
  2. 2024New Desiderata for Direct Preference Optimization
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024Fine-Tuning Language Models with Reward Learning on Policy
  5. 2024Reinforcement Learning for LLM Post-Training: A Survey
  6. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  7. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
  8. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models