mlsanityData to 7 Oct 2026

From r to Q^*: Your Language Model is Secretly a Q-Function

arXiv 2404.12358 ↗First posted 18 Apr 2024cs.LGcited by 242

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning From Human Feedback (RLHF) has been critical to the success of the latest generation of generative AI models. In response to the complex nature of the classical RLHF pipeline, direct alignment algorithms such as Direct Preference Optimization (DPO) have emerged as an alternative approach. Although DPO solves the same objective as the standard RLHF setup, there is a mismatch between the two approaches. Standard RLHF deploys reinforcement learning in a specific token-level MDP, while DPO is derived as a bandit problem in which the whole response of the model is treated as a single arm. In this work we rectify this difference. We theoretically show that we can derive DPO in the token-level MDP as a general inverse Q-learning algorithm, which satisfies the Bellman equation. Using our theoretical results, we provide three concrete empirical insights. First, we show that because of its token level interpretation, DPO is able to perform some type of credit assignment. Next, we prove that under the token level formulation, classical search-based algorithms, such as MCTS, which have recently been applied to the language generation space, are equivalent to likelihood-based search on a DPO policy. Empirically we show that a simple beam search yields meaningful improvement over the base DPO policy. Finally, we show how the choice of reference policy causes implicit rewards to decline during training. We conclude by discussing applications of our work, including information elicitation in multi-turn dialogue, reasoning, agentic applications and end-to-end training of multi-model systems.

Key facts. Cited by 242 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 64 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

36 of 64 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  2. 2024Decoding-time Realignment of Language Models5 Feb
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  9. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  11. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  12. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
20 more
  1. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  2. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  3. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  4. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  5. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  6. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  7. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  8. 2023Controlled Decoding from Language Modelscited by 133
  9. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  10. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  11. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  12. 2021Identifiability in inverse reinforcement learningcited by 55
  13. 2022Models of human preference for learning reward functionscited by 54
  14. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  15. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  16. 2023Coherent Soft Imitation Learningcited by 21
  17. 2023Learning Optimal Advantage from Preferences and Mistaking it for Rewardcited by 18
  18. 2017Learning to Decode for Future Successcited by 16
  19. 2023SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtrackingcited by 13
  20. 2023Let's Reinforce Step by Stepcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 94; 2025: 103; 2026: 45.

242 papers.

Sort:

By area:

  1. 2025Process Reinforcement through Implicit Rewardscited by 478
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  3. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  4. 2024Iterative Reasoning Preference Optimizationcited by 244
  5. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  6. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  7. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  9. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  10. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  11. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  12. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
38 more
  1. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  2. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  3. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  4. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  5. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  6. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 100
  7. 2024Learning Dynamics of LLM Finetuningcited by 93
  8. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  9. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 88
  10. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  11. 2024Process Reward Model with Q-Value Rankingscited by 78
  12. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  13. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  14. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  15. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  16. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
  17. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  18. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  19. 2025Self-rewarding correction for mathematical reasoningcited by 67
  20. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  21. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  22. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  23. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  24. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  25. 2024Direct Multi-Turn Preference Optimization for Language Agentscited by 63
  26. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  27. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  28. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  29. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  30. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  31. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  32. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  33. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
  34. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  35. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  36. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  37. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  38. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45

50 most cited of 242 All 242 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
  4. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
  5. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
  6. 2025A Principled Loss Function for Direct Language Model Alignment
  7. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  8. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study