mlsanityData to 14 Oct 2026

Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism

arXiv 2305.18438 ↗First posted 29 May 2023cs.LG cs.AI math.OCcited by 69

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

In this paper, we study offline Reinforcement Learning with Human Feedback (RLHF) where we aim to learn the human's underlying reward and the MDP's optimal policy from a set of trajectories induced by human choices. RLHF is challenging for multiple reasons: large state space but limited human feedback, the bounded rationality of human decisions, and the off-policy distribution shift. In this paper, we focus on the Dynamic Discrete Choice (DDC) model for modeling and understanding human choices. DCC, rooted in econometrics and decision theory, is widely used to model a human decision-making process with forward-looking and bounded rationality. We propose a underlineDynamic-underlineChoice-underlinePessimistic-underlinePolicy-underlineOptimization (DCPPO) method. The method involves a three-stage process: The first step is to estimate the human behavior policy and the state-action value function via maximum likelihood estimation (MLE); the second step recovers the human reward function via minimizing Bellman mean squared error using the learned value functions; the third step is to plug in the learned reward and invoke pessimistic value iteration for finding a near-optimal policy. With only single-policy coverage (i.e., optimal policy) of the dataset, we prove that the suboptimality of DCPPO almost matches the classical pessimistic offline RL algorithm in terms of suboptimality's dependency on distribution shift and dimension. To the best of our knowledge, this paper presents the first theoretical guarantees for off-policy offline RLHF with dynamic discrete choice model.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 60 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

36 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  3. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  5. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  6. 2017On Kernelized Multi-armed Banditscited by 352
  7. 2022Teaching language models to support answers with verified quotescited by 292
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  9. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  10. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  11. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  12. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 129
20 more
  1. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  2. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  3. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  4. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  5. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 85
  6. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  7. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 61
  8. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
  9. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  10. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  11. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 44
  12. 2022PAC Reinforcement Learning for Predictive State Representationscited by 43
  13. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  14. 2018Dynamic Assortment Optimization with Changing Contextual Informationcited by 28
  15. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  16. 2023On the Provable Advantage of Unsupervised Pretrainingcited by 22
  17. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  18. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
  19. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guaranteescited by 12
  20. 2022Interactively Learning Preference Constraints in Linear Banditscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 24; 2025: 29; 2026: 9.

69 papers.

Sort:

By area:

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  2. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  3. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  4. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  5. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  6. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  7. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  8. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  9. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  10. 2025A Survey of Research in Large Language Models for Electronic Design Automationcited by 41
  11. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  12. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
38 more
  1. 2024Diffusion Model for Data-Driven Black-Box Optimizationcited by 24
  2. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 22
  3. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 16
  4. 2025CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalizationcited by 16
  5. 2024Uncertainty Aware Learning for Language Model Alignmentcited by 13
  6. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  7. 2024Adversarial Representation Engineering: A General Model Editing Framework for Large Language Modelscited by 12
  8. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  9. 2025CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Modelcited by 9
  10. 2025Lightweight and Direct Document Relevance Optimization for Generative Information Retrievalcited by 9
  11. 2025Context Reasoner: Incentivizing Reasoning Capability for Contextualized Privacy and Safety Compliance via Reinforcement Learningcited by 8
  12. 2023Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generationcited by 7
  13. 2025MR. Judge: Multimodal Reasoner as a Judgecited by 7
  14. 2025Early Timestep Zero-Shot Candidate Selection for Instruction-Guided Image Editingcited by 5
  15. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modelingcited by 5
  16. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedbackcited by 4
  17. 2024Cooperative learning of Pl@ntNet's Artificial Intelligence algorithm: how does it work and how can we improve it?cited by 4
  18. 2025Enhancing Human-Like Responses in Large Language Modelscited by 4
  19. 2025Delta -- Contrastive Decoding Mitigates Text Hallucinations in Large Language Modelscited by 4
  20. 2025Harness Local Rewards for Global Benefits: Effective Text-to-Video Generation Alignment with Patch-level Reward Modelscited by 4
  21. 2025Generative Artificial Intelligence: Evolving Technology, Growing Societal Impact, and Opportunities for Information Systems Researchcited by 4
  22. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  23. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
  24. 2023Beyond Text: Unveiling Multimodal Proficiency of Large Language Models with MultiAPI Benchmarkcited by 3
  25. 2023Think Before You Duel: Understanding Complexities of Preference Learning under Constrained Resourcescited by 3
  26. 2024Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillationcited by 3
  27. 2025On the Limitations of Steering in Language Model Alignmentcited by 3
  28. 2025Understanding Post-Training Structural Changes in Large Language Modelscited by 3
  29. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  30. 2023Contextual Bandits and Imitation Learning via Preference-Based Active Queriescited by 2
  31. 2024Large Language Models for Simultaneous Named Entity Extraction and Spelling Correctioncited by 2
  32. 2025Prompt and Parameter Co-Optimization for Large Language Modelscited by 2
  33. 2025From Passive to Persuasive: Localized Activation Injection for Empathy and Negotiationcited by 2
  34. 2026MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Modelscited by 2
  35. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  36. 2024FARPLS: A Feature-Augmented Robot Trajectory Preference Labeling System to Assist Human Labelers' Preference Elicitationcited by 1
  37. 2024DP-Dueling: Learning from Preference Feedback without Compromising User Privacycited by 1
  38. 2024A Theoretical Survey on Foundation Modelscited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning from Human Feedback
  2. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2023A Survey of Reinforcement Learning from Human Feedback
  5. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis
  6. 2023The History and Risks of Reinforcement Learning and Human Feedback
  7. 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models
  8. 2023Contrastive Preference Learning: Learning from Human Feedback without RL