mlsanityData to 7 Oct 2026

Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks

arXiv 2310.10556 ↗First posted 16 Oct 2023cs.LG stat.MLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

A recently popular approach to solving reinforcement learning is with data from human preferences. In fact, human preference data are now used with classic reinforcement learning algorithms such as actor-critic methods, which involve evaluating an intermediate policy over a reward learned from human preference data with distribution shift, known as off-policy evaluation (OPE). Such algorithm includes (i) learning reward function from human preference dataset, and (ii) learning expected cumulative reward of a target policy. Despite the huge empirical success, existing OPE methods with preference data often lack theoretical understanding and rely heavily on heuristics. In this paper, we study the sample efficiency of OPE with human preference and establish a statistical guarantee for it. Specifically, we approach OPE by learning the value function by fitted-Q-evaluation with a deep neural network. By appropriately selecting the size of a ReLU network, we show that one can leverage any low-dimensional manifold structure in the Markov decision process and obtain a sample-efficient estimator without suffering from the curse of high data ambient dimensionality. Under the assumption of high reward smoothness, our results textitalmost align with the classical OPE results with observable reward data. To the best of our knowledge, this is the first result that establishes a textitprovably efficient guarantee for off-policy evaluation with RLHF.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 88 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

56 of 88 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback3 Aug
  2. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  4. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  5. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  6. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  7. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback11 Aug
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  9. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  7. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  8. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  9. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  10. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  11. 2022Teaching language models to support answers with verified quotescited by 290
  12. 2021Recursively Summarizing Books with Human Feedbackcited by 287
34 more
  1. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  3. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  4. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  5. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  6. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  7. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  8. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  9. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  10. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  11. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  12. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  13. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  14. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  15. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  16. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  17. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  18. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  19. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 59
  20. 2018On the Intrinsic Dimensionality of Image Representationscited by 57
  21. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  22. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  23. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  24. 2020Off-Policy Evaluation and Learning for External Validity under a Covariate Shiftcited by 26
  25. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  26. 2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
  27. 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
  28. 2013Credal Networks under Maximum Entropycited by 13
  29. 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
  30. 2022Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficientcited by 12
  31. 2022Interactively Learning Preference Constraints in Linear Banditscited by 11
  32. 2020Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approachcited by 10
  33. 2023Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspectivecited by 4
  34. 2017Inverse Reinforcement Learning with Conditional Choice Probabilitiescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Data
  2. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  3. 2025Design Considerations in Offline Preference-based RL
  4. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  5. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  6. 2024Preference Elicitation for Offline Reinforcement Learning
  7. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  8. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency