mlsanityData to 14 Oct 2026

Off-Policy Evaluation for Human Feedback

arXiv 2310.07123 ↗First posted 11 Oct 2023cs.LG cs.AIcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Off-policy evaluation (OPE) is important for closing the gap between offline training and evaluation of reinforcement learning (RL), by estimating performance and/or rank of target (evaluation) policies using offline trajectories only. It can improve the safety and efficiency of data collection and policy testing procedures in situations where online deployments are expensive, such as healthcare. However, existing OPE methods fall short in estimating human feedback (HF) signals, as HF may be conditioned over multiple underlying factors and is only sparsely available; as opposed to the agent-defined environmental rewards (used in policy optimization), which are usually determined over parametric functions or distributions. Consequently, the nature of HF signals makes extrapolating accurate OPE estimations to be challenging. To resolve this, we introduce an OPE for HF (OPEHF) framework that revives existing OPE methods in order to accurately evaluate the HF signals. Specifically, we develop an immediate human reward (IHR) reconstruction approach, regularized by environmental knowledge distilled in a latent space that captures the underlying dynamics of state transitions as well as issuing HF signals. Our approach has been tested over two real-world experiments, adaptive in-vivo neurostimulation and intelligent tutoring, as well as in a simulation environment (visual Q&A). Results show that our approach significantly improves the performance toward estimating HF signals accurately, compared to directly applying (variants of) existing OPE methods.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 83 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

44 of 83 references.

Closest ancestors · 2

Same direction, newest first

  1. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 42

  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  3. 2020Mastering Atari with Discrete World Modelscited by 1,007
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  5. 2016Visual Dialogcited by 377
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  8. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2019Batch Policy Learning under Constraintscited by 244
  12. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
30 more
  1. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  2. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  3. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  4. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  5. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  6. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 95
  7. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  8. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  9. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  10. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  11. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  12. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  13. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  14. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  15. 2019Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Samplingcited by 37
  16. 2020Align-RUDDER: Learning From Few Demonstrations by Reward Redistributioncited by 36
  17. 2018Importance Sampling Policy Evaluation with an Estimated Behavior Policycited by 34
  18. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
  19. 2022i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loopscited by 32
  20. 2020Offline Policy Selection under Uncertaintycited by 31
  21. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  22. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 28
  23. 2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
  24. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  25. 2023Reinforcement Learning Tutor Better Supported Lower Performers in a Math Taskcited by 11
  26. 2023Offline Learning of Closed-Loop Deep Brain Stimulation Controllers for Parkinson Disease Treatmentcited by 8
  27. 2021Gradient Importance Learning for Incomplete Observationscited by 7
  28. 2022Giving Feedback on Interactive Student Programs with Meta-Explorationcited by 6
  29. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  30. 2023HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcarecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  2. 2025Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shapingcited by 3
  3. 2025Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AIcited by 3
  4. 2024Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learningcited by 2
  5. 2024Off-Policy Selection for Initiating Human-Centric Experimental Designcited by 0
  6. 2026Formulating Reinforcement Learning for Human-Robot Collaboration through Off-Policy Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
  2. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  3. 2023Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks
  4. 2023Off-policy Evaluation in Doubly Inhomogeneous Environments
  5. 2021Active Offline Policy Selection
  6. 2024Off-Policy Evaluation from Logged Human Feedback
  7. 2024Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation
  8. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning