mlsanityData to 7 Oct 2026

Off-Policy Evaluation from Logged Human Feedback

arXiv 2406.10030 ↗First posted 14 Jun 2024cs.LG stat.MLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning from human feedback has been central to recent advances in artificial intelligence and machine learning. Since the collection of human feedback is costly, a natural question to ask is if the new feedback always needs to collected. Or could we evaluate a new model with the human feedback on responses of another model? This motivates us to study off-policy evaluation from logged human feedback. We formalize the problem, propose both model-based and model-free estimators for policy values, and show how to optimize them. We analyze unbiasedness of our estimators and evaluate them empirically. Our estimators can predict the absolute values of evaluated policies, rank them, and be optimized.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 34 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

13 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan

Other references · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  6. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  7. 2023Large Language Models as General Pattern Machinescited by 196
  8. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  9. 2016Unbiased Learning-to-Rank with Biased Feedbackcited by 155
  10. 2016Off-policy evaluation for slate recommendationcited by 102
  11. 2018Offline Evaluation of Ranking Policies with Click Modelscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  2. 2024OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Modelscited by 12
  3. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  4. 2026OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  2. 2023Off-Policy Evaluation for Human Feedback
  3. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism
  4. 2025Reinforcement Learning from Human Feedback
  5. 2023Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks
  6. 2021Active Offline Policy Selection
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators