# A General Framework for Off-Policy Learning with Partially-Observed Reward

arXiv: https://arxiv.org/abs/2506.14439 · HTML: https://mlsanity.com/paper/2506.14439/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Builds on (most cited first, 30 of 30)

- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://mlsanity.com/paper/1804.07931/) (2018)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Machine Learning for Survival Analysis: A Survey](https://mlsanity.com/paper/1708.04649/) (2017)
- [KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems](https://mlsanity.com/paper/2202.10842/) (2022)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Doubly robust off-policy evaluation with shrinkage](https://mlsanity.com/paper/1907.09623/) (2019)
- [Deep Recurrent Survival Analysis](https://mlsanity.com/paper/1809.02403/) (2018)
- [Off-policy Bandits with Deficient Support](https://mlsanity.com/paper/2006.09438/) (2020)
- [Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2002.07729/) (2020)
- [Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling](https://mlsanity.com/paper/2305.08062/) (2023)
- [Bayesian Counterfactual Risk Minimization](https://mlsanity.com/paper/1806.11500/) (2018)
- [Long-term Causal Inference Under Persistent Confounding via Data Combination](https://mlsanity.com/paper/2202.07234/) (2022)
- [Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR prediction](https://mlsanity.com/paper/1907.06558/) (2019)
- [Policy-Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2211.13904/) (2022)
- [Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning](https://mlsanity.com/paper/2405.14335/) (2024)
- [Importance-Weighted Offline Learning Done Right](https://mlsanity.com/paper/2309.15771/) (2023)
- [POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decomposition](https://mlsanity.com/paper/2402.06151/) (2024)
- [Rank and Rate: Multi-task Learning for Recommender Systems](https://mlsanity.com/paper/1807.11698/) (2018)
- [Causal Decision Transformer for Recommender Systems via Offline Reinforcement Learning](https://mlsanity.com/paper/2304.07920/) (2023)
- [Marginal Density Ratio for Off-Policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/2312.01457/) (2023)
- [Long-term Off-Policy Evaluation and Learning](https://mlsanity.com/paper/2404.15691/) (2024)
- [Reward Shaping for User Satisfaction in a REINFORCE Recommender](https://mlsanity.com/paper/2209.15166/) (2022)
- [Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with It](https://mlsanity.com/paper/2404.15084/) (2024)
- [CAB: Continuous Adaptive Blending Estimator for Policy Evaluation and Learning](https://mlsanity.com/paper/1811.02672/) (2018)
- [Indicators of retention in remote digital health studies: A cross-study evaluation of 100,000 participants](https://mlsanity.com/paper/1910.01165/) (2019)
- [IMO3: Interactive Multi-Objective Off-Policy Optimization](https://mlsanity.com/paper/2201.09798/) (2022)
- [Semi-supervised Batch Learning From Logged Data](https://mlsanity.com/paper/2209.07148/) (2022)
- [Pessimistic Off-Policy Multi-Objective Optimization](https://mlsanity.com/paper/2310.18617/) (2023)

## Built on by (most cited first, 1 of 1)

- [Beyond Match Maximization and Fairness: Retention-Optimized Two-Sided Matching](https://mlsanity.com/paper/2602.15752/) (2026)

## Abstract (arXiv)

Off-policy learning (OPL) in contextual bandits aims to learn a decision-making policy that maximizes the target rewards by using only historical interaction data collected under previously developed policies. Unfortunately, when rewards are only partially observed, the effectiveness of OPL degrades severely. Well-known examples of such partial rewards include explicit ratings in content recommendations, conversion signals on e-commerce platforms that are partial due to delay, and the issue of censoring in medical problems. One possible solution to deal with such partial rewards is to use secondary rewards, such as dwelling time, clicks, and medical indicators, which are more densely observed. However, relying solely on such secondary rewards can also lead to poor policy learning since they may not align with the target reward. Thus, this work studies a new and general problem of OPL where the goal is to learn a policy that maximizes the expected target reward by leveraging densely observed secondary rewards as supplemental data. We then propose a new method called Hybrid Policy Optimization for Partially-Observed Reward (HyPeR), which effectively uses the secondary rewards in addition to the partially-observed target reward to achieve effective OPL despite the challenging scenario. We also discuss a case where we aim to optimize not only the expected target reward but also the expected secondary rewards to some extent; counter-intuitively, we will show that leveraging the two objectives is in fact advantageous also for the optimization of only the target reward. Along with statistical analysis of our proposed methods, empirical evaluations on both synthetic and real-world data show that HyPeR outperforms existing methods in various scenarios.