# POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning

arXiv: https://arxiv.org/abs/2001.04032 · HTML: https://mlsanity.com/paper/2001.04032/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jan 2020.

## Built on by (most cited first, 30 of 36)

- [Hyperparameter Selection for Offline Reinforcement Learning](https://mlsanity.com/paper/2007.09055/) (2020)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings](https://mlsanity.com/paper/2107.11003/) (2021)
- [A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2111.06784/) (2021)
- [Planning in Observable POMDPs in Quasipolynomial Time](https://mlsanity.com/paper/2201.04735/) (2022)
- [Future-Dependent Value-Based Off-Policy Evaluation in POMDPs](https://mlsanity.com/paper/2207.13081/) (2022)
- [Scalable Decision-Focused Learning in Restless Multi-Armed Bandits with Application to Maternal and Child Health](https://mlsanity.com/paper/2202.00916/) (2022)
- [Decision-Focused Learning without Differentiable Optimization: Learning Locally Optimized Decision Losses](https://mlsanity.com/paper/2203.16067/) (2022)
- [Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data](https://mlsanity.com/paper/2210.08642/) (2022)
- [The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulation](https://mlsanity.com/paper/2106.04240/) (2021)
- [TaskMet: Task-Driven Metric Learning for Model Learning](https://mlsanity.com/paper/2312.05250/) (2023)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Solver-Free Decision-Focused Learning for Linear Optimization Problems](https://mlsanity.com/paper/2505.22224/) (2025)
- [Benchmarking PtO and PnO Methods in the Predictive Combinatorial Optimization Regime](https://mlsanity.com/paper/2311.07633/) (2023)
- [Offline Policy Optimization with Eligible Actions](https://mlsanity.com/paper/2207.00632/) (2022)
- [Identifying Decision Points for Safe and Interpretable Reinforcement Learning in Hypotension Treatment](https://mlsanity.com/paper/2101.03309/) (2021)
- [Learning Under Adversarial and Interventional Shifts](https://mlsanity.com/paper/2103.15933/) (2021)
- [Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning](https://mlsanity.com/paper/2106.03279/) (2021)
- [On Learning Prediction-Focused Mixtures](https://mlsanity.com/paper/2110.13221/) (2021)
- [Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI](https://mlsanity.com/paper/2508.21101/) (2025)
- [Relative Sparsity for Medical Decision Problems](https://mlsanity.com/paper/2211.16566/) (2022)
- [Benchmarking Offline Reinforcement Learning Algorithms for E-Commerce Order Fraud Evaluation](https://mlsanity.com/paper/2212.02620/) (2022)
- [Decision-Focused Evaluation: Analyzing Performance of Deployed Restless Multi-Arm Bandits](https://mlsanity.com/paper/2301.07835/) (2023)
- [Decision-Focused Model-based Reinforcement Learning for Reward Transfer](https://mlsanity.com/paper/2304.03365/) (2023)
- [Efficient Off-Policy Learning for High-Dimensional Action Spaces](https://mlsanity.com/paper/2403.04453/) (2024)
- [Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learning](https://mlsanity.com/paper/2601.10905/) (2026)
- [Extracting Expert's Goals by What-if Interpretable Modeling](https://mlsanity.com/paper/2110.15165/) (2021)
- [Towards Robust Off-Policy Evaluation via Human Inputs](https://mlsanity.com/paper/2209.08682/) (2022)
- [Inference for relative sparsity](https://mlsanity.com/paper/2306.14297/) (2023)
- [Efficient Public Health Intervention Planning Using Decomposition-Based Decision-Focused Learning](https://mlsanity.com/paper/2403.05683/) (2024)

## Abstract (arXiv)

Many medical decision-making tasks can be framed as partially observed Markov decision processes (POMDPs). However, prevailing two-stage approaches that first learn a POMDP and then solve it often fail because the model that best fits the data may not be well suited for planning. We introduce a new optimization objective that (a) produces both high-performing policies and high-quality generative models, even when some observations are irrelevant for planning, and (b) does so in batch off-policy settings that are typical in healthcare, when only retrospective data is available. We demonstrate our approach on synthetic examples and a challenging medical decision-making problem.