# Short-Long Policy Evaluation with Novel Actions

arXiv: https://arxiv.org/abs/2407.03674 · HTML: https://mlsanity.com/paper/2407.03674/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Batch Policy Learning under Constraints](https://mlsanity.com/paper/1903.08738/) (2019)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [Composing Parameter-Efficient Modules with Arithmetic Operations](https://mlsanity.com/paper/2306.14870/) (2023)
- [Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees](https://mlsanity.com/paper/1807.03858/) (2018)
- [AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning](https://mlsanity.com/paper/2210.17451/) (2022)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [A Game Theoretic Framework for Model Based Reinforcement Learning](https://mlsanity.com/paper/2004.07804/) (2020)
- [Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity](https://mlsanity.com/paper/1909.03539/) (2019)
- [Meta-learning framework with applications to zero-shot time-series forecasting](https://mlsanity.com/paper/2002.02887/) (2020)
- [Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health](https://mlsanity.com/paper/1912.13088/) (2019)
- [Best arm identification in multi-armed bandits with delayed feedback](https://mlsanity.com/paper/1803.10937/) (2018)
- [Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework](https://mlsanity.com/paper/2002.01711/) (2020)
- [A One-step Approach to Covariate Shift Adaptation](https://mlsanity.com/paper/2007.04043/) (2020)
- [Representation Balancing MDPs for Off-Policy Policy Evaluation](https://mlsanity.com/paper/1805.09044/) (2018)
- [Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay](https://mlsanity.com/paper/2307.09943/) (2023)
- [Inferring the Long-Term Causal Effects of Long-Term Treatments from Short-Term Experiments](https://mlsanity.com/paper/2311.08527/) (2023)
- [Optimization of anemia treatment in hemodialysis patients via reinforcement learning](https://mlsanity.com/paper/1509.03977/) (2015)
- [Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix](https://mlsanity.com/paper/2311.11922/) (2023)
- [A Reinforcement Learning Approach to Estimating Long-term Treatment Effects](https://mlsanity.com/paper/2210.07536/) (2022)
- [Learning to Transfer Dynamic Models of Underactuated Soft Robotic Hands](https://mlsanity.com/paper/2005.10418/) (2020)

## Abstract (arXiv)

From incorporating LLMs in education, to identifying new drugs and improving ways to charge batteries, innovators constantly try new strategies in search of better long-term outcomes for students, patients and consumers. One major bottleneck in this innovation cycle is the amount of time it takes to observe the downstream effects of a decision policy that incorporates new interventions. The key question is whether we can quickly evaluate long-term outcomes of a new decision policy without making long-term observations. Organizations often have access to prior data about past decision policies and their outcomes, evaluated over the full horizon of interest. Motivated by this, we introduce a new setting for short-long policy evaluation for sequential decision making tasks. Our proposed methods significantly outperform prior results on simulators of HIV treatment, kidney dialysis and battery charging. We also demonstrate that our methods can be useful for applications in AI safety by quickly identifying when a new decision policy is likely to have substantially lower performance than past policies.