# Optimal Policy Learning with Observational Data in Multi-Action Scenarios: Estimation, Risk Preference, and Potential Failures

arXiv: https://arxiv.org/abs/2403.20250 · HTML: https://mlsanity.com/paper/2403.20250/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 47 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Builds on (most cited first, 14 of 14)

- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [Online Learning: A Comprehensive Survey](https://mlsanity.com/paper/1802.02871/) (2018)
- [Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits](https://mlsanity.com/paper/1402.0555/) (2014)
- [Offline Multi-Action Policy Learning: Generalization and Optimization](https://mlsanity.com/paper/1810.04778/) (2018)
- [Imitation Learning: Progress, Taxonomies and Challenges](https://mlsanity.com/paper/2106.12177/) (2021)
- [Self-Supervised Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2006.05779/) (2020)
- [Residual Weighted Learning for Estimating Individualized Treatment Rules](https://mlsanity.com/paper/1508.03179/) (2015)
- [Recursive Partitioning for Personalization using Observational Data](https://mlsanity.com/paper/1608.08925/) (2016)
- [Randomized Gaussian Process Upper Confidence Bound with Tighter Bayesian Regret Bounds](https://mlsanity.com/paper/2302.01511/) (2023)
- [High-Confidence Off-Policy (or Counterfactual) Variance Estimation](https://mlsanity.com/paper/2101.09847/) (2021)
- [Convergence Guarantees for Deep Epsilon Greedy Policy Learning](https://mlsanity.com/paper/2112.03376/) (2021)
- [Deep Upper Confidence Bound Algorithm for Contextual Bandit Ranking of Information Selection](https://mlsanity.com/paper/2110.04127/) (2021)
- [Identifying Effects of Multivalued Treatments](https://mlsanity.com/paper/1805.00057/) (2018)
- [Performance Effectiveness of Multimedia Information Search Using the Epsilon-Greedy Algorithm](https://mlsanity.com/paper/1911.09891/) (2019)

## Built on by (most cited first, 1 of 1)

- [Should I stay or should I show? Learning to selectively disclose information](https://mlsanity.com/paper/2609.39818/) (2026)

## Abstract (arXiv)

This paper deals with optimal policy learning (OPL) with observational data, i.e. data-driven optimal decision-making, in multi-action (or multi-arm) settings, where a finite set of decision options is available. It is organized in three parts, where I discuss respectively: estimation, risk preference, and potential failures. The first part provides a brief review of the key approaches to estimating the reward (or value) function and optimal policy within this context of analysis. Here, I delineate the identification assumptions and statistical properties related to offline optimal policy learning estimators. In the second part, I delve into the analysis of decision risk. This analysis reveals that the optimal choice can be influenced by the decision maker's attitude towards risks, specifically in terms of the trade-off between reward conditional mean and conditional variance. Here, I present an application of the proposed model to real data, illustrating that the average regret of a policy with multi-valued treatment is contingent on the decision-maker's attitude towards risk. The third part of the paper discusses the limitations of optimal data-driven decision-making by highlighting conditions under which decision-making can falter. This aspect is linked to the failure of the two fundamental assumptions essential for identifying the optimal choice: (i) overlapping, and (ii) unconfoundedness. Some conclusions end the paper.