# Off-Policy Evaluation for Large Action Spaces via Policy Convolution

arXiv: https://arxiv.org/abs/2310.15433 · HTML: https://mlsanity.com/paper/2310.15433/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 28 of 28)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Recommendations as Treatments: Debiasing Learning and Evaluation](https://mlsanity.com/paper/1602.05352/) (2016)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016)
- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [The Importance of Pessimism in Fixed-Dataset Policy Optimization](https://mlsanity.com/paper/2009.06799/) (2020)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Doubly robust off-policy evaluation with shrinkage](https://mlsanity.com/paper/1907.09623/) (2019)
- [Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback](https://mlsanity.com/paper/1909.03601/) (2019)
- [Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning](https://mlsanity.com/paper/1909.05850/) (2019)
- [Discretizing Continuous Action Space for On-Policy Optimization](https://mlsanity.com/paper/1901.10500/) (2019)
- [Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding](https://mlsanity.com/paper/2003.05623/) (2020)
- [Off-policy Bandits with Deficient Support](https://mlsanity.com/paper/2006.09438/) (2020)
- [Policy Evaluation and Optimization with Continuous Treatments](https://mlsanity.com/paper/1802.06037/) (2018)
- [Off-Policy Evaluation for Large Action Spaces via Embeddings](https://mlsanity.com/paper/2202.06317/) (2022)
- [Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2002.07729/) (2020)
- [ECLARE: Extreme Classification with Label Graph Correlations](https://mlsanity.com/paper/2108.00261/) (2021)
- [Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling](https://mlsanity.com/paper/2305.08062/) (2023)
- [Infinite Recommendation Networks: A Data-Centric Approach](https://mlsanity.com/paper/2206.02626/) (2022)
- [BLOB : A Probabilistic Model for Recommendation that Combines Organic and Bandit Signals](https://mlsanity.com/paper/2008.12504/) (2020)
- [Learning from eXtreme Bandit Feedback](https://mlsanity.com/paper/2009.12947/) (2020)
- [Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Bandits](https://mlsanity.com/paper/2306.07923/) (2023)
- [Oracle Inequalities for Model Selection in Offline Reinforcement Learning](https://mlsanity.com/paper/2211.02016/) (2022)
- [Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actions](https://mlsanity.com/paper/2210.13373/) (2022)

## Built on by (most cited first, 14 of 14)

- [Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning](https://mlsanity.com/paper/2405.14335/) (2024)
- [POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decomposition](https://mlsanity.com/paper/2402.06151/) (2024)
- [Effective Off-Policy Evaluation and Learning in Contextual Combinatorial Bandits](https://mlsanity.com/paper/2408.11202/) (2024)
- [Bayesian Off-Policy Evaluation and Learning for Large Action Spaces](https://mlsanity.com/paper/2402.14664/) (2024)
- [Cross-Validated Off-Policy Evaluation](https://mlsanity.com/paper/2405.15332/) (2024)
- [Clustering Context in Off-Policy Evaluation](https://mlsanity.com/paper/2502.21304/) (2025)
- [Doubly Robust Fusion of Many Treatments for Policy Learning](https://mlsanity.com/paper/2505.08092/) (2025)
- [Exploiting Similarities in A/B Testing with Off-Policy Estimation](https://mlsanity.com/paper/2506.10677/) (2025)
- [Prompt Optimization with Logged Bandit Data](https://mlsanity.com/paper/2504.02646/) (2025)
- [Position Paper: Why the Shooting in the Dark Method Dominates Recommender Systems Practice; A Call to Abandon Anti-Utopian Thinking](https://mlsanity.com/paper/2402.02152/) (2024)
- [MultiScale Contextual Bandits for Long Term Objectives](https://mlsanity.com/paper/2503.17674/) (2025)
- [Off-Policy Evaluation of Ranking Policies via Embedding-Space User Behavior Modeling](https://mlsanity.com/paper/2506.00446/) (2025)
- [Context-Action Embedding Learning for Off-Policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/2509.00648/) (2025)
- [Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits](https://mlsanity.com/paper/2607.22012/) (2026)

## Abstract (arXiv)

Developing accurate off-policy estimators is crucial for both evaluating and optimizing for new policies. The main challenge in off-policy estimation is the distribution shift between the logging policy that generates data and the target policy that we aim to evaluate. Typically, techniques for correcting distribution shift involve some form of importance sampling. This approach results in unbiased value estimation but often comes with the trade-off of high variance, even in the simpler case of one-step contextual bandits. Furthermore, importance sampling relies on the common support assumption, which becomes impractical when the action space is large. To address these challenges, we introduce the Policy Convolution (PC) family of estimators. These methods leverage latent structure within actions -- made available through action embeddings -- to strategically convolve the logging and target policies. This convolution introduces a unique bias-variance trade-off, which can be controlled by adjusting the amount of convolution. Our experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC, especially when either the action space or policy mismatch becomes large, with gains of up to 5 - 6 orders of magnitude over existing estimators.