# Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actions

arXiv: https://arxiv.org/abs/2210.13373 · HTML: https://mlsanity.com/paper/2210.13373/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 40 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1911.06854/) (2019)
- [Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation](https://mlsanity.com/paper/2008.07146/) (2020)
- [A Survey on Metric Learning for Feature Vectors and Structured Data](https://mlsanity.com/paper/1306.6709/) (2013)
- [Policy Evaluation and Optimization with Continuous Treatments](https://mlsanity.com/paper/1802.06037/) (2018)
- [Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2002.07729/) (2020)
- [Dynamic Pricing with Demand Covariates](https://mlsanity.com/paper/1604.07463/) (2016)
- [Deep Jump Learning for Off-Policy Evaluation in Continuous Treatment Settings](https://mlsanity.com/paper/2010.15963/) (2020)

## Built on by (most cited first, 6 of 6)

- [Off-Policy Evaluation for Large Action Spaces via Policy Convolution](https://mlsanity.com/paper/2310.15433/) (2023)
- [Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies](https://mlsanity.com/paper/2405.18792/) (2024)
- [Off-policy estimation with adaptively collected data: the power of online learning](https://mlsanity.com/paper/2411.12786/) (2024)
- [Breaking Determinism: Stochastic Modeling for Reliable Off-Policy Evaluation in Ad Auctions](https://mlsanity.com/paper/2512.03354/) (2025)
- [Variational Weighting for Kernel Density Ratios](https://mlsanity.com/paper/2311.03001/) (2023)
- [Context-Action Embedding Learning for Off-Policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/2509.00648/) (2025)

## Abstract (arXiv)

We consider local kernel metric learning for off-policy evaluation (OPE) of deterministic policies in contextual bandits with continuous action spaces. Our work is motivated by practical scenarios where the target policy needs to be deterministic due to domain requirements, such as prescription of treatment dosage and duration in medicine. Although importance sampling (IS) provides a basic principle for OPE, it is ill-posed for the deterministic target policy with continuous actions. Our main idea is to relax the target policy and pose the problem as kernel-based estimation, where we learn the kernel metric in order to minimize the overall mean squared error (MSE). We present an analytic solution for the optimal metric, based on the analysis of bias and variance. Whereas prior work has been limited to scalar action spaces or kernel bandwidth selection, our work takes a step further being capable of vector action spaces and metric optimization. We show that our estimator is consistent, and significantly reduces the MSE compared to baseline OPE methods through experiments on various domains.