# Delta-rm OPE: Off-Policy Estimation with Pairs of Policies

arXiv: https://arxiv.org/abs/2405.10024 · HTML: https://mlsanity.com/paper/2405.10024/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 54 references that are in the map.
- First posted on arXiv 16 May 2024.

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Offline A/B testing for Recommender Systems](https://mlsanity.com/paper/1801.07030/) (2018)
- [Doubly robust off-policy evaluation with shrinkage](https://mlsanity.com/paper/1907.09623/) (2019)
- [Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation](https://mlsanity.com/paper/2008.07146/) (2020)
- [Distributionally Robust Counterfactual Risk Minimization](https://mlsanity.com/paper/1906.06211/) (2019)
- [Optimal Baseline Corrections for Off-Policy Contextual Bandits](https://mlsanity.com/paper/2405.05736/) (2024)
- [Ad-load Balancing via Off-policy Learning in a Content Marketplace](https://mlsanity.com/paper/2309.11518/) (2023)
- [Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization](https://mlsanity.com/paper/2305.01522/) (2023)
- [Multi-Objective Recommendation via Multivariate Policy Learning](https://mlsanity.com/paper/2405.02141/) (2024)
- [Practical Bandits: An Industry Perspective](https://mlsanity.com/paper/2302.01223/) (2023)
- [Learning Metrics that Maximise Power for Accelerated A/B-Tests](https://mlsanity.com/paper/2402.03915/) (2024)
- [Offline Recommender System Evaluation under Unobserved Confounding](https://mlsanity.com/paper/2309.04222/) (2023)

## Built on by (most cited first, 9 of 9)

- [Additive Control Variates Dominate Self-Normalisation in Off-Policy Evaluation](https://mlsanity.com/paper/2602.14914/) (2026)
- [Unifying On- and Off-Policy Variance Reduction Methods](https://mlsanity.com/paper/2603.08370/) (2026)
- [The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project](https://mlsanity.com/paper/2603.21354/) (2026)
- [A More Accurate Algorithm Comparison through A/B Testing using Offline Evaluation Methods](https://mlsanity.com/paper/2607.01958/) (2026)
- [A Simple Model to Estimate Sharing Effects in Social Networks](https://mlsanity.com/paper/2409.12203/) (2024)
- [Counterfactual Inference under Thompson Sampling](https://mlsanity.com/paper/2504.08773/) (2025)
- [Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems](https://mlsanity.com/paper/2507.09566/) (2025)
- [Meta Off-Policy Estimation](https://mlsanity.com/paper/2508.07914/) (2025)
- [Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap](https://mlsanity.com/paper/2607.14604/) (2026)

## Abstract (arXiv)

The off-policy paradigm casts recommendation as a counterfactual decision-making task, allowing practitioners to unbiasedly estimate online metrics using offline data. This leads to effective evaluation metrics, as well as learning procedures that directly optimise online success. Nevertheless, the high variance that comes with unbiasedness is typically the crux that complicates practical applications. An important insight is that the difference between policy values can often be estimated with significantly reduced variance, if said policies have positive covariance. This allows us to formulate a pairwise off-policy estimation task: Delta-rm OPE. Delta-rm OPE subsumes the common use-case of estimating improvements of a learnt policy over a production policy, using data collected by a stochastic logging policy. We introduce Delta-rm OPE methods based on the widely used Inverse Propensity Scoring estimator and its extensions. Moreover, we characterise a variance-optimal additive control variate that further enhances efficiency. Simulated, offline, and online experiments show that our methods significantly improve performance for both evaluation and learning tasks.