# Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling

arXiv: https://arxiv.org/abs/2305.08062 · HTML: https://mlsanity.com/paper/2305.08062/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 56 references that are in the map.
- First posted on arXiv 14 May 2023.

## Builds on (most cited first, 30 of 31)

- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation](https://mlsanity.com/paper/1810.12429/) (2018)
- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [RODE: Learning Roles to Decompose Multi-Agent Tasks](https://mlsanity.com/paper/2010.01523/) (2020)
- [Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1911.06854/) (2019)
- [Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling](https://mlsanity.com/paper/1906.03393/) (2019)
- [Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes](https://mlsanity.com/paper/1908.08526/) (2019)
- [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016)
- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [Doubly robust off-policy evaluation with shrinkage](https://mlsanity.com/paper/1907.09623/) (2019)
- [Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation](https://mlsanity.com/paper/2008.07146/) (2020)
- [Learning Action Representations for Reinforcement Learning](https://mlsanity.com/paper/1902.00183/) (2019)
- [Off-policy Bandits with Deficient Support](https://mlsanity.com/paper/2006.09438/) (2020)
- [Off-Policy Evaluation for Large Action Spaces via Embeddings](https://mlsanity.com/paper/2202.06317/) (2022)
- [Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2002.07729/) (2020)
- [Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling](https://mlsanity.com/paper/1910.06508/) (2019)
- [Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions](https://mlsanity.com/paper/2007.12986/) (2020)
- [Offline Evaluation of Ranking Policies with Click Models](https://mlsanity.com/paper/1804.10488/) (2018)
- [Evaluating the Robustness of Off-Policy Evaluation](https://mlsanity.com/paper/2108.13703/) (2021)
- [Optimal Off-Policy Evaluation from Multiple Logging Policies](https://mlsanity.com/paper/2010.11002/) (2020)
- [Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model](https://mlsanity.com/paper/2202.01562/) (2022)
- [The Natural Language of Actions](https://mlsanity.com/paper/1902.01119/) (2019)
- [Using Experiments to Correct for Selection in Observational Studies](https://mlsanity.com/paper/2006.09676/) (2020)
- [Policy-Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2211.13904/) (2022)
- [Learning from eXtreme Bandit Feedback](https://mlsanity.com/paper/2009.12947/) (2020)
- [Efficient Counterfactual Learning from Bandit Feedback](https://mlsanity.com/paper/1809.03084/) (2018)
- [Control Variates for Slate Off-Policy Evaluation](https://mlsanity.com/paper/2106.07914/) (2021)

## Built on by (most cited first, 30 of 34)

- [Off-Policy Evaluation of Ranking Policies under Diverse User Behavior](https://mlsanity.com/paper/2306.15098/) (2023)
- [Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning](https://mlsanity.com/paper/2405.14335/) (2024)
- [Off-Policy Evaluation for Large Action Spaces via Policy Convolution](https://mlsanity.com/paper/2310.15433/) (2023)
- [On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation](https://mlsanity.com/paper/2307.15053/) (2023)
- [Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation](https://mlsanity.com/paper/2311.18207/) (2023)
- [POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decomposition](https://mlsanity.com/paper/2402.06151/) (2024)
- [Effective Off-Policy Evaluation and Learning in Contextual Combinatorial Bandits](https://mlsanity.com/paper/2408.11202/) (2024)
- [SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluation](https://mlsanity.com/paper/2311.18206/) (2023)
- [Bayesian Off-Policy Evaluation and Learning for Large Action Spaces](https://mlsanity.com/paper/2402.14664/) (2024)
- [Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with It](https://mlsanity.com/paper/2404.15084/) (2024)
- [Cross-Validated Off-Policy Evaluation](https://mlsanity.com/paper/2405.15332/) (2024)
- [Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling](https://mlsanity.com/paper/2406.03434/) (2024)
- [Off-policy estimation with adaptively collected data: the power of online learning](https://mlsanity.com/paper/2411.12786/) (2024)
- [Structured Learning of Compositional Sequential Interventions](https://mlsanity.com/paper/2406.05745/) (2024)
- [Clustering Context in Off-Policy Evaluation](https://mlsanity.com/paper/2502.21304/) (2025)
- [Doubly Robust Fusion of Many Treatments for Policy Learning](https://mlsanity.com/paper/2505.08092/) (2025)
- [Language Model Distillation: A Temporal Difference Imitation Learning Perspective](https://mlsanity.com/paper/2505.20335/) (2025)
- [Off-Policy Evaluation and Learning for Matching Markets](https://mlsanity.com/paper/2507.13608/) (2025)
- [Causal methods for LLM development and evaluation](https://mlsanity.com/paper/2605.25998/) (2026)
- [Prompt Optimization with Logged Bandit Data](https://mlsanity.com/paper/2504.02646/) (2025)
- [A General Framework for Off-Policy Learning with Partially-Observed Reward](https://mlsanity.com/paper/2506.14439/) (2025)
- [Off-Policy Learning in Large Action Spaces: Optimization Matters More Than Estimation](https://mlsanity.com/paper/2509.03456/) (2025)
- [Off-Policy Evaluation for Ranking Policies under Deterministic Logging Policies](https://mlsanity.com/paper/2603.21485/) (2026)
- [Off-Policy Evaluation for Semantic ID Recommenders: Does the Model's Own Code Hierarchy Help?](https://mlsanity.com/paper/2608.28905/) (2026)
- [Counterfactual Inference under Thompson Sampling](https://mlsanity.com/paper/2504.08773/) (2025)
- [Q-function Decomposition with Intervention Semantics with Factored Action Spaces](https://mlsanity.com/paper/2504.21326/) (2025)
- [DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects](https://mlsanity.com/paper/2505.00961/) (2025)
- [Off-Policy Evaluation of Ranking Policies via Embedding-Space User Behavior Modeling](https://mlsanity.com/paper/2506.00446/) (2025)
- [CASP: Support-Aware Offline Policy Selection for Two-Stage Recommender Systems](https://mlsanity.com/paper/2604.23022/) (2026)
- [Offline Contextual Bandits in the Presence of New Actions](https://mlsanity.com/paper/2605.18509/) (2026)

## Abstract (arXiv)

We study off-policy evaluation (OPE) of contextual bandit policies for large discrete action spaces where conventional importance-weighting approaches suffer from excessive variance. To circumvent this variance issue, we propose a new estimator, called OffCEM, that is based on the conjunct effect model (CEM), a novel decomposition of the causal effect into a cluster effect and a residual effect. OffCEM applies importance weighting only to action clusters and addresses the residual causal effect through model-based reward estimation. We show that the proposed estimator is unbiased under a new condition, called local correctness, which only requires that the residual-effect model preserves the relative expected reward differences of the actions within each cluster. To best leverage the CEM and local correctness, we also propose a new two-step procedure for performing model-based estimation that minimizes bias in the first step and variance in the second step. We find that the resulting OffCEM estimator substantially improves bias and variance compared to a range of conventional estimators. Experiments demonstrate that OffCEM provides substantial improvements in OPE especially in the presence of many actions.