# Counterfactual Risk Minimization: Learning from Logged Bandit Feedback

arXiv: https://arxiv.org/abs/1502.02362 · HTML: https://mlsanity.com/paper/1502.02362/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2015.

## Built on by (most cited first, 30 of 33)

- [Understanding the role of importance weighting for deep learning](https://mlsanity.com/paper/2103.15209/) (2021)
- [Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning](https://mlsanity.com/paper/2202.09667/) (2022)
- [Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorability](https://mlsanity.com/paper/2110.12343/) (2021)
- [Model-based Constrained MDP for Budget Allocation in Sequential Incentive Marketing](https://mlsanity.com/paper/2303.01049/) (2023)
- [Causal Mediation Analysis with Hidden Confounders](https://mlsanity.com/paper/2102.11724/) (2021)
- [Local Policy Improvement for Recommender Systems](https://mlsanity.com/paper/2212.11431/) (2022)
- [Learning MAX-SAT from Contextual Examples for Combinatorial Optimisation](https://mlsanity.com/paper/2202.03888/) (2022)
- [Offline Policy Optimization with Eligible Actions](https://mlsanity.com/paper/2207.00632/) (2022)
- [Robust Generalization despite Distribution Shift via Minimum Discriminating Information](https://mlsanity.com/paper/2106.04443/) (2021)
- [Parallelizing Contextual Bandits](https://mlsanity.com/paper/2105.10590/) (2021)
- [PoseGU: 3D Human Pose Estimation with Novel Human Pose Generator and Unbiased Learning](https://mlsanity.com/paper/2207.03618/) (2022)
- [ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2407.13163/) (2024)
- [Offline Policy Comparison under Limited Historical Agent-Environment Interactions](https://mlsanity.com/paper/2106.03934/) (2021)
- [Constrained Classification and Policy Learning](https://mlsanity.com/paper/2106.12886/) (2021)
- [Interpretable Personalized Experimentation](https://mlsanity.com/paper/2111.03267/) (2021)
- [The Impact of Batch Learning in Stochastic Linear Bandits](https://mlsanity.com/paper/2202.06657/) (2022)
- [Counterfactual Adversarial Learning with Representation Interpolation](https://mlsanity.com/paper/2109.04746/) (2021)
- [The Bandwagon Effect: Not Just Another Bias](https://mlsanity.com/paper/2206.12701/) (2022)
- [How to select predictive models for causal inference?](https://mlsanity.com/paper/2302.00370/) (2023)
- [Adaptive Experimental Design for Policy Learning](https://mlsanity.com/paper/2401.03756/) (2024)
- [DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward](https://mlsanity.com/paper/2505.07257/) (2025)
- [Case-based off-policy policy evaluation using prototype learning](https://mlsanity.com/paper/2111.11113/) (2021)
- [Towards Robust Off-policy Learning for Runtime Uncertainty](https://mlsanity.com/paper/2202.13337/) (2022)
- [Simulated Contextual Bandits for Personalization Tasks from Recommendation Datasets](https://mlsanity.com/paper/2210.10631/) (2022)
- [Asymptotically Unbiased Off-Policy Policy Evaluation when Reusing Old Data in Nonstationary Environments](https://mlsanity.com/paper/2302.11725/) (2023)
- [Refined PAC-Bayes Bounds for Offline Bandits](https://mlsanity.com/paper/2502.11953/) (2025)
- [An Empirical Analysis on Transparent Algorithmic Exploration in Recommender Systems](https://mlsanity.com/paper/2108.00151/) (2021)
- [Mind the Gap: Optimal and Equitable Encouragement Policies](https://mlsanity.com/paper/2309.07176/) (2023)
- [Increasing Entropy to Boost Policy Gradient Performance on Personalization Tasks](https://mlsanity.com/paper/2310.05324/) (2023)
- [Off-Policy Evaluation for Recommendations with Missing-Not-At-Random Rewards](https://mlsanity.com/paper/2502.08993/) (2025)

## Abstract (arXiv)

We develop a learning principle and an efficient algorithm for batch learning from logged bandit feedback. This learning setting is ubiquitous in online systems (e.g., ad placement, web search, recommendation), where an algorithm makes a prediction (e.g., ad ranking) for a given input (e.g., query) and observes bandit feedback (e.g., user clicks on presented ads). We first address the counterfactual nature of the learning problem through propensity scoring. Next, we prove generalization error bounds that account for the variance of the propensity-weighted empirical risk estimator. These constructive bounds give rise to the Counterfactual Risk Minimization (CRM) principle. We show how CRM can be used to derive a new learning method -- called Policy Optimizer for Exponential Models (POEM) -- for learning stochastic linear rules for structured output prediction. We present a decomposition of the POEM objective that enables efficient stochastic gradient optimization. POEM is evaluated on several multi-label classification problems showing substantially improved robustness and generalization performance compared to the state-of-the-art.