# Concept-driven Off Policy Evaluation

arXiv: https://arxiv.org/abs/2411.19395 · HTML: https://mlsanity.com/paper/2411.19395/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 28 Nov 2024.

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections](https://mlsanity.com/paper/1906.04733/) (2019)
- [Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation](https://mlsanity.com/paper/1810.12429/) (2018)
- [Minimax Weight and Q-Function Learning for Off-Policy Evaluation](https://mlsanity.com/paper/1910.12809/) (2019)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Promises and Pitfalls of Black-Box Concept Learning Models](https://mlsanity.com/paper/2106.13314/) (2021)
- [GenDICE: Generalized Offline Estimation of Stationary Values](https://mlsanity.com/paper/2002.09072/) (2020)
- [Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling](https://mlsanity.com/paper/1906.03393/) (2019)
- [Concept Bottleneck Model with Additional Unsupervised Concepts](https://mlsanity.com/paper/2202.01459/) (2022)
- [GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values](https://mlsanity.com/paper/2001.11113/) (2020)
- [Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions](https://mlsanity.com/paper/2002.03478/) (2020)
- [Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling](https://mlsanity.com/paper/1910.06508/) (2019)
- [Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?](https://mlsanity.com/paper/2401.13544/) (2024)
- [Representation Balancing MDPs for Off-Policy Policy Evaluation](https://mlsanity.com/paper/1805.09044/) (2018)
- [A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation](https://mlsanity.com/paper/2106.06854/) (2021)
- [Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlap](https://mlsanity.com/paper/2402.08201/) (2024)
- [Offline Policy Optimization with Eligible Actions](https://mlsanity.com/paper/2207.00632/) (2022)
- [Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction](https://mlsanity.com/paper/2212.07486/) (2022)
- [Symbolic Regression for PDEs using Pruned Differentiable Programs](https://mlsanity.com/paper/2303.07009/) (2023)
- [Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluation](https://mlsanity.com/paper/2310.17146/) (2023)
- [Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation](https://mlsanity.com/paper/2111.14272/) (2021)
- [Low Variance Off-policy Evaluation with State-based Importance Sampling](https://mlsanity.com/paper/2212.03932/) (2022)
- [Marginalized Importance Sampling for Off-Environment Policy Evaluation](https://mlsanity.com/paper/2309.01807/) (2023)
- [Robust Policy Optimization with Baseline Guarantees](https://mlsanity.com/paper/1506.04514/) (2015)
- [State Relevance for Off-Policy Evaluation](https://mlsanity.com/paper/2109.06310/) (2021)
- [Medical Image Retrieval via Nearest Neighbor Search on Pre-trained Image Features](https://mlsanity.com/paper/2210.02401/) (2022)
- [Approximate kNN Classification for Biomedical Data](https://mlsanity.com/paper/2012.02149/) (2020)

## Built on by (most cited first, 1 of 1)

- [Quotient DAGs for Off-Policy Evaluation:Forward-Flow Importance Sampling and Exact Slate Propensities](https://mlsanity.com/paper/2605.29500/) (2026)

## Abstract (arXiv)

Evaluating off-policy decisions using batch data poses significant challenges due to limited sample sizes leading to high variance. To improve Off-Policy Evaluation (OPE), we must identify and address the sources of this variance. Recent research on Concept Bottleneck Models (CBMs) shows that using human-explainable concepts can improve predictions and provide better understanding. We propose incorporating concepts into OPE to reduce variance. Our work introduces a family of concept-based OPE estimators, proving that they remain unbiased and reduce variance when concepts are known and predefined. Since real-world applications often lack predefined concepts, we further develop an end-to-end algorithm to learn interpretable, concise, and diverse parameterized concepts optimized for variance reduction. Our experiments with synthetic and real-world datasets show that both known and learned concept-based estimators significantly improve OPE performance. Crucially, we show that, unlike other OPE methods, concept-based estimators are easily interpretable and allow for targeted interventions on specific concepts, further enhancing the quality of these estimators.