# BRPO: Batch Residual Policy Optimization

arXiv: https://arxiv.org/abs/2002.05522 · HTML: https://mlsanity.com/paper/2002.05522/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 8 Feb 2020.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Built on by (most cited first, 25 of 25)

- [Benchmarking and Survey of Explanation Methods for Black Box Models](https://mlsanity.com/paper/2102.13076/) (2021)
- [CF-GNNExplainer: Counterfactual Explanations for Graph Neural Networks](https://mlsanity.com/paper/2102.03322/) (2021)
- [If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques](https://mlsanity.com/paper/2103.01035/) (2021)
- [Robust Counterfactual Explanations for Tree-Based Ensembles](https://mlsanity.com/paper/2207.02739/) (2022)
- [Robust Counterfactual Explanations for Neural Networks With Probabilistic Guarantees](https://mlsanity.com/paper/2305.11997/) (2023)
- [Optimal Counterfactual Explanations in Tree Ensembles](https://mlsanity.com/paper/2106.06631/) (2021)
- [Offline Reinforcement Learning with Soft Behavior Regularization](https://mlsanity.com/paper/2110.07395/) (2021)
- [GLOBE-CE: A Translation-Based Approach for Global Counterfactual Explanations](https://mlsanity.com/paper/2305.17021/) (2023)
- [Formalising the Robustness of Counterfactual Explanations for Neural Networks](https://mlsanity.com/paper/2208.14878/) (2022)
- [Counterfactual Shapley Additive Explanations](https://mlsanity.com/paper/2110.14270/) (2021)
- [A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data](https://mlsanity.com/paper/2107.04680/) (2021)
- [CARE: Coherent Actionable Recourse based on Sound Counterfactual Explanations](https://mlsanity.com/paper/2108.08197/) (2021)
- [Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutions](https://mlsanity.com/paper/2107.09734/) (2021)
- [Counterfactual Explanations for Misclassified Images: How Human and Machine Explanations Differ](https://mlsanity.com/paper/2212.08733/) (2022)
- [Optimization Over Trained Neural Networks: Taking a Relaxing Walk](https://mlsanity.com/paper/2401.03451/) (2024)
- [Global Counterfactual Explanations: Investigations, Implementations and Improvements](https://mlsanity.com/paper/2204.06917/) (2022)
- [Generating collective counterfactual explanations in score-based classification via mathematical optimization](https://mlsanity.com/paper/2310.12822/) (2023)
- [Counterfactual Explanations Using Optimization With Constraint Learning](https://mlsanity.com/paper/2209.10997/) (2022)
- [Explainable Data-Driven Optimization: From Context to Decision and Back Again](https://mlsanity.com/paper/2301.10074/) (2023)
- [Learning Decision Trees and Forests with Algorithmic Recourse](https://mlsanity.com/paper/2406.01098/) (2024)
- [Bayesian Hierarchical Models for Counterfactual Estimation](https://mlsanity.com/paper/2301.08833/) (2023)
- [Computational Tradeoffs of Optimization-Based Bound Tightening in ReLU Networks](https://mlsanity.com/paper/2312.16699/) (2023)
- [Iterative Partial Fulfillment of Counterfactual Explanations: Benefits and Risks](https://mlsanity.com/paper/2303.11111/) (2023)
- [Weak Robust Compatibility Between Learning Algorithms and Counterfactual Explanation Generation Algorithms](https://mlsanity.com/paper/2405.20664/) (2024)
- [Explainable History Distillation by Marked Temporal Point Process](https://mlsanity.com/paper/2311.07797/) (2023)

## Abstract (arXiv)

In batch reinforcement learning (RL), one often constrains a learned policy to be close to the behavior (data-generating) policy, e.g., by constraining the learned action distribution to differ from the behavior policy by some maximum degree that is the same at each state. This can cause batch RL to be overly conservative, unable to exploit large policy changes at frequently-visited, high-confidence states without risking poor performance at sparsely-visited states. To remedy this, we propose residual policies, where the allowable deviation of the learned policy is state-action-dependent. We derive a new for RL method, BRPO, which learns both the policy and allowable deviation that jointly maximize a lower bound on policy performance. We show that BRPO achieves the state-of-the-art performance in a number of tasks.