mlsanityData to 7 Oct 2026

BRPO: Batch Residual Policy Optimization

arXiv 2002.05522 ↗First posted 8 Feb 2020cs.LG cs.AI stat.MLcited by 25

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In batch reinforcement learning (RL), one often constrains a learned policy to be close to the behavior (data-generating) policy, e.g., by constraining the learned action distribution to differ from the behavior policy by some maximum degree that is the same at each state. This can cause batch RL to be overly conservative, unable to exploit large policy changes at frequently-visited, high-confidence states without risking poor performance at sparsely-visited states. To remedy this, we propose residual policies, where the allowable deviation of the learned policy is state-action-dependent. We derive a new for RL method, BRPO, which learns both the policy and allowable deviation that jointly maximize a lower bound on policy performance. We show that BRPO achieves the state-of-the-art performance in a number of tasks.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 8 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 9; 2022: 5; 2023: 8; 2024: 3.

25 papers.

Sort:

By area:

  1. 2021Benchmarking and Survey of Explanation Methods for Black Box Modelscited by 136
  2. 2021CF-GNNExplainer: Counterfactual Explanations for Graph Neural Networkscited by 118
  3. 2021If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniquescited by 82
  4. 2022Robust Counterfactual Explanations for Tree-Based Ensemblescited by 49
  5. 2023Robust Counterfactual Explanations for Neural Networks With Probabilistic Guaranteescited by 45
  6. 2021Optimal Counterfactual Explanations in Tree Ensemblescited by 36
  7. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  8. 2023GLOBE-CE: A Translation-Based Approach for Global Counterfactual Explanationscited by 27
  9. 2022Formalising the Robustness of Counterfactual Explanations for Neural Networkscited by 24
  10. 2021Counterfactual Shapley Additive Explanationscited by 22
  11. 2021A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Datacited by 19
  12. 2021CARE: Coherent Actionable Recourse based on Sound Counterfactual Explanationscited by 17
13 more
  1. 2021Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutionscited by 16
  2. 2022Counterfactual Explanations for Misclassified Images: How Human and Machine Explanations Differcited by 12
  3. 2024Optimization Over Trained Neural Networks: Taking a Relaxing Walkcited by 10
  4. 2022Global Counterfactual Explanations: Investigations, Implementations and Improvementscited by 9
  5. 2023Generating collective counterfactual explanations in score-based classification via mathematical optimizationcited by 9
  6. 2022Counterfactual Explanations Using Optimization With Constraint Learningcited by 8
  7. 2023Explainable Data-Driven Optimization: From Context to Decision and Back Againcited by 7
  8. 2024Learning Decision Trees and Forests with Algorithmic Recoursecited by 7
  9. 2023Bayesian Hierarchical Models for Counterfactual Estimationcited by 6
  10. 2023Computational Tradeoffs of Optimization-Based Bound Tightening in ReLU Networkscited by 6
  11. 2023Iterative Partial Fulfillment of Counterfactual Explanations: Benefits and Riskscited by 1
  12. 2024Weak Robust Compatibility Between Learning Algorithms and Counterfactual Explanation Generation Algorithmscited by 1
  13. 2023Explainable History Distillation by Marked Temporal Point Processcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Continuous Doubly Constrained Batch Reinforcement Learning
  2. 2020Provably Good Batch Reinforcement Learning Without Great Exploration
  3. 2025Residual Policy Gradient: A Reward View of KL-regularized Objective
  4. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policies
  5. 2019Batch Policy Learning under Constraints
  6. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
  7. 2020Critic Regularized Regression
  8. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators