mlsanityData to 7 Oct 2026

Projected State-action Balancing Weights for Offline Reinforcement Learning

arXiv 2109.04640 ↗First posted 10 Sep 2021cs.LG stat.MEcited by 16

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline policy evaluation (OPE) is considered a fundamental and challenging problem in reinforcement learning (RL). This paper focuses on the value estimation of a target policy based on pre-collected data generated from a possibly different policy, under the framework of infinite-horizon Markov decision processes. Motivated by the recently developed marginal importance sampling method in RL and the covariate balancing idea in causal inference, we propose a novel estimator with approximately projected state-action balancing weights for the policy value estimation. We obtain the convergence rate of these weights and show that the proposed value estimator is semi-parametric efficient under technical conditions. In terms of asymptotics, our results scale with both the number of trajectories and the number of decision points at each trajectory. As such, consistency can still be achieved with a limited number of subjects when the number of decision points diverges. In addition, we develop a necessary and sufficient condition for establishing the well-posedness of the Bellman operator in the off-policy setting, which characterizes the difficulty of OPE and may be of independent interest. Numerical experiments demonstrate the promising performance of our proposed estimator.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 79 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

24 of 79 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 22

  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  4. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  5. 2019Batch Policy Learning under Constraintscited by 242
  6. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  7. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  8. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  9. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  10. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  11. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  12. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
10 more
  1. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  2. 2016Approximate Residual Balancing: De-Biased Inference of Average Treatment Effects in High Dimensionscited by 50
  3. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  4. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  5. 2016Generalized Optimal Matching Methods for Causal Inferencecited by 39
  6. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  7. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  8. 2020Statistically Efficient Off-Policy Policy Gradientscited by 24
  9. 2017Minimal Dispersion Approximately Balancing Weights: Asymptotic Properties and Practical Considerationscited by 18
  10. 2020Personalized Policy Learning using Longitudinal Mobile Health Datacited by 10

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 3; 2024: 3; 2025: 5.

16 papers.

Sort:

By area:

  1. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  2. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  3. 2022Conformal Off-policy Predictioncited by 14
  4. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  5. 2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
  6. 2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
  7. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  8. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  9. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  10. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  11. 2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
  12. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 2
4 more
  1. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  2. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  3. 2025Fitted Q-Evaluation without Bellman Completeness via Occupancy Weightingcited by 1
  4. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes
  2. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  3. 2020Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies
  4. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  5. 2026Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning
  6. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  7. 2023Conservative State Value Estimation for Offline Reinforcement Learning
  8. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples