mlsanityData to 7 Oct 2026

Residuals-based Offline Reinforcement Learning

arXiv 2604.01378 ↗First posted 1 Apr 2026cs.LG math.OCcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) has received increasing attention for learning policies from previously collected data without interaction with the real environment, which is particularly important in high-stakes applications. While a growing body of work has developed offline RL algorithms, these methods often rely on restrictive assumptions about data coverage and suffer from distribution shift. In this paper, we propose a residuals-based offline RL framework for general state and action spaces. Specifically, we define a residuals-based Bellman optimality operator that explicitly incorporates estimation error in learning transition dynamics into policy optimization by leveraging empirical residuals. We show that this Bellman operator is a contraction mapping and identify conditions under which its fixed point is asymptotically optimal and possesses finite-sample guarantees. We further develop a residuals-based offline deep Q-learning (DQN) algorithm. Using a stochastic CartPole environment, we demonstrate the effectiveness of our residuals-based offline DQN algorithm.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 10 of its 19 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

10 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees5 Oct
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 7

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  6. 2022Data-Driven Sample Average Approximation with Covariate Informationcited by 14
  7. 2024Residuals-Based Contextual Distributionally Robust Optimization with Decision-Dependent Uncertainty: Theoretical Guarantees and Decomposition Algorithmcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learning
  2. 2026Deep Weighted Bellman Residual Minimization for Q^* Estimation
  3. 2025Stability and Generalization for Bellman Residuals
  4. 2021The Least Restriction for Offline Reinforcement Learning
  5. 2026CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning
  6. 2022Learning Bellman Complete Representations for Offline Policy Evaluation
  7. 2019An Optimistic Perspective on Offline Reinforcement Learning
  8. 2021A Minimalist Approach to Offline Reinforcement Learning