mlsanityData to 14 Oct 2026

State-Constrained Offline Reinforcement Learning

arXiv 2405.14374 ↗First posted 23 May 2024stat.ML cs.AI cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Traditional offline reinforcement learning (RL) methods predominantly operate in a batch-constrained setting. This confines the algorithms to a specific state-action distribution present in the dataset, reducing the effects of distributional shift but restricting the policy to seen actions. In this paper, we alleviate this limitation by introducing state-constrained offline RL, a novel framework that focuses solely on the dataset's state distribution. This approach allows the policy to take high-quality out-of-distribution actions that lead to in-distribution states, significantly enhancing learning potential. The proposed setting not only broadens the learning horizon but also improves the ability to combine different trajectories from the dataset effectively, a desirable property inherent in offline RL. Our research is underpinned by theoretical findings that pave the way for subsequent advancements in this area. Additionally, we introduce StaCQ, a deep learning algorithm that achieves state-of-the-art performance on the D4RL benchmark datasets and aligns with our theoretical propositions. StaCQ establishes a strong baseline for forthcoming explorations in this domain.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 66 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

49 of 66 references.

Closest ancestors · 26

Same direction, newest first

  1. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
  2. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning26 Mar
  3. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learning21 Nov
  4. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  5. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  6. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  9. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  10. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  11. 2021Offline RL Without Off-Policy Evaluation16 Jun
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
14 more
  1. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  4. 2021Offline Reinforcement Learning with Pseudometric Learning2 Mar
  5. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  6. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  7. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  13. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  14. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 23

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  7. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  8. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  9. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  10. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  11. 2020Model-Based Offline Planningcited by 129
  12. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
11 more
  1. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  2. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  3. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  4. 2023Elastic Decision Transformercited by 67
  5. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  6. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
  7. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 29
  8. 2021Metrics and continuity in reinforcement learningcited by 26
  9. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  10. 2020Estimating Q(s,s') with Deep Deterministic Dynamics Gradientscited by 12
  11. 2022Model-based trajectory stitching for improved behavioural cloning and its applicationscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Action-Free Offline-to-Online RL via Discretised State Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021The Least Restriction for Offline Reinforcement Learning
  2. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  3. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  4. 2021Continuous Doubly Constrained Batch Reinforcement Learning
  5. 2023Offline Reinforcement Learning with Imbalanced Datasets
  6. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
  7. 2025Sample-Efficient Policy Constraint Offline Deep Reinforcement Learning based on Sample Filtering
  8. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning