mlsanityData to 14 Oct 2026

Supported Trust Region Optimization for Offline Reinforcement Learning

arXiv 2311.08935 ↗First posted 15 Nov 2023cs.LG cs.AIcited by 20

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning suffers from the out-of-distribution issue and extrapolation error. Most policy constraint methods regularize the density of the trained policy towards the behavior policy, which is too restrictive in most cases. We propose Supported Trust Region optimization (STR) which performs trust region policy optimization with the policy constrained within the support of the behavior policy, enjoying the less restrictive support constraint. We show that, when assuming no approximation and sampling error, STR guarantees strict policy improvement until convergence to the optimal support-constrained policy in the dataset. Further with both errors incorporated, STR still guarantees safe policy improvement for each step. Empirical results validate the theory of STR and demonstrate its state-of-the-art performance on MuJoCo locomotion domains and much more challenging AntMaze domains.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 41 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

29 of 41 references.

Closest ancestors · 14

Same direction, newest first

  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  2. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  4. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021Offline RL Without Off-Policy Evaluation16 Jun
  8. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2020Critic Regularized Regression26 Jun
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
2 more
  1. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2017Constrained Policy Optimizationcited by 983
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  12. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
3 more
  1. 2018Maximum a Posteriori Policy Optimisationcited by 328
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  3. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 7; 2026: 5.

20 papers.

Sort:

By area:

  1. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  2. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  3. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  4. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
  5. 2025Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environmentscited by 12
  6. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 11
  7. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 11
  8. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  9. 2024Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learningcited by 4
  10. 2025Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategiescited by 4
  11. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
  12. 2024Goal-Conditioned Reinforcement Learning from Sub-Optimal Data on Metric Spacescited by 3
8 more
  1. 2024Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learningcited by 3
  2. 2025Retrosynthesis Planning via Worst-path Policy Optimisation in Tree-structured MDPscited by 2
  3. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  4. 2025ExOSITO: Explainable Off-Policy Learning with Side Information for Intensive Care Unit Blood Test Orderscited by 1
  5. 2026Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experiencecited by 1
  6. 2026Automatic Constraint Policy Optimization based on Continuous Constraint Interpolation Framework for Offline Reinforcement Learningcited by 0
  7. 2026ISEP: Implicit Support Expansion for Offline Reinforcement Learning via Stochastic Policy Optimizationcited by 0
  8. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Embedding Safety into RL: A New Take on Trust Region Methods
  2. 2025Behavior Preference Regression for Offline Reinforcement Learning
  3. 2025SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
  4. 2026Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
  5. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  6. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
  7. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
  8. 2025Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces