mlsanityData to 7 Oct 2026

Supported Policy Optimization for Offline Reinforcement Learning

arXiv 2202.06239 ↗First posted 13 Feb 2022cs.LG cs.AIcited by 73

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Policy constraint methods to offline reinforcement learning (RL) typically utilize parameterization or regularization that constrains the policy to perform actions within the support set of the behavior policy. The elaborative designs of parameterization methods usually intrude into the policy networks, which may bring extra inference cost and cannot take full advantage of well-established online methods. Regularization methods reduce the divergence between the learned policy and the behavior policy, which may mismatch the inherent density-based definition of support set thereby failing to avoid the out-of-distribution actions effectively. This paper presents Supported Policy OpTimization (SPOT), which is directly derived from the theoretical formalization of the density-based support constraint. SPOT adopts a VAE-based density estimator to explicitly model the support set of behavior policy and presents a simple but effective density-based regularization term, which can be plugged non-intrusively into off-the-shelf off-policy RL algorithms. SPOT achieves the state-of-the-art performance on standard benchmarks for offline RL. Benefiting from the pluggable design, offline pretrained models from SPOT can also be applied to perform online fine-tuning seamlessly.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 73 references that are in the map. First posted on arXiv 13 Feb 2022.

Builds on

38 of 73 references.

Closest ancestors · 16

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Soft Behavior Regularization14 Oct
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
  4. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  5. 2021Offline RL Without Off-Policy Evaluation16 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  8. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  9. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  10. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  11. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  12. 2020Critic Regularized Regression26 Jun
4 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2019An Introduction to Variational Autoencoderscited by 997
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
10 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  5. 2015Importance Weighted Autoencoderscited by 454
  6. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 306
  7. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  8. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  9. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  10. 2021Active Offline Policy Selectioncited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 22; 2024: 19; 2025: 19; 2026: 11.

73 papers.

Sort:

By area:

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  3. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  4. 2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 67
  5. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
  6. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  7. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  8. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 33
  9. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  10. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  11. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  12. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
38 more
  1. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  2. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  3. 2023Supported Trust Region Optimization for Offline Reinforcement Learningcited by 20
  4. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  5. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  6. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
  7. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
  8. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 16
  9. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operatorscited by 14
  10. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
  11. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learningcited by 13
  12. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  13. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  14. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learningcited by 10
  15. 2023Goal-Conditioned Predictive Coding for Offline Reinforcement Learningcited by 10
  16. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 10
  17. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  18. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  19. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  20. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  21. 2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7
  22. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  23. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  24. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  25. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RLcited by 5
  26. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  27. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency modelcited by 4
  28. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learningcited by 4
  29. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
  30. 2024Context-Former: Stitching via Latent Conditioned Sequence Modelingcited by 3
  31. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learningcited by 3
  32. 2025Average Reward Reinforcement Learning for Wireless Radio Resource Managementcited by 3
  33. 2025Adaptive Replay Buffer for Offline-to-Online Reinforcement Learningcited by 3
  34. 2024Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidancecited by 2
  35. 2024Decision-Point Guided Safe Policy Improvementcited by 2
  36. 2025Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learningcited by 2
  37. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  38. 2025Guardian-regularized Safe Offline Reinforcement Learning for Smart Weaning of Mechanical Circulatory Devicescited by 2

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Policy Optimization in RL with Variance Regularizaton
  2. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  3. 2022Model-Based Offline Meta-Reinforcement Learning with Regularization
  4. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  5. 2024Policy-regularized Offline Multi-objective Reinforcement Learning
  6. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
  7. 2023Behavior Proximal Policy Optimization
  8. 2020MOPO: Model-based Offline Policy Optimization