mlsanityData to 14 Oct 2026

Offline Reinforcement Learning with Closed-Form Policy Improvement Operators

arXiv 2211.15956 ↗First posted 29 Nov 2022cs.LG cs.AIcited by 14

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Behavior constrained policy optimization has been demonstrated to be a successful paradigm for tackling Offline Reinforcement Learning. By exploiting historical transitions, a policy is trained to maximize a learned value function while constrained by the behavior policy to avoid a significant distributional shift. In this paper, we propose our closed-form policy improvement operators. We make a novel observation that the behavior constraint naturally motivates the use of first-order Taylor approximation, leading to a linear approximation of the policy objective. Additionally, as practical datasets are usually collected by heterogeneous policies, we model the behavior policies as a Gaussian Mixture and overcome the induced optimization difficulties by leveraging the LogSumExp's lower bound and Jensen's Inequality, giving rise to a closed-form policy improvement operator. We instantiate offline RL algorithms with our novel policy improvement operators and empirically demonstrate their effectiveness over state-of-the-art algorithms on the standard D4RL benchmark. Our code is available at https://cfpi-icml23.github.io/.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 52 of its 82 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

52 of 82 references.

Closest ancestors · 16

Same direction, newest first

  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  2. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  3. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  4. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  5. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Offline RL Without Off-Policy Evaluation16 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  10. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  11. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
4 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2021Convex optimizationcited by 2,515
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
24 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  3. 2017Deep Reinforcement Learning that Matterscited by 754
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  6. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  9. 2019Batch Policy Learning under Constraintscited by 244
  10. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
  11. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 178
  12. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  13. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  14. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  15. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  16. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  17. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  18. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  19. 2018Policy Optimization via Importance Samplingcited by 47
  20. 2016Local Maxima in the Likelihood of Gaussian Mixture Models: Structural Results and Algorithmic Consequencescited by 34
  21. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 32
  22. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18
  23. 2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
  24. 2020Taylor Expansion Policy Optimizationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 2; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 87
  2. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedbackcited by 25
  3. 2024Transcendence: Generative Models Can Outperform The Experts That Train Themcited by 20
  4. 2023Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuningcited by 15
  5. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignmentcited by 9
  6. 2024Value Improved Actor Critic Algorithmscited by 7
  7. 2024Offline Reinforcement Learning with Behavioral Supervisor Tuningcited by 3
  8. 2024NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulationcited by 2
  9. 2025Policy-Based Trajectory Clustering in Offline Reinforcement Learningcited by 2
  10. 2023Mildly Constrained Evaluation Policy for Offline Reinforcement Learningcited by 1
  11. 2025Behavior Preference Regression for Offline Reinforcement Learningcited by 1
  12. 2026Less is More: Clustered Cross-Covariance Control for Offline RLcited by 1
2 more
  1. 2026Non-Parametric Rehearsal Learning via Conditional Mean Embeddingscited by 0
  2. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  2. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  3. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
  4. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  5. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation
  6. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPs
  7. 2020MOReL : Model-Based Offline Reinforcement Learning
  8. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning