mlsanityData to 14 Oct 2026

Multi-step Proximal Policy Improvement in Offline Reinforcement Learning

arXiv 2609.03842 ↗First posted 3 Sep 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

Behavior regularization in offline reinforcement learning limits the exploitation of critic errors, but strong anchoring can also restrict policy improvement. We study how policy improvement is composed through multi-step proximal policy improvement (MPI), which re-centers each proximal objective on the preceding policy. We parameterize the procedure by a nominal total horizon T and K stages with local horizon T/K, distinguishing subdivision at a fixed total horizon from additional refinement at a common local horizon. Our analysis shows that sequential re-centering can reach endpoints unavailable to any single proximal step and characterizes how subdivision reduces the leading local discretization error of ideal updates under a fixed critic. We consider TD3+BC and IQL-based policy extraction to examine how improvement composition interacts with actor objectives and policy geometry. TD3+BC experiments on D4RL locomotion suggest that subdivision can broaden the range of useful total horizons, while adding refinement stages at a fixed small local horizon can improve return. The results identify improvement composition as a design choice alongside regularization strength, with distinct effects from horizon subdivision and additional policy extraction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 30 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

22 of 30 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Is Value Learning Really the Main Bottleneck in Offline RL?13 Jun
  2. 2023Supported Trust Region Optimization for Offline Reinforcement Learning15 Nov
  3. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning9 Jun
  4. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators29 Nov
  5. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Offline RL Without Off-Policy Evaluation16 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 10

  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  6. 2025Flow Q-Learningcited by 184
  7. 2018Policy Optimization as Wasserstein Gradient Flowscited by 34
  8. 2020Efficient Wasserstein Natural Gradients for Reinforcement Learningcited by 23
  9. 2025Wasserstein Policy Optimizationcited by 13
  10. 2022Trust Region Policy Optimization with Optimal Transport Discrepancies: Duality and Algorithm for Continuous Actionscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification
  2. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  3. 2025Deep SPI: Safe Policy Improvement via World Models
  4. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  5. 2026Flow Actor-Critic for Offline Reinforcement Learning
  6. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  7. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  8. 2025A Clean Slate for Offline Reinforcement Learning