mlsanityData to 7 Oct 2026

ISEP: Implicit Support Expansion for Offline Reinforcement Learning via Stochastic Policy Optimization

arXiv 2605.18320 ↗First posted 18 May 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning methods typically enforce strict constraints to ensure safety; yet this rigidity often prevents the discovery of optimal behaviors outside the immediate support of the behavior policy. To address this, we propose Implicit Support Expansion via stochastic Policy optimization (ISEP), which leverages a value function interpolated between in-distribution data and policy samples to implicitly expand the feasible action support. This mechanism "densifies" high-reward regions, creating a navigable path for policy improvement while theoretically guaranteeing bounded value error. However, optimizing against this expanded support creates a multimodal landscape where standard deterministic averaging leads to mode collapse and invalid actions. ISEP mitigates this via a stochastic action selection strategy, optimizing the policy by stochastically alternating between conservative cloning and optimistic expansion signals. We instantiate this framework as ISEP-FM using Conditional Flow Matching utilizing classifier-free guidance to effectively capture the interpolated value signal.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 38 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

28 of 38 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Supported Trust Region Optimization for Offline Reinforcement Learning15 Nov
  2. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning22 Sep
  3. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  5. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  8. 2021Offline RL Without Off-Policy Evaluation16 Jun
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
1 more
  1. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2016An overview of gradient descent optimization algorithmscited by 1,021
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  8. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  9. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  10. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  11. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  12. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 70
3 more
  1. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  2. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 46
  3. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learningcited by 40

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  2. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation
  3. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective
  4. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  5. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  6. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning
  7. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  8. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learning