Preserve Support, Not Correspondence: Dynamic Routing for Offline Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
One-step offline RL actors are attractive because they avoid backpropagating through long iterative samplers and keep inference cheap, but they still have to improve under a critic without drifting away from actions that the dataset can support. In recent one-step extraction pipelines, a strong iterative teacher provides one target action for each latent draw, and the same student output is asked to do both jobs: move toward higher Q and stay near that paired endpoint. If those two directions disagree, the loss resolves them as a compromise on that same sample, even when a nearby better action remains locally supported by the data. We propose DROL, a latent-conditioned one-step actor trained with top-1 dynamic routing. For each state, the actor samples K candidate actions from a bounded latent prior, assigns each dataset action to its nearest candidate, and updates only that winner with Behavior Cloning and critic guidance. Because the routing is recomputed from the current candidate geometry, ownership of a supported region can shift across candidates over the course of learning. This gives a one-step actor room to make local improvements that pointwise extraction struggles to capture, while retaining single-pass inference at test time. On OGBench and D4RL, DROL is competitive with the one-step FQL baseline, improving many OGBench task groups while remaining strong on both AntMaze and Adroit. Project page: https://muzhancun.github.io/preprints/DROL.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 21 references that are in the map. First posted on arXiv 24 Apr 2026.
Builds on
17 of 21 references.
Closest ancestors · 7
Same direction, newest first
- 2026Flow Actor-Critic for Offline Reinforcement Learning20 Feb
- 2025Scaling Offline RL via Efficient and Expressive Shortcut Models28 May
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
Other references · 10
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
- 2021Implicit Behavioral Cloningcited by 442
- 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
- 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
- 2018Implicit Maximum Likelihood Estimationcited by 52
- 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlowcited by 16
- 2025IMLE Policy: Fast and Sample Efficient Visuomotor Policy Learning via Implicit Maximum Likelihood Estimationcited by 13
- 2026DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extractioncited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
- 2026Latent Policy Steering through One-Step Flow Policies
- 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models
- 2026Decoupling Policy Extraction for Offline Reinforcement Learning
- 2022Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flows
- 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning
- 2024Augmenting Offline RL with Unlabeled Data
- 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning