mlsanityData to 14 Oct 2026

Occupancy Modeling via Drifting: One-Step Generation Enables Direct Policy Extraction

arXiv 2610.19883 ↗First posted 14 Oct 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations also fits: RL with diffusion and flow models

Abstract (arXiv)

The growing interest in behavior foundation models, combined with recent advancements in generative modeling, has led to an appealing recipe for reinforcement learning from unlabeled data: pre-train a generative model to model occupancies, i.e., discounted state visitations, on reward-free trajectories, such that a policy can be easily extracted once a small reward-labeled dataset becomes available. We decompose this recipe along three axes: (i) the degree to which the multi-modal structure of occupancies must be captured, (ii) the class of generative model, and (iii) the method used for policy extraction. Along each axis, we consider existing choices, propose simpler alternatives, and validate each decision experimentally. This analysis culminates in a novel, streamlined method that leverages drifting, a modern one-step generative sampler, to model the occupancy. We show that this enables policy extraction via direct differentiation through the occupancy model, and bypasses the need for critic distillation. Overall, we observe that this nimble setup leads to improvements in performance on standard benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 61 of its 69 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

61 of 69 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learning26 Mar
  2. 2025A Unified Framework for Zero-Shot Reinforcement Learning23 Oct
  3. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning1 Oct
  4. 2025Epistemically-guided forward-backward exploration7 Jul
  5. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models15 Apr
  6. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agent29 Nov
  7. 2024Foundation Policies with Hilbert Representations23 Feb
  8. 2024A Distributional Analogue to the Successor Representation13 Feb

Other references · 53

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  9. 2023Consistency Modelscited by 1,822
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
38 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  2. 2018Exploration by Random Network Distillationcited by 993
  3. 2018DeepMind Control Suitecited by 956
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  8. 2024Genie: Generative Interactive Environmentscited by 747
  9. 2025Mean Flows for One-step Generative Modelingcited by 561
  10. 2023Learning Interactive Real-World Simulatorscited by 458
  11. 2024One Step Diffusion via Shortcut Modelscited by 391
  12. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  13. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  14. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  15. 2025Flow Q-Learningcited by 184
  16. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  17. 2021Offline RL Without Off-Policy Evaluationcited by 151
  18. 2026Generative Modeling via Driftingcited by 116
  19. 2021Learning One Representation to Optimize All Rewardscited by 115
  20. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  21. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  22. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  23. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  24. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  25. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  26. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  27. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  28. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 68
  29. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 45
  30. 2025Temporal Difference Flowscited by 24
  31. 2025Terminal Velocity Matchingcited by 20
  32. 2025Value Flowscited by 15
  33. 2022Generalised Policy Improvement with Geometric Policy Compositioncited by 13
  34. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
  35. 2026Compositional Planning with Jumpy World Modelscited by 7
  36. 2024Bellman Diffusion Modelscited by 5
  37. 2026Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Controlcited by 5
  38. 2026Drift Q-Learningcited by 4

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
  2. 2024Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective
  3. 2025Generative Models in Decision Making: A Survey
  4. 2023Score Regularized Policy Optimization through Diffusion Behavior
  5. 2025Offline Reinforcement Learning with Generative Trajectory Policies
  6. 2024Policy-Guided Diffusion
  7. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  8. 2022Is Conditional Generative Modeling all you need for Decision-Making?