mlsanityData to 14 Oct 2026

Score Regularized Policy Optimization through Diffusion Behavior

arXiv 2310.07297 ↗First posted 11 Oct 2023cs.LGcited by 71

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Recent developments in offline reinforcement learning have uncovered the immense potential of diffusion modeling, which excels at representing heterogeneous behavior policies. However, sampling from diffusion policies is considerably slow because it necessitates tens to hundreds of iterative inference steps for one action. To address this issue, we propose to extract an efficient deterministic inference policy from critic models and pretrained diffusion behavior models, leveraging the latter to directly regularize the policy gradient with the behavior distribution's score function during optimization. Our method enjoys powerful generative capabilities of diffusion modeling while completely circumventing the computationally intensive and time-consuming diffusion sampling scheme, both during training and evaluation. Extensive results on D4RL tasks show that our method boosts action sampling speed by more than 25 times compared with various leading diffusion-based methods in locomotion tasks, while still maintaining state-of-the-art performance.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 30 of its 38 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

30 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  2. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  3. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  4. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  5. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 24

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  11. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
12 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  4. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  5. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 516
  6. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  7. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  8. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 98
  9. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
  10. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  11. 2022Boosting Offline Reinforcement Learning via Data Rebalancingcited by 22
  12. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 28; 2026: 33.

71 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 184
  2. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  3. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 89
  4. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 73
  5. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 59
  6. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 58
  7. 2024DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasetscited by 49
  8. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learningcited by 43
  9. 2026Q-learning with Adjoint Matchingcited by 37
  10. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 34
  11. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 30
  12. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
38 more
  1. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 20
  2. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
  3. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlowcited by 18
  4. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
  5. 2024Variational Distillation of Diffusion Policies into Mixture of Expertscited by 15
  6. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
  7. 2025Falcon: Fast Visuomotor Policies via Partial Denoisingcited by 15
  8. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learningcited by 14
  9. 2025ReAlign: Text-to-Motion Generation via Step-Aware Reward-Guided Alignmentcited by 13
  10. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
  11. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learningcited by 11
  12. 2025Multi-agent Coordination via Flow Matchingcited by 10
  13. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 10
  14. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Datacited by 9
  15. 2026Motus2: A Self-Evolving General World Model for Dexterous Manipulationcited by 9
  16. 2025Unleashing Flow Policies with Distributional Criticscited by 8
  17. 2025Habitizing Diffusion Planning for Efficient and Effective Decision Makingcited by 7
  18. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  19. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  20. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  21. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 6
  22. 2026DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extractioncited by 5
  23. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  24. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  25. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
  26. 2026Drift Q-Learningcited by 4
  27. 2025Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learningcited by 3
  28. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  29. 2026GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policiescited by 3
  30. 2026Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flowcited by 3
  31. 2026Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inferencecited by 3
  32. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 3
  33. 2025Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learningcited by 2
  34. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 2
  35. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learningcited by 2
  36. 2026Entropy-Regularized Adjoint Matching for Offline Reinforcement Learningcited by 2
  37. 2026Path-Coupled Bellman Flows for Distributional Reinforcement Learningcited by 2
  38. 2026Score-Based One-step MeanFlow Policy Optimizationcited by 2

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Diffusion Policy through Conditional Proximal Policy Optimization
  2. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
  3. 2026Guided Discovery of New Behaviors using Diffusion Policies
  4. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  5. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RL
  6. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies
  7. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning
  8. 2026Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies