mlsanityData to 14 Oct 2026

Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching

arXiv 2306.14079 ↗First posted 24 Jun 2023cs.LG cs.AI cs.ROcited by 17

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Gradient-based methods enable efficient search capabilities in high dimensions. However, in order to apply them effectively in offline optimization paradigms such as offline Reinforcement Learning (RL) or Imitation Learning (IL), we require a more careful consideration of how uncertainty estimation interplays with first-order methods that attempt to minimize them. We study smoothed distance to data as an uncertainty metric, and claim that it has two beneficial properties: (i) it allows gradient-based methods that attempt to minimize uncertainty to drive iterates to data as smoothing is annealed, and (ii) it facilitates analysis of model bias with Lipschitz constants. As distance to data can be expensive to compute online, we consider settings where we need amortize this computation. Instead of learning the distance however, we propose to learn its gradients directly as an oracle for first-order optimizers. We show these gradients can be efficiently learned with score-matching techniques by leveraging the equivalence between distance to data and data likelihood. Using this insight, we propose Score-Guided Planning (SGP), a planning algorithm for offline RL that utilizes score-matching to enable first-order planning in high-dimensional problems, where zeroth-order methods were unable to scale, and ensembles were unable to overcome local minima. Website: https://sites.google.com/view/score-guided-planning/home

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 80 references that are in the map. First posted on arXiv 24 Jun 2023.

Builds on

58 of 80 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Generative Adversarial Networkscited by 5,627
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2013Auto-Encoding Variational Bayescited by 4,053
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  10. 2015Variational Inference with Normalizing Flowscited by 2,170
  11. 2016Generative Adversarial Imitation Learningcited by 1,597
  12. 2023Mastering Diverse Domains through World Modelscited by 1,253
38 more
  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  3. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  4. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  10. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  11. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  12. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  13. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  14. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  15. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 220
  16. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  17. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  18. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
  19. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  20. 2020Model-Based Offline Planningcited by 129
  21. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
  22. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 97
  23. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  24. 2021Conservative Objective Models for Effective Offline Model-Based Optimizationcited by 88
  25. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  26. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 70
  27. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  28. 2022Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Modelscited by 49
  29. 2019Calibrated Model-Based Deep Reinforcement Learningcited by 31
  30. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 28
  31. 2022Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Controlcited by 28
  32. 2020The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulationcited by 25
  33. 2020Planning with Learned Dynamics: Probabilistic Guarantees on Safety and Reachability via Lipschitz Constantscited by 25
  34. 2021Bundled Gradients through Contact via Randomized Smoothingcited by 24
  35. 2022Safe Output Feedback Motion Planning from Images via Learned Perception Modules and Contraction Theorycited by 19
  36. 2021Model Error Propagation via Learned Contraction Metrics for Safe Feedback Motion Planning of Unknown Systemscited by 13
  37. 2022Statistical Safety and Robustness Guarantees for Feedback Motion Planning of Unknown Underactuated Stochastic Systemscited by 11
  38. 2023Synthesizing Stable Reduced-Order Visuomotor Policies for Nonlinear Systems via Sums-of-Squares Optimizationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 7; 2026: 3.

17 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 184
  2. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  3. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
  4. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  5. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  6. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 28
  7. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  8. 2023Efficient Planning with Latent Diffusioncited by 12
  9. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 9
  10. 2025Unleashing Flow Policies with Distributional Criticscited by 8
  11. 2026Safety Beyond the Training Data: Robust Out-of-Distribution MPC via Conformalized System Level Synthesiscited by 7
  12. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
5 more
  1. 2025Accelerating Residual Reinforcement Learning with Uncertainty Estimationcited by 6
  2. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency modelcited by 4
  3. 2026Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPCcited by 4
  4. 2025COLSON: Controllable Learning-Based Social Navigation via Diffusion-Based Reinforcement Learningcited by 3
  5. 2026Denoising Particle Filters: Learning State Estimation with Single-Step Objectivescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  2. 2023Selective Uncertainty Propagation in Offline RL
  3. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning
  4. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach
  5. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  6. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  7. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  8. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning