mlsanityData to 14 Oct 2026

Bellman Diffusion Models

arXiv 2407.12163 ↗First posted 16 Jul 2024cs.LG cs.ROcited by 5

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Diffusion models have seen tremendous success as generative architectures. Recently, they have been shown to be effective at modelling policies for offline reinforcement learning and imitation learning. We explore using diffusion as a model class for the successor state measure (SSM) of a policy. We find that enforcing the Bellman flow constraints leads to a simple Bellman update on the diffusion step distribution.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 25 of its 45 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

25 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  2. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 23

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2016Generative Adversarial Imitation Learningcited by 1,597
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  10. 2019Efficient Exploration via State Marginal Matchingcited by 188
  11. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  12. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
11 more
  1. 2018Tsallis-INF: An Optimal Algorithm for Stochastic and Adversarial Banditscited by 124
  2. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  3. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  4. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  5. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  6. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 66
  7. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 45
  8. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  9. 2022Learning Infinite-Horizon Average-Reward Markov Decision Processes with Constraintscited by 29
  10. 2023Improved Best-of-Both-Worlds Guarantees for Multi-Armed Bandits: FTRL with General Regularizers and Multiple Optimal Armscited by 18
  11. 2023Best of Both Worlds Policy Optimizationcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 3.

5 papers.

  1. 2025Temporal Difference Flowscited by 24
  2. 2026Compositional Planning with Jumpy World Modelscited by 7
  3. 2024Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Spacecited by 4
  4. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  5. 2026Occupancy Modeling via Drifting: One-Step Generation Enables Direct Policy Extractioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  2. 2023Diffusion Models for Reinforcement Learning: A Survey
  3. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning
  4. 2026Path-Coupled Bellman Flows for Distributional Reinforcement Learning
  5. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization
  6. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning
  7. 2025Theoretical Barriers in Bellman-Based Reinforcement Learning
  8. 2024Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusion