mlsanityData to 7 Oct 2026

Energy-Weighted Flow Matching for Offline Reinforcement Learning

arXiv 2503.04975 ↗First posted 6 Mar 2025cs.LGcited by 70

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

This paper investigates energy guidance in generative modeling, where the target distribution is defined as q(mathbf x) propto p(mathbf x)exp(-β mathcal E(mathbf x)), with p(mathbf x) being the data distribution and mathcal E(mathcal x) as the energy function. To comply with energy guidance, existing methods often require auxiliary procedures to learn intermediate guidance during the diffusion process. To overcome this limitation, we explore energy-guided flow matching, a generalized form of the diffusion process. We introduce energy-weighted flow matching (EFM), a method that directly learns the energy-guided flow without the need for auxiliary models. Theoretical analysis shows that energy-weighted flow matching accurately captures the guided flow. Additionally, we extend this methodology to energy-weighted diffusion models and apply it to offline reinforcement learning (RL) by proposing the Q-weighted Iterative Policy Optimization (QIPO). Empirically, we demonstrate that the proposed QIPO algorithm improves performance in offline RL tasks. Notably, our algorithm is the first energy-guided diffusion model that operates independently of auxiliary models and the first exact energy-guided flow matching model in the literature.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 35 of its 38 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

35 of 38 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Models30 May
  2. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  3. 2024Policy-Guided Diffusion9 Apr
  4. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  5. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning9 Oct
  6. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  7. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  8. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  9. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 25

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  7. 2018Neural Ordinary Differential Equationscited by 3,733
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2022Video Diffusion Modelscited by 2,280
  10. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  11. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
13 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2021E(n) Equivariant Graph Neural Networkscited by 951
  3. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
  6. 2022Rectified Flow: A Marginal Preserving Approach to Optimal Transportcited by 277
  7. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  8. 2023Synthetic Experience Replaycited by 100
  9. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  10. 2024Improved off-policy training of diffusion samplerscited by 47
  11. 2024Protein Conformation Generation via Force-Guided SE(3) Diffusion Modelscited by 44
  12. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  13. 2024PILOT: Equivariant diffusion for pocket conditioned de novo ligand generation with multi-objective guidance via importance samplingcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 22; 2026: 48.

70 papers.

Sort:

By area:

  1. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  2. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  3. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
  4. 2025Flow-Based Policy for Online Reinforcement Learningcited by 33
  5. 2026Q-learning with Adjoint Matchingcited by 33
  6. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  7. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  8. 2025One-Step Flow Policy Mirror Descentcited by 18
  9. 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learningcited by 14
  10. 2026Flow Policy Gradients for Robot Controlcited by 14
  11. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 12
  12. 2025Multi-agent Coordination via Flow Matchingcited by 9
38 more
  1. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 8
  2. 2025OM2P: Offline Multi-Agent Mean-Flow Policycited by 7
  3. 2025Unleashing Flow Policies with Distributional Criticscited by 7
  4. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
  5. 2025Energy-Weighted Flow Matching: Unlocking Continuous Normalizing Flows for Efficient and Scalable Boltzmann Samplingcited by 6
  6. 2025Discrete Guidance Matching: Exact Guidance for Discrete Flow Matchingcited by 6
  7. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  8. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  9. 2026DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extractioncited by 5
  10. 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learningcited by 5
  11. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 5
  12. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 5
  13. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
  14. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 4
  15. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  16. 2025Extremum Flow Matching for Offline Goal Conditioned Reinforcement Learningcited by 3
  17. 2026Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policiescited by 3
  18. 2026Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inferencecited by 3
  19. 2026FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectificationcited by 3
  20. 2026Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learningcited by 2
  21. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 2
  22. 2026CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Makingcited by 2
  23. 2026Reversal Q-Learningcited by 2
  24. 2025Weighted Conditional Flow Matchingcited by 1
  25. 2025Offline Reinforcement Learning with Generative Trajectory Policiescited by 1
  26. 2026Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learningcited by 1
  27. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1
  28. 2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraintscited by 1
  29. 2026ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matchingcited by 1
  30. 2026Mean Flow Policy Optimizationcited by 1
  31. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 1
  32. 2026Path-Coupled Bellman Flows for Distributional Reinforcement Learningcited by 1
  33. 2026SPAR: Support-Preserving Action Rectificationcited by 1
  34. 2026Dual-Flow Reinforcement Learning with State-Aware Explorationcited by 1
  35. 2026Offline RL with Hierarchical Action Chunkingcited by 1
  36. 2025Decision Flow Policy Optimizationcited by 0
  37. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regressioncited by 0
  38. 2025Energy-Guided Generative Modeling for Low-Energy Molecular Structure Discoverycited by 0

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning
  2. 2026Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learning
  3. 2026Discrete Flow Matching for Offline-to-Online Reinforcement Learning
  4. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
  5. 2025On the Guidance of Flow Matching
  6. 2025Flow Matching Policy Gradients
  7. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning
  8. 2026Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matching