Data to 7 Oct 2026

Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling

arXiv 2209.14548 ↗First posted 29 Sep 2022cs.LGcited by 177

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

In offline reinforcement learning, weighted regression is a common method to ensure the learned policy stays close to the behavior policy and to prevent selecting out-of-sample actions. In this work, we show that due to the limited distributional expressivity of policy models, previous methods might still select unseen actions during training, which deviates from their initial motivation. To address this problem, we adopt a generative approach by decoupling the learned policy into two parts: an expressive generative behavior model and an action evaluation model. The key insight is that such decoupling avoids learning an explicitly parameterized policy model with a closed-form expression. Directly learning the behavior policy allows us to leverage existing advances in generative modeling, such as diffusion-based methods, to model diverse behaviors. As for action evaluation, we combine our method with an in-sample planning technique to further avoid selecting out-of-sample actions and increase computational efficiency. Experimental results on D4RL datasets show that our proposed method achieves competitive or superior performance compared with state-of-the-art offline RL methods, especially in complex tasks such as AntMaze. We also empirically demonstrate that our method can successfully learn from a heterogeneous dataset containing multiple distinctive but similarly successful strategies, whereas previous unimodal policies fail.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 54 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

40 of 54 references.

Closest ancestors · 1

Same direction, newest first

2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 39

2020Denoising Diffusion Probabilistic Modelscited by 20,851
2020Denoising Diffusion Implicit Modelscited by 9,655
2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
2022Generative Adversarial Networkscited by 5,603
2013Auto-Encoding Variational Bayescited by 3,980
2015Continuous control with deep reinforcement learningcited by 3,279
2016Density estimation using Real NVPcited by 2,029
2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961

27 more

2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2016Generative Adversarial Imitation Learningcited by 1,576
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2019Behavior Regularized Offline Reinforcement Learningcited by 654
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
2015MADE: Masked Autoencoder for Distribution Estimationcited by 306
2020Critic Regularized Regressioncited by 269
2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
2021Offline RL Without Off-Policy Evaluationcited by 150
2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 113
2016Model-Free Episodic Controlcited by 100
2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
2020Energy-Based Imitation Learningcited by 39
2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
2017Efficient Parallel Methods for Deep Reinforcement Learningcited by 32
2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 19; 2024: 53; 2025: 61; 2026: 43.

177 papers.

Sort:

By area:

2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
2024Diffusion Policy Policy Optimizationcited by 255
2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
2025Flow Q-Learningcited by 162
2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
2022How to Backdoor Diffusion Models?cited by 102
2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99

38 more

2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
2024Simple Hierarchical Planning with Diffusioncited by 71
2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 70
2023VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Modelscited by 68
2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 67
2023Elastic Decision Transformercited by 66
2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
2023Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimizationcited by 56
2025Horizon Reduction Makes RL Scalablecited by 55
2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 48
2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learningcited by 41
2024Potential Based Diffusion Motion Planningcited by 40
2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 38
2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
2024Towards Efficient Exact Optimization of Language Model Alignmentcited by 37
2025What Makes a Good Diffusion Planner for Decision Making?cited by 37
2023Boosting Continuous Control with Consistency Policycited by 36
2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
2025Flow-Based Policy for Online Reinforcement Learningcited by 33
2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 29
2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 29
2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 27
2024Provable Acceleration for Diffusion Models under Minimal Assumptionscited by 27
2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
2024DNAct: Diffusion Guided Multi-Task 3D Policy Learningcited by 22
2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20

50 most cited of 177 All 177 (JSON)

Similar, not cited

Close in wording, no citation either way