mlsanityData to 14 Oct 2026

Generative Actor Critic

arXiv 2512.21527 ↗First posted 25 Dec 2025cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

Conventional Reinforcement Learning (RL) algorithms, typically focused on estimating or maximizing expected returns, face challenges when refining offline pretrained models with online experiences. This paper introduces Generative Actor Critic (GAC), a novel framework that decouples sequential decision-making by reframing textitpolicy evaluation as learning a generative model of the joint distribution over trajectories and returns, p(τ, y), and textitpolicy improvement as performing versatile inference on this learned model. To operationalize GAC, we introduce a specific instantiation based on a latent variable model that features continuous latent plan vectors. We develop novel inference strategies for both textitexploitation, by optimizing latent plans to maximize expected returns, and textitexploration, by sampling latent plans conditioned on dynamically adjusted target returns. Experiments on Gym-MuJoCo and Maze2D benchmarks demonstrate GAC's strong offline performance and significantly enhanced offline-to-online improvement compared to state-of-the-art methods, even in absence of step-wise rewards.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 68 of its 87 references that are in the map. First posted on arXiv 25 Dec 2025.

Builds on

68 of 87 references.

References · 68

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  6. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  10. 2016Variational Inference: A Review for Statisticianscited by 1,472
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
38 more
  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2015Prioritized Experience Replaycited by 1,099
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  4. 2018Exploration by Random Network Distillationcited by 993
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  13. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  14. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  15. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  16. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  17. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  18. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  19. 2018Maximum a Posteriori Policy Optimisationcited by 328
  20. 2023Language Models Represent Space and Timecited by 316
  21. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  22. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  23. 2017Noisy Networks for Explorationcited by 266
  24. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  25. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  26. 2022Multi-Game Decision Transformerscited by 214
  27. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  28. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  29. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  30. 2022Online Decision Transformercited by 196
  31. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  32. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  33. 2024Evaluating the World Model Implicit in a Generative Modelcited by 113
  34. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  35. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  36. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  37. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  38. 2023Guided Flows for Generative Modeling and Decision Makingcited by 101

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Inference-Time Rethinking with Latent Thought Vectors for Math Reasoningcited by 2
  2. 2026Learning to Plan from Random Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  2. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
  3. 2026ReBRAC-v2: The Return of the King
  4. 2025Generative Models in Decision Making: A Survey
  5. 2025Offline Reinforcement Learning with Generative Trajectory Policies
  6. 2026Generative Actor-Critic with Soft Bridge Policies
  7. 2026Model-Based Online Decision Making via Generative Trajectory Planning
  8. 2024Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective