mlsanityData to 7 Oct 2026

Offline Reinforcement Learning with Discrete Diffusion Skills

arXiv 2503.20176 ↗First posted 26 Mar 2025cs.LG cs.AI cs.ROcited by 1

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Skills have been introduced to offline reinforcement learning (RL) as temporal abstractions to tackle complex, long-horizon tasks, promoting consistent behavior and enabling meaningful exploration. While skills in offline RL are predominantly modeled within a continuous latent space, the potential of discrete skill spaces remains largely underexplored. In this paper, we propose a compact discrete skill space for offline RL tasks supported by state-of-the-art transformer-based encoder and diffusion-based decoder. Coupled with a high-level policy trained via offline RL techniques, our method establishes a hierarchical RL framework where the trained diffusion decoder plays a pivotal role. Empirical evaluations show that the proposed algorithm, Discrete Diffusion Skill (DDS), is a powerful offline RL method. DDS performs competitively on Locomotion and Kitchen tasks and excels on long-horizon tasks, achieving at least a 12 percent improvement on AntMaze-v2 benchmarks compared to existing offline RL approaches. Furthermore, DDS offers improved interpretability, training stability, and online exploration compared to previous skill-based methods.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 37 of its 40 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

37 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Robust Policy Learning via Offline Skill Diffusion1 Mar
  2. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution18 Dec
  3. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning12 Sep
  4. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  5. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
19 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  8. 2016The Option-Critic Architecturecited by 433
  9. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  10. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  11. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  12. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  13. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  14. 2024QueST: Self-Supervised Skill Abstractions for Learning Continuous Controlcited by 51
  15. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 34
  16. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9
  17. 2022Skill Machines: Temporal Logic Skill Composition in Reinforcement Learningcited by 7
  18. 2023Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learningcited by 6
  19. 2024When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractionscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026From Noise to Control: Parameterized Diffusion Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient Diffusion Policies for Offline Reinforcement Learning
  2. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  3. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  4. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  6. 2025DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
  7. 2025Modular Diffusion Policy Training: Decoupling and Recombining Guidance and Diffusion for Offline RL
  8. 2026Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets