# Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learning

arXiv: https://arxiv.org/abs/2605.02777 · HTML: https://mlsanity.com/paper/2605.02777/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 35 of its 39 references that are in the map.
- First posted on arXiv 4 May 2026.

## Placed in

- Offline RL and imitation learning from data / [Safe offline RL](https://mlsanity.com/offlinerl/safe-offline-rl/)

## Builds on (most cited first, 30 of 35)

- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2310.12567/) (2023)
- [Diffusion-Based Planning for Autonomous Driving with Flexible Guidance](https://mlsanity.com/paper/2501.15564/) (2025)
- [OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2106.10783/) (2021)
- [Constraints Penalized Q-learning for Safe Offline Reinforcement Learning](https://mlsanity.com/paper/2107.09003/) (2021)
- [SafeDiffuser: Safe Planning with Diffusion Probabilistic Models](https://mlsanity.com/paper/2306.00148/) (2023)
- [COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2204.08957/) (2022)
- [Constrained Decision Transformer for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2302.07351/) (2023)
- [Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model](https://mlsanity.com/paper/2401.10700/) (2024)
- [Datasets and Benchmarks for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2306.09303/) (2023)
- [On the role of planning in model-based deep reinforcement learning](https://mlsanity.com/paper/2011.04021/) (2020)
- [Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models](https://mlsanity.com/paper/2405.17374/) (2024)
- [CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics](https://mlsanity.com/paper/2406.14558/) (2024)
- [Reasoning with Latent Diffusion in Offline Reinforcement Learning](https://mlsanity.com/paper/2309.06599/) (2023)
- [Safe Offline Reinforcement Learning with Real-Time Budget Constraints](https://mlsanity.com/paper/2306.00603/) (2023)
- [M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes](https://mlsanity.com/paper/2410.11402/) (2024)
- [In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought](https://mlsanity.com/paper/2405.20692/) (2024)
- [Efficient Robotic Policy Learning via Latent Space Backward Planning](https://mlsanity.com/paper/2505.06861/) (2025)
- [Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2412.18946/) (2024)
- [Scalable Offline Model-Based RL with Action Chunks](https://mlsanity.com/paper/2512.08108/) (2025)
- [Efficient Planning with Latent Diffusion](https://mlsanity.com/paper/2310.00311/) (2023)
- [VICtoR: Learning Hierarchical Vision-Instruction Correlation Rewards for Long-horizon Manipulation](https://mlsanity.com/paper/2405.16545/) (2024)
- [Constraint-Aware Diffusion Guidance for Robotics: Real-Time Obstacle Avoidance for Autonomous Racing](https://mlsanity.com/paper/2505.13131/) (2025)

## Abstract (arXiv)

Offline safe reinforcement learning often requires policies to adapt at deployment time to safety budgets that vary across episodes or change within a single episode. While diffusion-based planners enable flexible trajectory generation, existing guidance schemes often treat reward improvement and constraint satisfaction as competing gradient objectives, which can lead to unreliable safety compliance under cost limits. We reinterpret adaptive safe trajectory generation as sampling from a constrained trajectory distribution, where the budget restricts the trajectory region, and reward shapes preferences within that region. This perspective motivates Safe Decoupled Guidance Diffusion (SDGD), which conditions classifier-free guidance on the cost limit to bias sampling toward trajectories satisfying the specified limit, while using reward-gradient guidance to refine trajectories for higher return. Because direct reward guidance can increase return while also steering samples toward trajectories with higher cumulative cost, we introduce Feasible Trajectory Relabeling (FTR) to reshape reward targets and discourage such directions. We further provide a first-order sampling-time analysis showing that FTR suppresses reward-induced cost drift under a prefix-restorative alignment condition. Extensive evaluations on the DSRL benchmark show that SDGD achieves the strongest safety compliance among baselines, satisfying the constraint on 94.7% of tasks (36/38), while obtaining the highest reward among safe methods on 21 tasks.