# Learning When to Stop: Prefix-Optimal Dynamic Diffusion Policies for Continuous Control

arXiv: https://arxiv.org/abs/2608.05084 · HTML: https://mlsanity.com/paper/2608.05084/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 24 of 24)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies](https://mlsanity.com/paper/2304.10573/) (2023)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [Efficient Diffusion Policies for Offline Reinforcement Learning](https://mlsanity.com/paper/2305.20081/) (2023)
- [Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization](https://mlsanity.com/paper/2405.16173/) (2024)
- [Flow Matching for Scalable Simulation-Based Inference](https://mlsanity.com/paper/2305.17161/) (2023)
- [Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning](https://mlsanity.com/paper/2309.16984/) (2023)
- [Diffusion Actor-Critic with Entropy Regulator](https://mlsanity.com/paper/2405.15177/) (2024)
- [Efficient Online Reinforcement Learning for Diffusion Policy](https://mlsanity.com/paper/2502.00361/) (2025)
- [Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models](https://mlsanity.com/paper/2111.13129/) (2021)
- [D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning](https://mlsanity.com/paper/2508.06804/) (2025)
- [Distributional Soft Actor-Critic with Diffusion Policy](https://mlsanity.com/paper/2507.01381/) (2025)

## Built on by (most cited first, 2 of 2)

- [Think Fast, Plan Selectively: Adaptive Deliberation for Efficient Data-Driven MPC](https://mlsanity.com/paper/2609.32591/) (2026)
- [Urgent Actions Go First: Urgency-Aware Denoising for Real-Time VLA Control](https://mlsanity.com/paper/2609.37772/) (2026)

## Abstract (arXiv)

Diffusion policies are a powerful policy class for continuous control, but their iterative denoising process creates a substantial computational bottleneck. Reducing this cost requires adapting the number of denoising steps to the difficulty of each action while preserving task performance. We introduce Prefix-Optimal Generative Policies (POGP), a framework that learns a prefix value function at every intermediate denoising step through a Bellman-style recursion over the denoising chain. The prefix value function serves two purposes: it provides an auxiliary training objective that encourages intermediate outputs to become high-quality actions, and it enables a test-time stopping rule that terminates denoising when additional steps are unlikely to produce meaningful improvement. Across four MuJoCo environments and comparisons with 12 baselines, POGP reduces the required number of denoising iterations by approximately 2.7-fold while retaining near-full task performance. Compared with state-of-the-art dynamic diffusion baselines, prefix training also improves final task performance by approximately 3.5%. These results indicate that supervising intermediate denoising steps is useful not only for adaptive early stopping, but also as an auxiliary objective that improves the learned policy.