# Dynamic Neural Koopman Distillation for Real-Time Robot Control Using Diffusion Models

arXiv: https://arxiv.org/abs/2605.24924 · HTML: https://mlsanity.com/paper/2605.24924/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 32 of its 35 references that are in the map.
- First posted on arXiv 24 May 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 32)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Improved Techniques for Training Consistency Models](https://mlsanity.com/paper/2310.14189/) (2023)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation](https://mlsanity.com/paper/2405.07503/) (2024)
- [One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation](https://mlsanity.com/paper/2410.21257/) (2024)
- [DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasets](https://mlsanity.com/paper/2404.19264/) (2024)
- [Koopman Operator Dynamical Models: Learning, Analysis and Control](https://mlsanity.com/paper/2102.02522/) (2021)
- [CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Making](https://mlsanity.com/paper/2406.09509/) (2024)
- [Model-Based Diffusion for Trajectory Optimization](https://mlsanity.com/paper/2407.01573/) (2024)
- [Koopman Kernel Regression](https://mlsanity.com/paper/2305.16215/) (2023)
- [Diffusion Models for Intelligent Transportation Systems: A Survey](https://mlsanity.com/paper/2409.15816/) (2024)
- [Koopman Operators in Robot Learning](https://mlsanity.com/paper/2408.04200/) (2024)
- [DARE: Diffusion Policy for Autonomous Robot Exploration](https://mlsanity.com/paper/2410.16687/) (2024)
- [SICNav-Diffusion: Safe and Interactive Crowd Navigation with Diffusion Trajectory Predictions](https://mlsanity.com/paper/2503.08858/) (2025)
- [An overview of Koopman-based control: From error bounds to closed-loop guarantees](https://mlsanity.com/paper/2509.02839/) (2025)
- [One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling](https://mlsanity.com/paper/2505.13358/) (2025)
- [Toward Near-Globally Optimal Nonlinear Model Predictive Control via Diffusion Models](https://mlsanity.com/paper/2412.08278/) (2024)
- [CoPlanner: An Interactive Motion Planner with Contingency-Aware Diffusion for Autonomous Driving](https://mlsanity.com/paper/2509.17080/) (2025)
- [Imitation Learning with Limited Actions via Diffusion Planners and Deep Koopman Controllers](https://mlsanity.com/paper/2410.07584/) (2024)

## Abstract (arXiv)

Diffusion models excel at generating diverse, multimodal trajectories for robotic control, yet their iterative denoising process introduces latency that can constrain real-time closed-loop execution. To address this problem, we propose a Dynamic Neural Koopman (DNK) distillation framework, which distills multistep diffusion inference into a single forward pass conditioned on observations and sampled noise. Specifically, we introduce a Factorized Dynamic Koopman (FDK) layer that models the denoising process through a latent linear transition. The noisy trajectory input is lifted into a latent space, where the FDK layer parameterizes a factorized linear operator with condition-dependent eigenvalues that adapt the noisy-to-denoised transition to the current context. We evaluate DNK across robot-control benchmarks that span locomotion, state-based, and image-based manipulation, comparing against accelerated generative-policy baselines. The results demonstrate that DNK achieves competitive or improved closed-loop performance relative to accelerated generative-policy baselines, while substantially reducing inference latency relative to the strongest high-performing one-step baselines. On long-horizon manipulation tasks, DNK achieves competitive performance with substantially fewer inference parameters than the one-step policy baselines. Hardware experiments on a physical Kinova manipulator further demonstrate millisecond-scale inference and reduced task completion time. A project page is available at https://fdkoopman.github.io/.