mlsanityData to 7 Oct 2026

DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies

arXiv 2610.00317 ↗First posted 29 Sep 2026cs.RO cs.AI cs.CVcited by 0

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Vision-Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control. While chunk-level training is convenient across robot embodiments, it optimizes local action likelihood without explicitly accounting for long-horizon task success. Sequence-level reinforcement learning can address this limitation, but typically requires policy rollouts and closed-loop interaction, which are costly for real-robot manipulation. We introduce DriftOPD, a teacher-free, rollout-free framework for sequence-level on-policy distillation of continuous VLA action experts. We show that the sequence-level reverse Kullback-Leibler (KL) divergence decomposes into a chunk-level reverse-KL term and a future-potential term that captures the long-horizon effect of the current action. DriftOPD optimizes these two terms using a one-step drifting objective and a Q-function critic learned from offline demonstrations, respectively, enabling sequence-level optimization with only offline data and one-step action generation. Across multiple VLA architectures in simulation and real-world manipulation, DriftOPD generally outperforms existing one-step distillation baselines while achieving task success performance comparable to multi-step teacher policies. These results demonstrate that long-horizon behavior can be effectively distilled into one-step VLA action experts without online interaction or a separate teacher.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 58 of its 66 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

58 of 66 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation27 Mar
  2. 2025Reinforcing Action Policies by Prophesying25 Nov
  3. 2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Models12 Nov
  4. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators1 Oct
  5. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning11 Sep
  6. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning24 May

Other references · 52

  1. 2020Denoising Diffusion Implicit Modelscited by 9,655
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  11. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  12. 2023Consistency Modelscited by 1,778
38 more
  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  3. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  4. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  5. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  6. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  8. 2025Mean Flows for One-step Generative Modelingcited by 524
  9. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  10. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  11. 2023Improved Techniques for Training Consistency Modelscited by 430
  12. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  13. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
  14. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  15. 2022Convergence of denoising diffusion models under the manifold hypothesiscited by 237
  16. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  17. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  18. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  19. 2025Improved Mean Flows: On the Challenges of Fastforward Generative Modelscited by 102
  20. 2026Generative Modeling via Driftingcited by 98
  21. 2021Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimationcited by 95
  22. 2026ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learningcited by 94
  23. 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robotscited by 65
  24. 2024Target Score Matchingcited by 46
  25. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 35
  26. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 29
  27. 2026RLDX-1 Technical Reportcited by 25
  28. 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillationcited by 16
  29. 2024Stable-BC: Controlling Covariate Shift with Stable Behavior Cloningcited by 14
  30. 2024Score and Distribution Matching Policy: Advanced Accelerated Visuomotor Policies via Matched Distillationcited by 12
  31. 2026Mean-Flow based One-Step Vision-Language-Actioncited by 7
  32. 2026VGAS: Value-Guided Action-Chunk Selection for Few-Shot Vision-Language-Action Adaptationcited by 5
  33. 2026One-Step Flow Policy: Self-Distillation for Fast Visuomotor Policiescited by 5
  34. 2026Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inferencecited by 3
  35. 2026Mean Flow Distillation: Robust and Stable Distillation for Flow Matching Modelscited by 3
  36. 2026Drift Q-Learningcited by 2
  37. 2026Distilling Drifting Transformers with Representation Autoencoderscited by 2
  38. 2026DriftWorld: Fast World Modeling through Driftingcited by 2

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
  2. 2025Towards Deploying VLA without Fine-Tuning: Plug-and-Play Inference-Time VLA Policy Steering via Embodied Evolutionary Diffusion
  3. 2026On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning
  4. 2026Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models
  5. 2025Refined Policy Distillation: From VLA Generalists to RL Experts
  6. 2026SymVD: Symmetric Vision Language Action Distillation for Robot Manipulation
  7. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
  8. 2026Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation