mlsanityData to 7 Oct 2026

Principal Steering Subspaces for Online Adaptation of Frozen Generative Robot Policies

arXiv 2609.33765 ↗First posted 27 Sep 2026cs.ROcited by 0

On the map: Vision-language-action models → Reinforcement learning and post-training

also in 1 more
Abstract (arXiv)

Generative robot policies provide expressive behavior priors, but updating a large diffusion or flow-matching model through online interaction is costly. Latent-space reinforcement learning avoids updating the pretrained generator by controlling its initial sampling noise, yet high-dimensional noise can have strongly anisotropic effects on decoded actions. We introduce Principal Steering Subspaces (PSS), a forward-query interface that constructs a fixed low-dimensional control basis from finite-difference decoder responses. Soft Actor-Critic controls the leading response directions, while the orthogonal complement is independently resampled from the Gaussian prior at each query. On three RoboMimic tasks with diffusion and flow-matching policies, response spectra reveal substantial concentration. Across five matched task-generator pairs, the training curves indicate that PSS generally converges faster and exhibits more stable late-training behavior than full-latent control, while achieving stronger final performance overall. Controlled Diffusion-Square ablations further show that leading-response directions outperform random and least-responsive subspaces of equal dimension. We further integrate PSS with a frozen, closed-source 3B-parameter vision-language-action (VLA) policy in a humanoid learning system with synchronous transition collection, reset-time optimization, and latency-aware asynchronous deployment. In an exploratory screwdriver-placement evaluation, success is observed in 2/10 trials for the frozen VLA policy and 6/10 after SAC+PSS adaptation. These results support decoder-response geometry as a practical basis for online adaptation of frozen generative robot policies.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 25 of its 26 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

25 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Improving Robotic Generalist Policies via Flow Reversal Steering11 Jun
  2. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Models24 Apr

Other references · 23

  1. 2020Denoising Diffusion Implicit Modelscited by 9,655
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  10. 2024Diffusion Policy Policy Optimizationcited by 255
  11. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  12. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
11 more
  1. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  2. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  3. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  4. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  5. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  6. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  7. 2021LASER: Learning a Latent Action Space for Efficient Reinforcement Learningcited by 32
  8. 2026You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vectorcited by 5
  9. 2026UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptationcited by 5
  10. 2026Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learningcited by 2
  11. 2026Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policiescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Steering Generative Reinforcement Learning into Stable Robotic Controller ↗
  2. 2026VLS: Steering Pretrained Robot Policies via Vision-Language Models
  3. 2026FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space
  4. 2026Latent Policy Steering through One-Step Flow Policies
  5. 2026Safe Real-Time Policy Steering via Noise-Space Trajectory Optimization for One-Step Generative Policies ↗
  6. 2025Exploration and Adaptation in Non-Stationary Tasks with Diffusion Policies
  7. 2025Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance
  8. 2026Generative Control as Optimization: Time Unconditional Flow Matching for Adaptive and Robust Robotic Control