mlsanityData to 14 Oct 2026

StructRL: Structured Action-Space Exploration for Flow-Based VLAs

arXiv 2608.15139 ↗First posted 15 Aug 2026cs.ROcited by 1

On the map: Vision-language-action models → Efficient VLA inference and adaptation

Abstract (arXiv)

Flow-based Vision-Language-Action (VLA) models are now widely used for continuous robotic manipulation, and online reinforcement learning (RL) is emerging as a key technique for adapting them to new tasks. Existing RL methods typically inject stochasticity inside the denoising chain, often through isotropic or temporally independent noise. However, effective robot exploration calls for structured noise: temporally smooth and scaled differently across action groups. We show that simply switching the in-chain noise to a structured form does not suffice: noise added at an intermediate flow time can be weakened by the remaining denoising steps before execution, a phenomenon we call Structured Noise Dilution. We propose StructRL, which avoids dilution by relocating policy stochasticity to the action space via three coupled choices: (i) a deterministic ODE decoder, (ii) structured noise injected directly in the action space, and (iii) last-step replay, where policy-gradient updates avoid assigning likelihoods to intermediate denoising states. This keeps structured exploration tied to the executed action while providing a tractable training signal for the flow decoder. Across three flow-based VLA models on multiple simulated manipulation benchmarks and two real-world tasks, StructRL improves exploration efficiency and OOD performance over prior in-chain baselines, demonstrating the effectiveness of structured action-space exploration for adapting flow-based VLA with RL. Project page: https://flyfaerss.github.io/structrl/

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Efficient VLA inference and adaptation, one of 13 directions in Vision-language-action models. Builds on 32 of its 36 references that are in the map. First posted on arXiv 15 Aug 2026.

Builds on

32 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs2 Mar
  2. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Models16 Jan

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  11. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  12. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
18 more
  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  4. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  5. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  8. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  9. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  10. 2017Parameter Space Noise for Explorationcited by 161
  11. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matchingcited by 56
  12. 2020Smooth Exploration for Robotic Reinforcement Learningcited by 54
  13. 2023Boosting Continuous Control with Consistency Policycited by 38
  14. 2026Q-learning with Adjoint Matchingcited by 37
  15. 2023Latent Exploration for Reinforcement Learningcited by 22
  16. 2025Actor-Critic for Continuous Action Chunks: A Reinforcement Learning Framework for Long-Horizon Robotic Manipulation with Sparse Rewardcited by 13
  17. 2023Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Samplingcited by 8
  18. 2025SOE: Sample-Efficient Robot Policy Self-Improvement via On-Manifold Explorationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks
  2. 2026dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models
  3. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
  4. 2026Self-Improving VLA Policies: Selected Diffusion Noise for Spurious-Robust Action Smoothing
  5. 2026UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation
  6. 2026REACT: Rolling Denoising and Dual Decoupling for Reactive Robot Control with VLA Models
  7. 2026Catch Me If You Can: Real-Time Feedback Denoising for Responsive VLAs
  8. 2026StairVLA: Stage-Aware Hierarchical Action Generation for Vision-Language-Action Models