StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks
On the map: Vision-language-action models → Reinforcement learning and post-training
Abstract (arXiv)
Vision-language-action (VLA) models perform well on shorter-horizon manipulation tasks but still struggle with long-horizon tasks that require multiple dependent manipulations from a single command. Online reinforcement learning (RL) can improve these policies through environment interaction, yet many existing methods provide reward only after the complete task succeeds. However, such terminal supervision is sparse and does not distinguish early failures from rollouts that make substantial partial progress. We propose StructRL, an online RL framework that constructs structured intermediate supervision from verifiable subtask completions. StructRL decomposes each task into verifiable subtasks, grants intermediate rewards only after the prerequisite subtasks have been completed, and scales each reward according to completion pace. Across RoboCasa365 and LIBERO-Long with GR00T-N1.5 and pi 0.5, StructRL consistently outperforms evaluated online RL baselines. These results show that verifiable, structured intermediate rewards improve long-horizon VLA post-training. Code is available at https://github.com/amazon-science/StructRL.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 20 of its 31 references that are in the map. First posted on arXiv 28 Sep 2026.
Builds on
20 of 31 references.
Closest ancestors · 9
Same direction, newest first
- 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models8 Oct
- 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning11 Sep
- 2025CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning4 Aug
- 2025TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization10 Jun
- 2025What Can RL Bring to VLA Generalization? An Empirical Study26 May
- 2025Interactive Post-Training for Vision-Language-Action Models22 May
- 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning12 May
- 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy8 Feb
- 2024GRAPE: Generalizing Robot Policy via Preference Alignment28 Nov
Other references · 11
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
- 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
- 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
- 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
- 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robotscited by 65
- 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
- 2025RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluationcited by 31
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
- 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Models
- 2026StructRL: Structured Action-Space Exploration for Flow-Based VLAs
- 2025VLA-R1: Enhancing Reasoning in Vision-Language-Action Models
- 2026TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models
- 2025Improving Vision-Language-Action Model with Online Reinforcement Learning
- 2026Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models
- 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models