# VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2506.17561 · HTML: https://mlsanity.com/paper/2506.17561/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models.
- Builds on 92 of its 103 references that are in the map.
- First posted on arXiv 21 Jun 2025.

## Placed in

- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)

## Builds on (most cited first, 30 of 92)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)

## Built on by (most cited first, 30 of 36)

- [MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2508.19236/) (2025)
- [VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model](https://mlsanity.com/paper/2509.09372/) (2025)
- [Galaxea Open-World Dataset and G0 Dual-System VLA Model](https://mlsanity.com/paper/2509.00576/) (2025)
- [SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overhead](https://mlsanity.com/paper/2512.00903/) (2025)
- [FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenization](https://mlsanity.com/paper/2512.04952/) (2025)
- [Mixture of Horizons in Action Chunking](https://mlsanity.com/paper/2511.19433/) (2025)
- [Continually Evolving Skill Knowledge in Vision Language Action Model](https://mlsanity.com/paper/2511.18085/) (2025)
- [RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation](https://mlsanity.com/paper/2602.09973/) (2026)
- [Mantis: A Versatile Vision-Language-Action Model with Disentangled Visual Foresight](https://mlsanity.com/paper/2511.16175/) (2025)
- [Chain of World: World Model Thinking in Latent Motion](https://mlsanity.com/paper/2603.03195/) (2026)
- [What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agents](https://mlsanity.com/paper/2606.10267/) (2026)
- [SimVLA: A Simple VLA Baseline for Robotic Manipulation](https://mlsanity.com/paper/2602.18224/) (2026)
- [AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation](https://mlsanity.com/paper/2605.22816/) (2026)
- [VLA-Thinker: Boosting Vision-Language-Action Models through Thinking-with-Image Reasoning](https://mlsanity.com/paper/2603.14523/) (2026)
- [PokeVLA: Empowering Pocket-Sized Vision-Language-Action Model with Comprehensive World Knowledge Guidance](https://mlsanity.com/paper/2604.20834/) (2026)
- [TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches](https://mlsanity.com/paper/2603.23117/) (2026)
- [Escaping the Diversity Trap in Robotic Manipulation via Anchor-Centric Adaptation](https://mlsanity.com/paper/2605.07381/) (2026)
- [Afford-VLA: Action-Aligned Visual Planning via Internalized Affordance](https://mlsanity.com/paper/2605.24203/) (2026)
- [From Noise to Intent: Anchoring Generative VLA Policies with Residual Bridges](https://mlsanity.com/paper/2604.21391/) (2026)
- [Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models](https://mlsanity.com/paper/2606.07107/) (2026)
- [SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass](https://mlsanity.com/paper/2602.03134/) (2026)
- [Anticipation-VLA: Solving Long-Horizon Embodied Tasks via Anticipation-based Subgoal Generation](https://mlsanity.com/paper/2605.01772/) (2026)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [VLA-GSE: Boosting Parameter-Efficient Fine-Tuning in VLA with Generalized and Specialized Experts](https://mlsanity.com/paper/2605.06175/) (2026)
- [S2-VLA: State-Space Guided Vision-Language-Action Models for Long-Horizon Manipulation](https://mlsanity.com/paper/2606.27872/) (2026)
- [Rethinking Intermediate Representation for VLM-based Robot Manipulation](https://mlsanity.com/paper/2511.19315/) (2025)
- [HaltNav: Reactive Visual Halting over Lightweight Topological Priors for Robust Vision-Language Navigation](https://mlsanity.com/paper/2603.12696/) (2026)
- [TS-Mask VLA: 2D Temporal-Spatial Masking for Vision-Language-Action Model with Effective Bridging](https://mlsanity.com/paper/2607.09818/) (2026)
- [UniETP: Unifying Environments for Generalizable Embodied Task Planning](https://mlsanity.com/paper/2607.18062/) (2026)
- [RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation](https://mlsanity.com/paper/2607.18709/) (2026)

## Abstract (arXiv)

Recent studies on Vision-Language-Action (VLA) models have shifted from the end-to-end action-generation paradigm toward a pipeline involving task planning followed by action generation, demonstrating improved performance on various complex, long-horizon manipulation tasks. However, existing approaches vary significantly in terms of network architectures, planning paradigms, representations, and training data sources, making it challenging for researchers to identify the precise sources of performance gains and components to be further improved. To systematically investigate the impacts of different planning paradigms and representations isolating from network architectures and training data, in this paper, we introduce VLA-OS, a unified VLA architecture series capable of various task planning paradigms, and design a comprehensive suite of controlled experiments across diverse object categories (rigid and deformable), visual modalities (2D and 3D), environments (simulation and real-world), and end-effectors (grippers and dexterous hands). Our results demonstrate that: 1) visually grounded planning representations are generally better than language planning representations; 2) the Hierarchical-VLA paradigm generally achieves superior or comparable performance than other paradigms on task performance, pretraining, generalization ability, scalability, and continual learning ability, albeit at the cost of slower training and inference speeds.