mlsanityData to 7 Oct 2026

Octo: An Open-Source Generalist Robot Policy

arXiv 2405.12213 ↗First posted 20 May 2024cs.RO cs.LGcited by 1,427

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Large policies pretrained on diverse robot datasets have the potential to transform robotic learning: instead of training new policies from scratch, such generalist robot policies may be finetuned with only a little in-domain data, yet generalize broadly. However, to be widely applicable across a range of robotic learning scenarios, environments, and tasks, such policies need to handle diverse sensors and action spaces, accommodate a variety of commonly used robotic platforms, and finetune readily and efficiently to new domains. In this work, we aim to lay the groundwork for developing open-source, widely applicable, generalist policies for robotic manipulation. As a first step, we introduce Octo, a large transformer-based policy trained on 800k trajectories from the Open X-Embodiment dataset, the largest robot manipulation dataset to date. It can be instructed via language commands or goal images and can be effectively finetuned to robot setups with new sensory inputs and action spaces within a few hours on standard consumer GPUs. In experiments across 9 robotic platforms, we demonstrate that Octo serves as a versatile policy initialization that can be effectively finetuned to new observation and action spaces. We also perform detailed ablations of design decisions for the Octo model, from architecture to training data, to guide future research on building generalist robot models.

Key facts. Cited by 1,427 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 80 of its 99 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

80 of 99 references.

References · 80

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  11. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
38 more
  1. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  5. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2016End to End Learning for Self-Driving Carscited by 1,000
  8. 2021Scaling Vision Transformerscited by 993
  9. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  10. 2022A Generalist Agentcited by 802
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  13. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  14. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  15. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  16. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  17. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  18. 2022Visual Language Maps for Robot Navigationcited by 362
  19. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  20. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  21. 2019Causal Confusion in Imitation Learningcited by 286
  22. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  23. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  24. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  25. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  26. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 234
  27. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  28. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  29. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  30. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  31. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  32. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  33. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  34. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  35. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  36. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  37. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  38. 2022Human-to-Robot Imitation in the Wildcited by 157

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 141; 2025: 519; 2026: 767.

1,427 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  3. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  4. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  5. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  6. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  7. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  8. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  9. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  10. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  11. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  12. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
38 more
  1. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  2. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  3. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  4. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  5. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  6. 2024Latent Action Pretraining from Videoscited by 302
  7. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  8. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  9. 2026Causal World Modeling for Robot Controlcited by 289
  10. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  11. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  12. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  13. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  14. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  15. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  16. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  17. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  18. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  19. 2025Real-Time Execution of Action Chunking Flow Policiescited by 199
  20. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  21. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  22. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  23. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  24. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 167
  25. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  26. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  27. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
  28. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  29. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  30. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  31. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  32. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  33. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  34. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 142
  35. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  36. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 140
  37. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  38. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138

50 most cited of 1,427 All 1,427 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning
  2. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learning
  3. 2026Robots Need More than VLA and World Models
  4. 2024RoLD: Robot Latent Diffusion for Multi-task Policy Modeling
  5. 2026AXIS: A Growable Community-Driven Data Engine for Scalable Robot Manipulation
  6. 2025PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies
  7. 2024Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Frames
  8. 2026ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm