mlsanityData to 7 Oct 2026

Robotic Control via Embodied Chain-of-Thought Reasoning

arXiv 2407.08693 ↗First posted 11 Jul 2024cs.RO cs.LGcited by 340

On the map: World models → Embodied and robot world models

also in 3 more
Abstract (arXiv)

A key limitation of learned robot control policies is their inability to generalize outside their training data. Recent works on vision-language-action models (VLAs) have shown that the use of large, internet pre-trained vision-language models as the backbone of learned robot policies can substantially improve their robustness and generalization ability. Yet, one of the most exciting capabilities of large vision-language models in other domains is their ability to reason iteratively through complex problems. Can that same capability be brought into robotics to allow policies to improve performance by reasoning about a given task before acting? Naive use of "chain-of-thought" (CoT) style prompting is significantly less effective with standard VLAs because of the relatively simple training examples that are available to them. Additionally, purely semantic reasoning about sub-tasks, as is common in regular CoT, is insufficient for robot policies that need to ground their reasoning in sensory observations and the robot state. To this end, we introduce Embodied Chain-of-Thought Reasoning (ECoT) for VLAs, in which we train VLAs to perform multiple steps of reasoning about plans, sub-tasks, motions, and visually grounded features like object bounding boxes and end effector positions, before predicting the robot action. We design a scalable pipeline for generating synthetic training data for ECoT on large robot datasets. We demonstrate, that ECoT increases the absolute success rate of OpenVLA, the current strongest open-source VLA policy, by 28% across challenging generalization tasks, without any additional robot training data. Additionally, ECoT makes it easier for humans to interpret a policy's failures and correct its behavior using natural language.

Key facts. Cited by 340 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 38 of its 56 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

38 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models16 Oct
  2. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Segment Anythingcited by 8,158
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  12. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
24 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  3. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  4. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  5. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  6. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  7. 2022Visual Language Maps for Robot Navigationcited by 362
  8. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  9. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  10. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  11. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  12. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  13. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  14. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  15. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  16. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  17. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  18. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  19. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  20. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  21. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
  22. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  23. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  24. 2023FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detectioncited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 170; 2026: 156.

340 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  3. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  4. 2024Latent Action Pretraining from Videoscited by 302
  5. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  6. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  7. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  8. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  9. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 189
  10. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  11. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  12. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
38 more
  1. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 142
  2. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  3. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  4. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  5. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 120
  6. 2025NORA: A Small Open-Sourced Generalist Vision Language Action Model for Embodied Taskscited by 119
  7. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  8. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  9. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 101
  10. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  11. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
  12. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  13. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  14. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
  15. 2025ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceivercited by 80
  16. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  17. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  18. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69
  19. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 68
  20. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 67
  21. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  22. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  23. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  24. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 63
  25. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  26. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  27. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  28. 2025Hume: Introducing System-2 Thinking in Visual-Language-Action Modelcited by 56
  29. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 56
  30. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  31. 2025FlowVLA: Visual Chain of Thought-based Motion Reasoning for Vision-Language-Action Modelscited by 50
  32. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  33. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  34. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 47
  35. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 46
  36. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  37. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 45
  38. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43

50 most cited of 340 All 340 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control
  2. 2026VLA-Thinker: Boosting Vision-Language-Action Models through Thinking-with-Image Reasoning
  3. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  4. 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robots
  5. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning
  6. 2026In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
  7. 2026MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation
  8. 2024Improve Vision Language Model Chain-of-thought Reasoning