mlsanityData to 7 Oct 2026

CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation

arXiv 2411.19650 ↗First posted 29 Nov 2024cs.RO cs.AI cs.CLcited by 411

On the map: Vision-language-action models → Generalist VLA models and benchmarks

Abstract (arXiv)

The advancement of large Vision-Language-Action (VLA) models has significantly improved robotic manipulation in terms of language-guided task execution and generalization to unseen scenarios. While existing VLAs adapted from pretrained large Vision-Language-Models (VLM) have demonstrated promising generalizability, their task performance is still unsatisfactory as indicated by the low tasks success rates in different environments. In this paper, we present a new advanced VLA architecture derived from VLM. Unlike previous works that directly repurpose VLM for action prediction by simple action quantization, we propose a omponentized VLA architecture that has a specialized action module conditioned on VLM output. We systematically study the design of the action module and demonstrates the strong performance enhancement with diffusion action transformers for action sequence modeling, as well as their favorable scaling behaviors. We also conduct comprehensive experiments and ablation studies to evaluate the efficacy of our models with varied designs. The evaluation on 5 robot embodiments in simulation and real work shows that our model not only significantly surpasses existing VLAs in task performance and but also exhibits remarkable adaptation to new robots and generalization to unseen objects and backgrounds. It exceeds the average success rates of OpenVLA which has similar model size (7B) with ours by over 35% in simulated evaluation and 55% in real robot experiments. It also outperforms the large RT-2-X model (55B) by 18% absolute success rates in simulation. Code and models can be found on our project page (https://cogact.github.io/).

Key facts. Cited by 411 papers in the map (data through 7 Oct 2026). Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models. Builds on 58 of its 67 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

58 of 67 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation19 Sep
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Model13 Jun
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  6. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  7. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  10. 2022High Fidelity Neural Audio Compressioncited by 673
  11. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  12. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  13. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  14. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  15. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  16. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  17. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  18. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  19. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  20. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  21. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  22. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  23. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  24. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  25. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  26. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  27. 2023Structured World Models from Human Videoscited by 144
  28. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  29. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  30. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  31. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  32. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  33. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  34. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 82
  35. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  36. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 79
  37. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  38. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 170; 2026: 236.

411 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  3. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  4. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
  5. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  6. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  7. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 189
  8. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  9. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  10. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  11. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  12. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
38 more
  1. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  2. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 142
  3. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  5. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  6. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 120
  7. 2025GR-3 Technical Reportcited by 110
  8. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  9. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  10. 2026ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learningcited by 94
  11. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  12. 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 90
  13. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 89
  14. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 83
  15. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  16. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  17. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  18. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  19. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
  20. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 70
  21. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 69
  22. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 68
  23. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 68
  24. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  25. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  26. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  27. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  28. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 61
  29. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  30. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  31. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  32. 2025Hume: Introducing System-2 Thinking in Visual-Language-Action Modelcited by 56
  33. 2025SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Accelerationcited by 56
  34. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 56
  35. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  36. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  37. 2025TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimizationcited by 47
  38. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 46

50 most cited of 411 All 411 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks
  2. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulation
  3. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
  4. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modification
  5. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model
  6. 2026S2-VLA: State-Space Guided Vision-Language-Action Models for Long-Horizon Manipulation
  7. 2026M2-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills
  8. 2025VITA-VLA: Efficiently Teaching Vision-Language Models to Act via Action Expert Distillation