mlsanityData to 7 Oct 2026

CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification

arXiv 2508.21046 ↗First posted 28 Aug 2025cs.CV cs.ROcited by 70

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Recent Vision-Language-Action (VLA) models built on pre-trained Vision-Language Models (VLMs) require extensive post-training, resulting in high computational overhead that limits scalability and deployment.We propose CogVLA, a Cognition-Aligned Vision-Language-Action framework that leverages instruction-driven routing and sparsification to improve both efficiency and performance. CogVLA draws inspiration from human multimodal coordination and introduces a 3-stage progressive architecture. 1) Encoder-FiLM based Aggregation Routing (EFA-Routing) injects instruction information into the vision encoder to selectively aggregate and compress dual-stream visual tokens, forming a instruction-aware latent representation. 2) Building upon this compact visual encoding, LLM-FiLM based Pruning Routing (LFP-Routing) introduces action intent into the language model by pruning instruction-irrelevant visually grounded tokens, thereby achieving token-level sparsity. 3) To ensure that compressed perception inputs can still support accurate and coherent action generation, we introduce V-L-A Coupled Attention (CAtten), which combines causal vision-language attention with bidirectional action parallel decoding. Extensive experiments on the LIBERO benchmark and real-world robotic tasks demonstrate that CogVLA achieves state-of-the-art performance with success rates of 97.4% and 70.0%, respectively, while reducing training costs by 2.5-fold and decreasing inference latency by 2.8-fold compared to OpenVLA. CogVLA is open-sourced and publicly available at https://github.com/JiuTian-VL/CogVLA.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 89 references that are in the map. First posted on arXiv 28 Aug 2025.

Builds on

73 of 89 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better29 May
  2. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models27 Mar
  3. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation26 Mar
  4. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success27 Feb
  5. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution4 Nov
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 67

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  10. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
38 more
  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  5. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  6. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  11. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  12. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  13. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  14. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  15. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  16. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  17. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  18. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  19. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  20. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  21. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  22. 2024RT-H: Action Hierarchies Using Languagecited by 204
  23. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  24. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  25. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  26. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  27. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  28. 2024Behavior Generation with Latent Actionscited by 159
  29. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  30. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  31. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  32. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
  33. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  34. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  35. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  36. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  37. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  38. 2025GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoningcited by 56

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 57.

70 papers.

Sort:

By area:

  1. 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 274
  2. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  3. 2026DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Controlcited by 71
  4. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  5. 2026BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generationcited by 25
  6. 2026Adaptive Action Chunking at Inference-time for Vision-Language-Action Modelscited by 24
  7. 2025SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overheadcited by 23
  8. 2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 22
  9. 2025SemanticVLA: Semantic-Aligned Sparsification and Enhancement for Efficient Robotic Manipulationcited by 20
  10. 2026Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulationcited by 19
  11. 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Recordscited by 16
  12. 2026S-VAM: Shortcut Video-Action Model by Self-Distilling Geometric and Semantic Foresightcited by 16
38 more
  1. 2025Cost-Effective Communication: An Auction-based Method for Language Agent Interactioncited by 14
  2. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoningcited by 14
  3. 2026VLA-Corrector: Lightweight Detect-and-Correct Inference for Adaptive Action Horizoncited by 14
  4. 2025Reasoning-Enhanced Domain-Adaptive Pretraining of Multimodal Large Language Models for Short Video Content Governancecited by 13
  5. 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
  6. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 13
  7. 2026DeltaVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variationcited by 9
  8. 2026HATS: Hardness-Aware Trajectory Synthesis for GUI Agentscited by 9
  9. 2026TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAMcited by 9
  10. 2026Shallow-π: Knowledge Distillation for Flow-based VLAscited by 7
  11. 2026AsyncVLA: An Asynchronous VLA for Fast and Robust Navigation on the Edgecited by 7
  12. 2026Point Tracking Improves World Action Modelscited by 7
  13. 2026AC^2-VLA: Action-Context-Aware Adaptive Computation in Vision-Language-Action Models for Efficient Robotic Manipulationcited by 6
  14. 2026ForeAct: Steering Your VLA with Efficient Visual Foresight Planningcited by 6
  15. 2026RoboRouter: Training-Free Policy Routing for Robotic Manipulationcited by 6
  16. 2026ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulationcited by 6
  17. 2026NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Modelscited by 6
  18. 2026The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Managementcited by 6
  19. 2026ReViP: Mitigating False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalancecited by 5
  20. 2026UAOR: Uncertainty-aware Observation Reinjection for Vision-Language-Action Modelscited by 5
  21. 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Modelcited by 5
  22. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  23. 2026VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesiscited by 4
  24. 2026Foresight Without Seeing: Latent Futures for World Action Modelscited by 4
  25. 2025PhysicalAgent: Towards General Cognitive Robotics with Foundation World Modelscited by 3
  26. 2026TDEC: Deep Embedded Image Clustering with Transformer and Distribution Informationcited by 3
  27. 2026LARA: Latent Action Representation Alignment for Vision-Language-Action Modelscited by 3
  28. 2026Learning Transferable Dynamics Priors from Action to World Modelingcited by 3
  29. 2026AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learningcited by 3
  30. 2026Acting While Understanding: Asynchronous Semantic-Action Decoupling for Real-Time Vision-Language-Action Modelscited by 2
  31. 2026DeVA: Decoupled Video-Action Model with physical guidance for robot policy learningcited by 2
  32. 2026CofactVLA: Deconfounding Vision-Language-Action Models via Counterfactual Interventioncited by 2
  33. 2025Automated Genomic Interpretation via Concept Bottleneck Models for Medical Roboticscited by 1
  34. 2025Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cachecited by 1
  35. 2026Learning to Accelerate Vision-Language-Action Models through Adaptive Visual Token Cachingcited by 1
  36. 2026NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Modelscited by 1
  37. 2026Interpretable Traffic Responsibility from Dashcam Video via Legal Multi Agent Reasoningcited by 1
  38. 2026CNMBI: Determining the Number of Clusters Using Center Pairwise Matching and Boundary Filteringcited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models
  2. 2026What Makes an Efficient VLA? Navigating Action-Head Design, Scaling, and Latency
  3. 2026Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models
  4. 2025Think Twice, Act Once: Token-Aware Compression and Action Reuse for Efficient Inference in Vision-Language-Action Models
  5. 2025SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models
  6. 2025Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inference
  7. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model
  8. 2026StreamingVLA: Streaming Vision-Language-Action Model with Action Flow Matching and Adaptive Early Observation