mlsanityData to 7 Oct 2026

UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent

arXiv 2501.18867 ↗First posted 31 Jan 2025cs.CV cs.AIcited by 99

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 2 more
Abstract (arXiv)

Recent advancements in Vision-Language-Action (VLA) models have leveraged pre-trained Vision-Language Models (VLMs) to improve the generalization capabilities. VLMs, typically pre-trained on vision-language understanding tasks, provide rich semantic knowledge and reasoning abilities. However, prior research has shown that VLMs often focus on high-level semantic content and neglect low-level features, limiting their ability to capture detailed spatial information and understand physical dynamics. These aspects, which are crucial for embodied control tasks, remain underexplored in existing pre-training paradigms. In this paper, we investigate the training paradigm for VLAs, and introduce UP-VLA, a Unified VLA model training with both multi-modal Understanding and future Prediction objectives, enhancing both high-level semantic comprehension and low-level spatial understanding. Experimental results show that UP-VLA achieves a 33% improvement on the Calvin ABC-D benchmark compared to the previous state-of-the-art method. Additionally, UP-VLA demonstrates improved success rates in real-world manipulation tasks, particularly those requiring precise spatial information.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 47 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

39 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers12 Sep
  2. 2024Exploring Failure Cases in Multimodal Reasoning About Physical Dynamics24 Feb
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  4. 2023Vision-Language Foundation Models as Effective Robot Imitators2 Nov
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  6. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  12. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
21 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  3. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  5. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  6. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  7. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  9. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  10. 2022MAGVIT: Masked Generative Video Transformercited by 341
  11. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  12. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  13. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  14. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  15. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  16. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  17. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  18. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  19. 2023Prompt a Robot to Walk with Large Language Modelscited by 44
  20. 2024LM4LV: A Frozen Large Language Model for Low-level Vision Taskscited by 13
  21. 2024Can Transformers Capture Spatial Relations between Objects?cited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 32; 2026: 67.

99 papers.

Sort:

By area:

  1. 2026Causal World Modeling for Robot Controlcited by 289
  2. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  3. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  4. 2025Unified Vision-Language-Action Modelcited by 128
  5. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 104
  6. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  7. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  8. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  9. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  10. 2025DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoningcited by 44
  11. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  12. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 39
38 more
  1. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  2. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 38
  3. 2025Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Processcited by 37
  4. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Modelcited by 33
  5. 2025HiF-VLA: Hindsight, Insight and Foresight through Motion Representation for Vision-Language-Action Modelscited by 30
  6. 2026ACoT-VLA: Action Chain-of-Thought for Vision-Language-Action Modelscited by 30
  7. 2026LaST0: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Modelcited by 28
  8. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
  9. 2025From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priorscited by 25
  10. 2026BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generationcited by 25
  11. 2025MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulationcited by 22
  12. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  13. 2026Latent Reasoning VLA: Latent Thinking and Prediction for Vision-Language-Action Modelscited by 19
  14. 2026Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulationcited by 19
  15. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  16. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  17. 2026FAVLA: A Force-Adaptive Fast-Slow VLA model for Contact-Rich Robotic Manipulationcited by 14
  18. 2026HALO: A Unified Vision-Language-Action Model for Embodied Multimodal Chain-of-Thought Reasoningcited by 13
  19. 2025Unifying Perception and Action: A Hybrid-Modality Pipeline with Implicit Visual Chain-of-Thought for Robotic Action Generationcited by 12
  20. 2025MM-ACT: Learn from Multimodal Parallel Generation to Actcited by 12
  21. 2026VLANeXt: Recipes for Building Strong VLA Modelscited by 12
  22. 2025Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulationcited by 10
  23. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  24. 2026VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memorycited by 10
  25. 2026MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generationcited by 10
  26. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraftcited by 9
  27. 2026Rethinking the Practicality of Vision-language-action Model: A Comprehensive Benchmark and An Improved Baselinecited by 9
  28. 2026DeltaVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variationcited by 9
  29. 2026Fast-dVLA: Accelerating Discrete Diffusion VLA to Real-Time Performancecited by 9
  30. 2026TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAMcited by 9
  31. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  32. 2026AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understandingcited by 8
  33. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  34. 2026One Token Per Frame: Reconsidering Visual Bandwidth in World Models for VLA Policycited by 6
  35. 2026GEM: Generative Supervision Helps Embodied Intelligencecited by 6
  36. 2025ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuningcited by 5
  37. 2026DFM-VLA: Iterative Action Refinement for Robot Manipulation via Discrete Flow Matchingcited by 5
  38. 2026Veo-Act: Enhancing VLA Policies with Frontier Video Modelscited by 5

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Vision-Language-Action Models for Embodied AI
  2. 202510 Open Challenges Steering the Future of Vision-Language-Action Models
  3. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
  4. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning
  5. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing
  6. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  7. 2025Igniting VLMs toward the Embodied Space
  8. 2026Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models