mlsanityData to 7 Oct 2026

RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation

arXiv 2406.18977 ↗First posted 27 Jun 2024cs.RO cs.CL cs.CVcited by 20

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Utilizing Vision-Language Models (VLMs) for robotic manipulation represents a novel paradigm, aiming to enhance the model's ability to generalize to new objects and instructions. However, due to variations in camera specifications and mounting positions, existing methods exhibit significant performance disparities across different robotic platforms. To address this challenge, we propose RoboUniView in this paper, an innovative approach that decouples visual feature extraction from action learning. We first learn a unified view representation from multi-perspective views by pre-training on readily accessible data, and then derive actions from this unified view representation to control robotic manipulation. This unified view representation more accurately mirrors the physical world and is not constrained by the robotic platform's camera parameters. Thanks to this methodology, we achieve state-of-the-art performance on the demanding CALVIN benchmark, enhancing the success rate in the D → D setting from 93.0% to 96.2%, and in the ABC → D setting from 92.2% to 94.2%. Moreover, our model exhibits outstanding adaptability and flexibility: it maintains high performance under unseen camera parameters, can utilize multiple datasets with varying camera parameters, and is capable of joint cross-task learning across datasets. Code is provided for re-implementation. https://github.com/liufanfanlff/RoboUniview

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 57 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

51 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Vision-Language Foundation Models as Effective Robot Imitators2 Nov
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  3. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May
  4. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  5. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  10. 2022Planning-oriented Autonomous Drivingcited by 1,068
  11. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
34 more
  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  2. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  3. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  6. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  7. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  8. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  9. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  10. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  11. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  12. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  13. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  14. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  15. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  16. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  17. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  18. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  19. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  20. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  21. 2023Language-Driven Representation Learning for Roboticscited by 163
  22. 2019Goal-conditioned Imitation Learningcited by 159
  23. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  24. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  25. 2020Visual Imitation Made Easycited by 111
  26. 2021Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Imagescited by 98
  27. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 95
  28. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  29. 2020Transformers for One-Shot Visual Imitationcited by 77
  30. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  31. 2020A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye Viewcited by 71
  32. 2022Language Control Diffusion: Efficiently Scaling through Space, Time, and Taskscited by 20
  33. 2023Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Datacited by 20
  34. 2024VIHE: Virtual In-Hand Eye Transformer for 3D Robotic Manipulationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 7.

20 papers.

Sort:

By area:

  1. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  3. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  4. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  5. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  6. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  7. 2024Law of Vision Representation in MLLMscited by 16
  8. 2024RoboCAS: A Benchmark for Robotic Manipulation in Complex Object Arrangement Scenarioscited by 7
  9. 2026Future Optical Flow Prediction Improves Robot Control & Video Generationcited by 7
  10. 2025View Invariant Learning for Vision-Language Navigation in Continuous Environmentscited by 6
  11. 2026Learning to Act Robustly with View-Invariant Latent Actionscited by 6
  12. 2025ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representationscited by 5
8 more
  1. 2025ADPro: a Test-time Adaptive Diffusion Policy via Manifold-constrained Denoising and Task-aware Initialization for Robotic Manipulationcited by 4
  2. 2026CF-VLA: Efficient Coarse-to-Fine Action Generation for Vision-Language-Action Policiescited by 4
  3. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challengescited by 3
  4. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Modelscited by 2
  5. 2025Boosting Robotic Manipulation Generalization with Minimal Costly Datacited by 1
  6. 2025TopoDiT-3D: Topology-Aware Diffusion Transformer with Bottleneck Structure for 3D Point Cloud Generationcited by 1
  7. 2026See like a Robot: Robot-Centric Pointmaps for VLA Modelscited by 1
  8. 2026Disentangling Spurious Correlations in Vision-Language-Action Models via Predicting Domain-Invariant Latent Lookaheadcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation
  2. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
  3. 2025VLA-LPAF: Lightweight Perspective-Adaptive Fusion for Vision-Language-Action to Enable More Unconstrained Robotic Manipulation
  4. 2026V-Link: Recovering Lost Visual Representations in Action DiT for Vision-Language-Action Models
  5. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation
  6. 2025CronusVLA: Towards Efficient and Robust Manipulation via Multi-Frame Vision-Language-Action Modeling
  7. 2025Improving Generalization of Language-Conditioned Robot Manipulation
  8. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization