mlsanityData to 7 Oct 2026

3D-MVP: 3D Multiview Pretraining for Robotic Manipulation

arXiv 2406.18158 ↗First posted 26 Jun 2024cs.RO cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent works have shown that visual pretraining on egocentric datasets using masked autoencoders (MAE) can improve generalization for downstream robotics tasks. However, these approaches pretrain only on 2D images, while many robotics applications require 3D scene understanding. In this work, we propose 3D-MVP, a novel approach for 3D Multi-View Pretraining using masked autoencoders. We leverage Robotic View Transformer (RVT), which uses a multi-view transformer to understand the 3D scene and predict gripper pose actions. We split RVT's multi-view transformer into visual encoder and action decoder, and pretrain its visual encoder using masked autoencoding on large-scale 3D datasets such as Objaverse. We evaluate 3D-MVP on a suite of virtual robot manipulation tasks and demonstrate improved performance over baselines. Our results suggest that 3D-aware pretraining is a promising approach to improve generalization of vision-based robotic manipulation policies. Project site: https://jasonqsy.github.io/3DMVP

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 42 of its 47 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

42 of 47 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  12. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
30 more
  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  2. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  3. 2022A Generalist Agentcited by 802
  4. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  5. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  6. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  8. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  9. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  10. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  12. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  13. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  14. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  15. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  16. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  17. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  18. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  19. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  20. 2022Masked Visual Pre-training for Motor Controlcited by 246
  21. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  22. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  23. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  24. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  25. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  26. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  27. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  28. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  29. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 41
  30. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 8.

10 papers.

  1. 2026Any3D-VLA: Enhancing VLA Robustness via Diverse Point Cloudscited by 13
  2. 2026PEAfowl: Perception-Enhanced Multi-View Vision-Language-Action for Bimanual Manipulationcited by 6
  3. 2024PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Roboticscited by 4
  4. 2025Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulationcited by 2
  5. 2026CLAMP: Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretrainingcited by 1
  6. 2026Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulationcited by 1
  7. 2026Structural Action Transformer for 3D Dexterous Manipulationcited by 0
  8. 2026Spatially Conditioned Diffusion Policy: Learning Precise and Robust Manipulation with a Single RGB Cameracited by 0
  9. 2026VOCA: Visual Odometry with Codec Awarenesscited by 0
  10. 2026Learning 3D Affordances for Blade Insertion in Cluttered Stowingcited by 0