mlsanityData to 7 Oct 2026

Multi-View Masked World Models for Visual Robotic Manipulation

arXiv 2302.02408 ↗First posted 5 Feb 2023cs.RO cs.CV cs.LGcited by 78

On the map: World models → Embodied and robot world models

Abstract (arXiv)

Visual robotic manipulation research and applications often use multiple cameras, or views, to better perceive the world. How else can we utilize the richness of multi-view data? In this paper, we investigate how to learn good representations with multi-view data and utilize them for visual robotic manipulation. Specifically, we train a multi-view masked autoencoder which reconstructs pixels of randomly masked viewpoints and then learn a world model operating on the representations from the autoencoder. We demonstrate the effectiveness of our method in a range of scenarios, including multi-view control and single-view control with auxiliary cameras for representation learning. We also show that the multi-view masked autoencoder trained with multiple randomized viewpoints enables training a policy with strong viewpoint randomization and transferring the policy to solve real-robot tasks without camera calibration and an adaptation procedure. Video demonstrations are available at: https://sites.google.com/view/mv-mwm.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 44 of its 53 references that are in the map. First posted on arXiv 5 Feb 2023.

Builds on

44 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Masked World Models for Visual Control28 Jun
  2. 2019Solving Rubik's Cube with a Robot Hand16 Oct

Other references · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
30 more
  1. 2020Mastering Atari with Discrete World Modelscited by 990
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  5. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  6. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  7. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  8. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  9. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  10. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  11. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  12. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  13. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  14. 2022Masked Visual Pre-training for Motor Controlcited by 246
  15. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  16. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  17. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  18. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  19. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  20. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  21. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  22. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  23. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  24. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  25. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  26. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  27. 2020Learning Precise 3D Manipulation from Multiple Uncalibrated Camerascited by 18
  28. 2022Coarse-to-Fine Q-attention with Learned Path Rankingcited by 13
  29. 2021Asynchronous Multi-View SLAMcited by 12
  30. 2022Coarse-to-fine Q-attention with Tree Expansioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 22; 2025: 24; 2026: 19.

78 papers.

Sort:

By area:

  1. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  2. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  3. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 60
  4. 2023Robot Learning with Sensorimotor Pre-trainingcited by 51
  5. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  6. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  7. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
  8. 2024Masked Image Modeling: A Surveycited by 28
  9. 2023The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learningcited by 24
  10. 2025GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Modelcited by 24
  11. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 20
  12. 2023TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfercited by 18
38 more
  1. 2024Privileged Sensing Scaffolds Reinforcement Learningcited by 18
  2. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  3. 2023FOCUS: Object-Centric World Models for Robotics Manipulationcited by 17
  4. 2025RoboHorizon: An LLM-Assisted Multi-View World Model for Long-Horizon Robotic Manipulationcited by 17
  5. 2023A Universal Semantic-Geometric Representation for Robotic Manipulationcited by 16
  6. 2025Contrastive Representation Regularization for Vision-Language-Action Modelscited by 16
  7. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  8. 2025Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioningcited by 14
  9. 2024Leveraging Locality to Boost Sample Efficiency in Robotic Manipulationcited by 13
  10. 2024Open-World Reinforcement Learning over Long Short-Term Imaginationcited by 13
  11. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  12. 2025GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulationcited by 11
  13. 2026Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Testcited by 11
  14. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 11
  15. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  16. 2024Visual Representation Learning with Stochastic Frame Predictioncited by 10
  17. 20243D-MVP: 3D Multiview Pretraining for Robotic Manipulationcited by 10
  18. 2023Combining Reconstruction and Contrastive Methods for Multimodal Representations in RLcited by 9
  19. 2023Language-Conditioned Path Planningcited by 9
  20. 2024World Models for General Surgical Graspingcited by 9
  21. 2025Mobi-π: Mobilizing Your Robot Learning Policycited by 9
  22. 2023DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothingcited by 8
  23. 2023Robo360: A 3D Omnispective Multi-Material Robotic Manipulation Datasetcited by 8
  24. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  25. 2026Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Trainingcited by 8
  26. 2026AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Modelscited by 7
  27. 2024MV2MAE: Multi-View Video Masked Autoencoderscited by 6
  28. 2025A Data-Centric Revisit of Pre-Trained Vision Models for Robot Learningcited by 6
  29. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  30. 2025View Invariant Learning for Vision-Language Navigation in Continuous Environmentscited by 6
  31. 2026PEAfowl: Perception-Enhanced Multi-View Vision-Language-Action for Bimanual Manipulationcited by 6
  32. 2026Multi-Camera View Scaling for Data-Efficient Robot Imitation Learningcited by 6
  33. 2024Redundancy-aware Action Spaces for Robot Learningcited by 5
  34. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  35. 2024PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Cloudscited by 5
  36. 2025AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoningcited by 5
  37. 2025ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representationscited by 5
  38. 2025SpatialActor: Exploring Disentangled Spatial Representations for Robust Robotic Manipulationcited by 5

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation
  2. 2021Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations
  3. 2025Zero-Shot Visual Generalization in Robot Manipulation
  4. 2026Viewpoint Matters: Dynamically Optimizing Viewpoints with Masked Autoencoder for Visual Manipulation
  5. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation
  6. 2026RoboMP-DINOv2: Prompts, Not Filters for Robust Robot Manipulation ↗
  7. 2026Dexterity-BEV: Aligning 3D World and Actions for Generalizable Robot Policies Learning
  8. 2025IGen: Scalable Data Generation for Robot Learning from Open-World Images