mlsanityData to 14 Oct 2026

MoVie: Visual Model-Based Policy Adaptation for View Generalization

arXiv 2307.00972 ↗First posted 3 Jul 2023cs.LG cs.CV cs.ROcited by 16

Not on the map. Why?

Abstract (arXiv)

Visual Reinforcement Learning (RL) agents trained on limited views face significant challenges in generalizing their learned abilities to unseen views. This inherent difficulty is known as the problem of textitview generalization. In this work, we systematically categorize this fundamental problem into four distinct and highly challenging scenarios that closely resemble real-world situations. Subsequently, we propose a straightforward yet effective approach to enable successful adaptation of visual Model-based policies for View generalization (MoVie) during test time, without any need for explicit reward signals and any modification during training time. Our method demonstrates substantial advancements across all four scenarios encompassing a total of 18 tasks sourced from DMControl, xArm, and Adroit, with a relative improvement of 33%, 86%, and 152% respectively. The superior results highlight the immense potential of our approach for real-world robotics applications. Videos are available at https://yangsizhe.github.io/MoVie/ .

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 34 of its 37 references that are in the map. First posted on arXiv 3 Jul 2023.

Builds on

34 of 37 references.

References · 34

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  3. 2015Spatial Transformer Networkscited by 1,429
  4. 2020Mastering Atari with Discrete World Modelscited by 1,007
  5. 2018DeepMind Control Suitecited by 956
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  7. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  8. 2020Reinforcement Learning with Augmented Datacited by 468
  9. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  10. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  11. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  12. 2022Test-Time Training with Masked Autoencoderscited by 212
22 more
  1. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  2. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  3. 20213D Neural Scene Representations for Visuomotor Controlcited by 111
  4. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  5. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 104
  6. 2017Third-Person Imitation Learningcited by 103
  7. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 100
  8. 2021RRL: Resnet as representation for Reinforcement Learningcited by 91
  9. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 85
  10. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  11. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 79
  12. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  13. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 62
  14. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
  15. 2019Test-Time Training with Self-Supervision for Generalization under Distribution Shiftscited by 59
  16. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 56
  17. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  18. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  19. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 34
  20. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  21. 2021Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learningcited by 23
  22. 20203D-OES: Viewpoint-Invariant Object-Factorized Environment Simulatorscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 5; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  2. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 339
  3. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 116
  4. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 41
  5. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  6. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  7. 2024A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learningcited by 14
  8. 2025Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioningcited by 14
  9. 2023Generalizable Visual Reinforcement Learning with Segment Anything Modelcited by 12
  10. 2025ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representationscited by 5
  11. 2025Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulationcited by 2
  12. 2026Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasetscited by 2
4 more
  1. 2024Efficient Training of Generalizable Visuomotor Policies via Control-Aware Augmentationcited by 1
  2. 2025Self-Consistent Model-based Adaptation for Visual Reinforcement Learningcited by 1
  3. 2025Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processescited by 0
  4. 2026VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learningcited by 0