# View-Invariant Policy Learning via Zero-Shot Novel View Synthesis

arXiv: https://arxiv.org/abs/2409.03685 · HTML: https://mlsanity.com/paper/2409.03685/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 46 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 30 of 38)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [ReconFusion: 3D Reconstruction with Diffusion Priors](https://mlsanity.com/paper/2312.02981/) (2023)
- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models](https://mlsanity.com/paper/2310.10639/) (2023)
- [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023)
- [GRF: Learning a General Radiance Field for 3D Representation and Rendering](https://mlsanity.com/paper/2010.04595/) (2020)
- [Scaling Robot Learning with Semantically Imagined Experience](https://mlsanity.com/paper/2302.11550/) (2023)
- [THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation](https://mlsanity.com/paper/2402.08191/) (2024)
- [DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics](https://mlsanity.com/paper/2210.02438/) (2022)
- [Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation](https://mlsanity.com/paper/2307.03659/) (2023)
- [Multi-View Masked World Models for Visual Robotic Manipulation](https://mlsanity.com/paper/2302.02408/) (2023)
- [CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning](https://mlsanity.com/paper/2212.05711/) (2022)
- [Learning Generalizable Manipulation Policies with Object-Centric 3D Representations](https://mlsanity.com/paper/2310.14386/) (2023)

## Built on by (most cited first, 30 of 35)

- [DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning](https://mlsanity.com/paper/2410.24185/) (2024)
- [OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction](https://mlsanity.com/paper/2509.26633/) (2025)
- [Generalizable Humanoid Manipulation with 3D Diffusion Policies](https://mlsanity.com/paper/2410.10803/) (2024)
- [Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation](https://mlsanity.com/paper/2508.06426/) (2025)
- [GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Model](https://mlsanity.com/paper/2509.14117/) (2025)
- [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024)
- [Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimization](https://mlsanity.com/paper/2502.20382/) (2025)
- [AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletons](https://mlsanity.com/paper/2503.03081/) (2025)
- [Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning](https://mlsanity.com/paper/2510.02268/) (2025)
- [MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction](https://mlsanity.com/paper/2602.03668/) (2026)
- [AnyView: Synthesizing Any Novel View in Dynamic Scenes](https://mlsanity.com/paper/2601.16982/) (2026)
- [Mobi-π: Mobilizing Your Robot Learning Policy](https://mlsanity.com/paper/2505.23692/) (2025)
- [OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning](https://mlsanity.com/paper/2512.13100/) (2025)
- [Learning to Act Robustly with View-Invariant Latent Actions](https://mlsanity.com/paper/2601.02994/) (2026)
- [AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models](https://mlsanity.com/paper/2603.05868/) (2026)
- [See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrations](https://mlsanity.com/paper/2512.07582/) (2025)
- [D-CODA: Diffusion for Coordinated Dual-Arm Data Augmentation](https://mlsanity.com/paper/2505.04860/) (2025)
- [ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representations](https://mlsanity.com/paper/2509.11125/) (2025)
- [Invariance Co-training for Robot Visual Generalization](https://mlsanity.com/paper/2512.05230/) (2025)
- [CRAFT: Video Diffusion for Bimanual Robot Data Generation](https://mlsanity.com/paper/2604.03552/) (2026)
- [ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation](https://mlsanity.com/paper/2509.19454/) (2025)
- [From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model](https://mlsanity.com/paper/2607.05396/) (2026)
- [Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)](https://mlsanity.com/paper/2509.06191/) (2025)
- [Observer-Actor: Active Vision Imitation Learning with Sparse-View Gaussian Splatting](https://mlsanity.com/paper/2511.18140/) (2025)
- [Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasets](https://mlsanity.com/paper/2601.09605/) (2026)
- [Selective Cross-View Consistency for World Action Models: Held-Out Viewpoint Robustness Without Test-Time Camera Information](https://mlsanity.com/paper/2608.21402/) (2026)
- [R2S-EGO: Dual-Proxy Refinement for Sparse-Capture Real-to-Sim](https://mlsanity.com/paper/2608.06827/) (2026)
- [Cross-View Action Consistency for Camera-Robust Vision-Language-Action Policies](https://mlsanity.com/paper/2608.06965/) (2026)
- [UniviewVLA: A Unified Multiview Vision-Language-Action Model with World Modeling](https://mlsanity.com/paper/2606.21501/) (2026)
- [The Moving Eye: Enhancing VLA Spatial Generalization via Hybrid Dynamic Data Collection](https://mlsanity.com/paper/2607.02322/) (2026)

## Abstract (arXiv)

Large-scale visuomotor policy learning is a promising approach toward developing generalizable manipulation systems. Yet, policies that can be deployed on diverse embodiments, environments, and observational modalities remain elusive. In this work, we investigate how knowledge from large-scale visual data of the world may be used to address one axis of variation for generalizable manipulation: observational viewpoint. Specifically, we study single-image novel view synthesis models, which learn 3D-aware scene-level priors by rendering images of the same scene from alternate camera viewpoints given a single input image. For practical application to diverse robotic data, these models must operate zero-shot, performing view synthesis on unseen tasks and environments. We empirically analyze view synthesis models within a simple data-augmentation scheme that we call View Synthesis Augmentation (VISTA) to understand their capabilities for learning viewpoint-invariant policies from single-viewpoint demonstration data. Upon evaluating the robustness of policies trained with our method to out-of-distribution camera viewpoints, we find that they outperform baselines in both simulated and real-world manipulation tasks. Videos and additional visualizations are available at https://s-tian.github.io/projects/vista.