# R2RDreamer: 3D-aware Data Augmentation for Spatially-generalized 2D Manipulation Policies

arXiv: https://arxiv.org/abs/2606.17040 · HTML: https://mlsanity.com/paper/2606.17040/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 40 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Builds on (most cited first, 30 of 38)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning](https://mlsanity.com/paper/2601.16163/) (2026)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023)
- [Video Depth Anything: Consistent Depth Estimation for Super-Long Videos](https://mlsanity.com/paper/2501.12375/) (2025)
- [RoboDreamer: Learning Compositional World Models for Robot Imagination](https://mlsanity.com/paper/2404.12377/) (2024)
- [DreamGen: Unlocking Generalization in Robot Learning through Video World Models](https://mlsanity.com/paper/2505.12705/) (2025)
- [Data Scaling Laws in Imitation Learning for Robotic Manipulation](https://mlsanity.com/paper/2410.18647/) (2024)
- [Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation](https://mlsanity.com/paper/2508.05635/) (2025)
- [GenSim: Generating Robotic Simulation Tasks via Large Language Models](https://mlsanity.com/paper/2310.01361/) (2023)
- [DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning](https://mlsanity.com/paper/2410.24185/) (2024)
- [Phantom: Training Robots Without Robots Using Only Human Videos](https://mlsanity.com/paper/2503.00779/) (2025)
- [DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning](https://mlsanity.com/paper/2502.16932/) (2025)
- [GigaWorld-0: World Models as Data Engine to Empower Embodied AI](https://mlsanity.com/paper/2511.19861/) (2025)
- [World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Training](https://mlsanity.com/paper/2509.24948/) (2025)
- [GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs](https://mlsanity.com/paper/2410.03645/) (2024)
- [SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment](https://mlsanity.com/paper/2410.18907/) (2024)
- [Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware](https://mlsanity.com/paper/2505.09601/) (2025)

## Abstract (arXiv)

Spatial generalization is critical for imitation-learned manipulation policies, but achieving it typically requires scaling demonstrations across diverse object poses, robot configurations, and camera viewpoints. Data augmentation from a few source demonstrations offers a practical alternative to costly real-world collection. Simulation-based augmentation can create controllable variation, but requires complex environment and object setup and may introduce a sim-to-real gap. Recent real-to-real methods avoid these issues by jointly editing 3D observations and action trajectories from real demonstrations, yet they still rely on strong 3D scene parsing and geometry completion, and often produce observations tailored to 3D pointcloud policies rather than RGB-based 2D policies. We propose R2RDreamer, a real-to-real demonstration augmentation framework that preserves the geometric consistency of 3D action-observation editing while moving visual completion to 2D video space. Specifically, R2RDreamer first performs lightweight 3D augmentation by editing incomplete object pointclouds and end-effector trajectories in a shared 3D frame; it then projects the edited scene into masked image-space control videos with occlusion-aware reasoning and uses a dense-control image-to-video model to complete temporally coherent RGB observations. Experiments on spatially shifted manipulation tasks with both 2D diffusion-style policies and vision-language-action policies show that R2RDreamer improves spatial generalization from limited source demonstrations, with analyses validating the contributions of 3D editing, occlusion-aware projection, and video completion.