Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)
Not on the map. Why?
Abstract (arXiv)
Recent 3D generative models, which are capable of generating full object shapes from just a few images, now open up new opportunities in robotics. In this work, we show that 3D generative models can be used to augment a dataset from a single real-world demonstration, after which an omnidirectional policy can be learned within this imagined dataset. We found that this enables a robot to perform a task when initialised from states very far from those observed during the demonstration, including starting from the opposite side of the object relative to the real-world demonstration, significantly reducing the number of demonstrations required for policy learning. Through several real-world experiments across tasks such as grasping objects, opening a drawer, and placing trash into a bin, we study these omnidirectional policies by investigating the effect of various design choices on policy behaviour, and we show superior performance to recent baselines which use alternative methods for data augmentation.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 61 of its 64 references that are in the map. First posted on arXiv 7 Sep 2025.
Builds on
61 of 64 references.
References · 61
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2020Denoising Diffusion Implicit Modelscited by 9,781
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2021Zero-Shot Text-to-Image Generationcited by 4,288
- 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
- 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
- 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
38 more
- 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
- 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
- 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 645
- 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
- 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
- 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 521
- 2020Reinforcement Learning with Augmented Datacited by 468
- 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
- 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
- 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
- 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
- 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
- 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 169
- 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 144
- 2024ALOHA Unleashed: A Simple Recipe for Robot Dexteritycited by 134
- 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 116
- 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 106
- 2024EscherNet: A Generative Model for Scalable View Synthesiscited by 95
- 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 93
- 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 91
- 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 81
- 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Roboticscited by 74
- 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 70
- 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 69
- 2025DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learningcited by 69
- 2023Language Models as Zero-Shot Trajectory Generatorscited by 60
- 2023NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesiscited by 57
- 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 54
- 2023cuRobo: Parallelized Collision-Free Minimum-Jerk Robot Motion Generationcited by 52
- 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 52
- 2025Novel Demonstration Generation with Gaussian Splatting Enables Robust One-Shot Manipulationcited by 48
- 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
- 2025Re3Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulationcited by 43
- 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
- 2024Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Roboticscited by 40
- 2024Instant Policy: In-Context Imitation Learning via Graph Diffusioncited by 39
- 2022Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learningcited by 38
- 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 35
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.