mlsanityData to 7 Oct 2026

NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis

arXiv 2301.08556 ↗First posted 18 Jan 2023cs.LG cs.CV cs.ROcited by 56

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Expert demonstrations are a rich source of supervision for training visual robotic manipulation policies, but imitation learning methods often require either a large number of demonstrations or expensive online expert supervision to learn reactive closed-loop behaviors. In this work, we introduce SPARTN (Synthetic Perturbations for Augmenting Robot Trajectories via NeRF): a fully-offline data augmentation scheme for improving robot policies that use eye-in-hand cameras. Our approach leverages neural radiance fields (NeRFs) to synthetically inject corrective noise into visual demonstrations, using NeRFs to generate perturbed viewpoints while simultaneously calculating the corrective actions. This requires no additional expert supervision or environment interaction, and distills the geometric information in NeRFs into a real-time reactive RGB-only policy. In a simulated 6-DoF visual grasping benchmark, SPARTN improves success rates by 2.8× over imitation learning without the corrective augmentations and even outperforms some methods that use online supervision. It additionally closes the gap between RGB-only and RGB-D success rates, eliminating the previous need for depth sensors. In real-world 6-DoF robotic grasping experiments from limited human demonstrations, our method improves absolute success rates by 22.5% on average, including objects that are traditionally challenging for depth-based methods. See video results at urlhttps://bland.website/spartn.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 63 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

48 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  2. 2020Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulation13 Nov
  3. 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep

Other references · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  5. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  12. 2020Reinforcement Learning with Augmented Datacited by 452
33 more
  1. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  2. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 381
  3. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  4. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  5. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  6. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  7. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  8. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  9. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  10. 2020GRF: Learning a General Radiance Field for 3D Representation and Renderingcited by 183
  11. 2021Vision-Only Robot Navigation in a Neural Radiance Worldcited by 153
  12. 2021Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objectscited by 119
  13. 2018Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approachcited by 117
  14. 2020ACRONYM: A Large-Scale Grasp Dataset Based on Simulationcited by 115
  15. 2013Deep Learning for Detecting Robotic Graspscited by 114
  16. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  17. 2017Grasp Pose Detection in Point Cloudscited by 107
  18. 2013Data-Driven Grasp Synthesis - A Surveycited by 104
  19. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  20. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 98
  21. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  22. 2018EnsembleDAgger: A Bayesian Approach to Safe Imitation Learningcited by 96
  23. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  24. 2014Real-Time Grasp Detection Using Convolutional Neural Networkscited by 75
  25. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  26. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  27. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  28. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  29. 2022Differentiable Physics Simulation of Dynamics-Augmented Neural Objectscited by 45
  30. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  31. 2020Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Cloudscited by 24
  32. 2019Manipulation Trajectory Optimization with Online Grasp Synthesis and Selectioncited by 15
  33. 2021Hierarchical Policies for Cluttered-Scene Grasping with Latent Planscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 17; 2025: 20; 2026: 10.

56 papers.

Sort:

By area:

  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  2. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  3. 2024Advances in 3D Generation: A Surveycited by 62
  4. 2023Waypoint-Based Imitation Learning for Robotic Manipulationcited by 59
  5. 2024Instant Policy: In-Context Imitation Learning via Graph Diffusioncited by 38
  6. 2024JUICER: Data-Efficient Imitation Learning for Robotic Assemblycited by 36
  7. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  8. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
  9. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 34
  10. 2024Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation and Reconstructioncited by 32
  11. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 26
  12. 2024Neural Fields in Robotics: A Surveycited by 21
38 more
  1. 2023Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMscited by 19
  2. 2024SAFE-GIL: SAFEty Guided Imitation Learning for Robotic Systemscited by 13
  3. 2024SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuumcited by 13
  4. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  5. 2025IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environmentscited by 11
  6. 2023Neural radiance fields in the industrial and robotics domain: applications, research opportunities and use casescited by 10
  7. 2025OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learningcited by 10
  8. 2024MILES: Making Imitation Learning Easy with Self-Supervisioncited by 9
  9. 2025Tool-as-Interface: Learning Robot Policies from Observing Human Tool Usecited by 8
  10. 2025DWTNeRF: Boosting Few-shot Neural Radiance Fields via Discrete Wavelet Transformcited by 6
  11. 2023ManifoldNeRF: View-dependent Image Feature Supervision for Few-shot Neural Radiance Fieldscited by 5
  12. 2025Real-Time Operator Takeover for Visuomotor Diffusion Policy Trainingcited by 5
  13. 2025Elastic Motion Policy: An Adaptive Dynamical System for Robust and Efficient One-Shot Imitation Learningcited by 4
  14. 2025D-CODA: Diffusion for Coordinated Dual-Arm Data Augmentationcited by 4
  15. 2025RF4D:Neural Radar Fields for Novel View Synthesis in Outdoor Dynamic Scenescited by 4
  16. 2024SUP-NeRF: A Streamlined Unification of Pose Estimation and NeRF for Monocular 3D Object Reconstructioncited by 3
  17. 2026TTT-Parkour: Rapid Test-Time Training for Perceptive Robot Parkourcited by 3
  18. 2024Enhancing Temporal Consistency in Video Editing by Reconstructing Videos with 3D Gaussian Splattingcited by 2
  19. 2025Sample-Efficient Behavior Cloning Using General Domain Knowledgecited by 2
  20. 2025Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)cited by 2
  21. 2025Self-Augmented Robot Trajectory: Efficient Imitation Learning via Safe Self-augmentation with Demonstrator-annotated Precisioncited by 2
  22. 2026CRAFT: Video Diffusion for Bimanual Robot Data Generationcited by 2
  23. 2024Self-Supervised Geometry-Guided Initialization for Robust Monocular Visual Odometrycited by 1
  24. 2024NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fieldscited by 1
  25. 2025UGNA-VPR: A Novel Training Paradigm for Visual Place Recognition Based on Uncertainty-Guided NeRF Augmentationcited by 1
  26. 2025SVR-GS: Spatially Variant Regularization for Probabilistic Masks in 3D Gaussian Splattingcited by 1
  27. 2025ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentationcited by 1
  28. 2026D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Graspingcited by 1
  29. 2026Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitationcited by 1
  30. 2023Active Vision Reinforcement Learning under Limited Visual Observabilitycited by 0
  31. 2023Tube-NeRF: Efficient Imitation Learning of Visuomotor Policies from MPC using Tube-Guided Data Augmentation and NeRFscited by 0
  32. 2024Learning from Demonstration with Implicit Nonlinear Dynamics Modelscited by 0
  33. 2024NeRF-To-Real Tester: Neural Radiance Fields as Test Image Generators for Vision of Autonomous Systemscited by 0
  34. 2025COMBO-Grasp: Learning Constraint-Based Manipulation for Bimanual Occluded Graspingcited by 0
  35. 2025Denoising-based Contractive Imitation Learningcited by 0
  36. 20253D Scene-Camera Representation with Joint Camera Photometric Optimizationcited by 0
  37. 2025Safety-Aware Imitation Learning via MPC-Guided Disturbance Injectioncited by 0
  38. 2025Enhancing Monocular 3D Hand Reconstruction with Learned Texture Priorscited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Zero-Shot Visual Generalization in Robot Manipulation
  2. 2026GHOST: Hierarchical Sub-Goal Policies for Generalizing Robot Manipulation
  3. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation
  4. 2026Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models
  5. 2024Green Screen Augmentation Enables Scene Generalisation in Robotic Manipulation
  6. 2026WorldSample: Closed-loop Real-robot RL with World Modelling
  7. 2025Imagination at Inference: Synthesizing In-Hand Views for Robust Visuomotor Policy Inference
  8. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer