mlsanityData to 7 Oct 2026

NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fields

arXiv 2411.02482 ↗First posted 4 Nov 2024cs.RO cs.CV cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Training a policy that can generalize to unknown objects is a long standing challenge within the field of robotics. The performance of a policy often drops significantly in situations where an object in the scene was not seen during training. To solve this problem, we present NeRF-Aug, a novel method that is capable of teaching a policy to interact with objects that are not present in the dataset. This approach differs from existing approaches by leveraging the speed, photorealism, and 3D consistency of a neural radiance field for augmentation. NeRF-Aug both creates more photorealistic data and runs 63% faster than existing methods. We demonstrate the effectiveness of our method on 5 tasks with 9 novel objects that are not present in the expert demonstrations. We achieve an average performance boost of 55.6% when comparing our method to the next best method. You can see video results at https://nerf-aug.github.io.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 50 of its 58 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

50 of 58 references.

References · 50

  1. 2023Segment Anythingcited by 8,158
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  6. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  9. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  10. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  11. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  12. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
38 more
  1. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  2. 2018Behavioral Cloning from Observationcited by 482
  3. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 465
  4. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  5. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  6. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 193
  7. 2022NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fieldscited by 188
  8. 2023Putting the Object Back into Video Object Segmentationcited by 182
  9. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
  10. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  11. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  12. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  13. 2021Efficient Neural Radiance Fields for Interactive Free-viewpoint Videocited by 150
  14. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  15. 2021NeRFReN: Neural Radiance Fields with Reflectionscited by 121
  16. 2021Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objectscited by 119
  17. 2023F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectoriescited by 118
  18. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  19. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 103
  20. 2024BAKU: An Efficient Transformer for Multi-Task Policy Learningcited by 80
  21. 2022Orbeez-SLAM: A Real-time Monocular Visual SLAM with ORB Features and NeRF-realized Mappingcited by 79
  22. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  23. 2023NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesiscited by 56
  24. 2022Uncertainty Guided Policy for Active Robotic 3D Reconstruction using Neural Radiance Fieldscited by 53
  25. 2022NAN: Noise-Aware NeRFs for Burst-Denoisingcited by 48
  26. 2024RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learningcited by 46
  27. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  28. 2022NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fieldscited by 41
  29. 2023Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perceptioncited by 40
  30. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 35
  31. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 34
  32. 2022Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervisioncited by 29
  33. 2023MEGA-DAgger: Imitation Learning with Multiple Imperfect Expertscited by 17
  34. 2023SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learningcited by 16
  35. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  36. 20216-DoF Contrastive Grasp Proposal Networkcited by 12
  37. 2024ARDuP: Active Region Video Diffusion for Universal Policiescited by 10
  38. 2023Perceiving Unseen 3D Objects by Poking the Objectscited by 5

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)cited by 2