mlsanityData to 7 Oct 2026

Reinforcement Learning with Neural Radiance Fields

arXiv 2206.01634 ↗First posted 3 Jun 2022cs.LG cs.CV cs.ROcited by 43

Not on the map. Why?

Abstract (arXiv)

It is a long-standing problem to find effective representations for training reinforcement learning (RL) agents. This paper demonstrates that learning state representations with supervision from Neural Radiance Fields (NeRFs) can improve the performance of RL compared to other learned representations or even low-dimensional, hand-engineered state information. Specifically, we propose to train an encoder that maps multiple image observations to a latent space describing the objects in the scene. The decoder built from a latent-conditioned NeRF serves as the supervision signal to learn the latent space. An RL algorithm then operates on the learned latent space as its state representation. We call this NeRF-RL. Our experiments indicate that NeRF as supervision leads to a latent space better suited for the downstream RL tasks involving robotic object manipulations like hanging mugs on hooks, pushing objects, or opening doors. Video: https://dannydriess.github.io/nerf-rl

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 79 of its 102 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

79 of 102 references.

References · 79

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2017Mask R-CNNcited by 7,259
  9. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  10. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  11. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  12. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
38 more
  1. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  2. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  3. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  4. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  5. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  6. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  7. 2020Neural Sparse Voxel Fieldscited by 959
  8. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  9. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  10. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  11. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  12. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  13. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  14. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  15. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  16. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  17. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  18. 2021Neural Fields in Visual Computing and Beyondcited by 539
  19. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 510
  20. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 493
  21. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  22. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 446
  23. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  24. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 381
  25. 2021Advances in Neural Renderingcited by 365
  26. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  27. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  28. 2021Mixture of Volumetric Primitives for Efficient Neural Renderingcited by 277
  29. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  30. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  31. 2022Masked Visual Pre-training for Motor Controlcited by 246
  32. 2021DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networkscited by 232
  33. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  34. 2020AutoInt: Automatic Integration for Fast Neural Volume Renderingcited by 204
  35. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  36. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  37. 2020DeRF: Decomposed Radiance Fieldscited by 163
  38. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 15; 2024: 11; 2025: 6; 2026: 5.

43 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  3. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  4. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  5. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  6. 2022Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fieldscited by 58
  7. 2023NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesiscited by 56
  8. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  9. 2023Renderable Neural Radiance Map for Visual Navigationcited by 51
  10. 2023Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmarkcited by 51
  11. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  12. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
31 more
  1. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  2. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
  3. 2023BeyondPixels: A Comprehensive Review of the Evolution of Neural Radiance Fieldscited by 27
  4. 2024DNAct: Diffusion Guided Multi-Task 3D Policy Learningcited by 22
  5. 2023SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learningcited by 16
  6. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  7. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  8. 2022IR-MCL: Implicit Representation-Based Online Global Localizationcited by 14
  9. 2023Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flightcited by 11
  10. 2024Reinforcement Learning with Generalizable Gaussian Splattingcited by 11
  11. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 11
  12. 2022RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Controlcited by 10
  13. 2024Never-Ending Behavior-Cloning Agent for Robotic Manipulationcited by 10
  14. 2025ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Modelcited by 10
  15. 2023Multi-Object Manipulation via Object-Centric Neural Scattering Functionscited by 9
  16. 2025PINGS: Gaussian Splatting Meets Distance Fields within a Point-Based Implicit Neural Mapcited by 9
  17. 2023Robo360: A 3D Omnispective Multi-Material Robotic Manipulation Datasetcited by 8
  18. 2024Multi-view Disentanglement for Reinforcement Learning with Multiple Camerascited by 8
  19. 2024Neural Radiance Fields in Medical Imaging: A Surveycited by 7
  20. 2022One-Shot Neural Fields for 3D Object Understandingcited by 5
  21. 2024PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Cloudscited by 5
  22. 2025Time-Unified Diffusion Policy with Action Discrimination for Robotic Manipulationcited by 5
  23. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  24. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4
  25. 2023Integrated Object Deformation and Contact Patch Estimation from Visuo-Tactile Feedbackcited by 3
  26. 2025Omni-Scan: Creating Visually-Accurate Digital Twin Object Models Using a Bimanual Robot with Handover and Gaussian Splat Mergingcited by 2
  27. 2024NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fieldscited by 1
  28. 2026Uncertainty-Aware Gaussian Map for Vision-Language Navigationcited by 1
  29. 2025Self-Consistent Model-based Adaptation for Visual Reinforcement Learningcited by 0
  30. 2026StegoNGP: 3D Cryptographic Steganography using Instant-NGPcited by 0
  31. 2026ParticleSplat: Self-supervised Object-centric Latent Particle Splattingcited by 0