mlsanityData to 7 Oct 2026

SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning

arXiv 2301.11520 ↗First posted 27 Jan 2023cs.LG cs.AI cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

As previous representations for reinforcement learning cannot effectively incorporate a human-intuitive understanding of the 3D environment, they usually suffer from sub-optimal performances. In this paper, we present Semantic-aware Neural Radiance Fields for Reinforcement Learning (SNeRL), which jointly optimizes semantic-aware neural radiance fields (NeRF) with a convolutional encoder to learn 3D-aware neural implicit representation from multi-view images. We introduce 3D semantic and distilled feature fields in parallel to the RGB radiance fields in NeRF to learn semantic and object-centric representation for reinforcement learning. SNeRL outperforms not only previous pixel-based representations but also recent 3D-aware representations both in model-free and model-based reinforcement learning.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 38 of its 41 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

38 of 41 references.

References · 38

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  7. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  11. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  12. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
26 more
  1. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  2. 2020Reinforcement Learning with Augmented Datacited by 452
  3. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  4. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  5. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  6. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  7. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  8. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  9. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  10. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  11. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  12. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  13. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  14. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  15. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  16. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 69
  17. 2015Multi-view 3D Models from Single Images with a Convolutional Networkcited by 57
  18. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  19. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  20. 2018Learning Actionable Representations from Visual Observationscited by 35
  21. 2017Interpretable Transformations with Encoder-Decoder Networkscited by 33
  22. 2020Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentationcited by 20
  23. 2022Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Imagescited by 12
  24. 2022Task-Induced Representation Learningcited by 10
  25. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 10
  26. 2022Multi-View Dreaming: Multi-View World Model with Contrastive Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 3; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  2. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  3. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  4. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  5. 2024DNAct: Diffusion Guided Multi-Task 3D Policy Learningcited by 22
  6. 2024NeRFs in Robotics: A Surveycited by 16
  7. 2024Reinforcement Learning with Generalizable Gaussian Splattingcited by 11
  8. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 11
  9. 2025ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Modelcited by 10
  10. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  11. 2023Robo360: A 3D Omnispective Multi-Material Robotic Manipulation Datasetcited by 8
  12. 2023Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representationscited by 5
4 more
  1. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4
  2. 2025Efficient Alignment of Unconditioned Action Prior for Language-conditioned Pick and Place in Cluttercited by 3
  3. 2024NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fieldscited by 1
  4. 2026ParticleSplat: Self-supervised Object-centric Latent Particle Splattingcited by 0