mlsanityData to 7 Oct 2026

Neural Radiance Field Codebooks

arXiv 2301.04101 ↗First posted 10 Jan 2023cs.CV cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Compositional representations of the world are a promising step towards enabling high-level scene understanding and efficient transfer to downstream tasks. Learning such representations for complex scenes and tasks remains an open challenge. Towards this goal, we introduce Neural Radiance Field Codebooks (NRC), a scalable method for learning object-centric representations through novel view reconstruction. NRC learns to reconstruct scenes from novel views using a dictionary of object codes which are decoded through a volumetric renderer. This enables the discovery of reoccurring visual and geometric patterns across scenes which are transferable to downstream tasks. We show that NRC representations transfer well to object navigation in THOR, outperforming 2D and 3D representation learning methods by 3.1% success rate. We demonstrate that our approach is able to perform unsupervised segmentation for more complex synthetic (THOR) and real scenes (NYU Depth) better than prior methods (29% relative improvement). Finally, we show that NRC improves on the task of depth ordering by 5.5% accuracy in THOR.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 10 Jan 2023.

Builds on

42 of 54 references.

References · 42

  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  3. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  7. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  8. 2020Object-Centric Learning with Slot Attentioncited by 865
  9. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  10. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  11. 2018On Evaluation of Embodied Navigation Agentscited by 526
  12. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
30 more
  1. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  2. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  3. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  4. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  5. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  6. 2021NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Imagescited by 336
  7. 2021Conditional Object-Centric Learning from Videocited by 247
  8. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  9. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  10. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  11. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  12. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  13. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  14. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  15. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  16. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  17. 2021NeRF-VAE: A Geometry Aware 3D Scene Generative Modelcited by 111
  18. 2021Unsupervised Discovery of Object Radiance Fieldscited by 100
  19. 2021ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentationcited by 93
  20. 2022Object Scene Representation Transformercited by 90
  21. 2021Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentationcited by 88
  22. 2022NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenescited by 71
  23. 2021SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decompositioncited by 69
  24. 2017SeGAN: Segmenting and Generating the Invisiblecited by 60
  25. 2022Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulationcited by 57
  26. 2021Unsupervised Layered Image Decomposition into Object Prototypescited by 50
  27. 2021MarioNette: Self-Supervised Sprite Learningcited by 35
  28. 2022Unsupervised Discovery and Composition of Object Light Fieldscited by 28
  29. 2022Inductive Biases for Object-Centric Representations in the Presence of Complex Texturescited by 13
  30. 2021LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 0; 2026: 1.

7 papers.

  1. 2023Factor Fields: A Unified Framework for Neural Fields and Beyondcited by 58
  2. 2023OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fieldscited by 34
  3. 2023RICO: Regularizing the Unobservable for Indoor Compositional Reconstructioncited by 19
  4. 2023Selective Visual Representations Improve Convergence and Generalization for Embodied AIcited by 19
  5. 2024Bootstrapping Top-down Information for Self-modulating Slot Attentioncited by 6
  6. 2024The Interpretability of Codebooks in Model-Based Reinforcement Learning is Limitedcited by 0
  7. 2026LooC: Effective Low-Dimensional Codebook for Compositional Vector Quantizationcited by 0