mlsanityData to 7 Oct 2026

FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models

arXiv 2303.12786 ↗First posted 22 Mar 2023cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

Recent works on generalizable NeRFs have shown promising results on novel view synthesis from single or few images. However, such models have rarely been applied on other downstream tasks beyond synthesis such as semantic understanding and parsing. In this paper, we propose a novel framework named FeatureNeRF to learn generalizable NeRFs by distilling pre-trained vision foundation models (e.g., DINO, Latent Diffusion). FeatureNeRF leverages 2D pre-trained foundation models to 3D space via neural rendering, and then extract deep features for 3D query points from NeRF MLPs. Consequently, it allows to map 2D images to continuous 3D semantic feature volumes, which can be used for various downstream tasks. We evaluate FeatureNeRF on tasks of 2D/3D semantic keypoint transfer and 2D/3D object part segmentation. Our extensive experiments demonstrate the effectiveness of FeatureNeRF as a generalizable 3D semantic feature extractor. Our project page is available at https://jianglongye.com/featurenerf/ .

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 61 of its 73 references that are in the map. First posted on arXiv 22 Mar 2023.

Builds on

61 of 73 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  4. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  6. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  7. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  8. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  9. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  10. 2019Relational Knowledge Distillationcited by 873
  11. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  12. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  13. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  14. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
  15. 2019Contrastive Representation Distillationcited by 734
  16. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  17. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  18. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  19. 2020Learning Continuous Image Representation with Local Implicit Image Functioncited by 550
  20. 2021Neural Fields in Visual Computing and Beyondcited by 539
  21. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  22. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  23. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  24. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  25. 2021Advances in Neural Renderingcited by 365
  26. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  27. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  28. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  29. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  30. 2021Diffusion Models for Implicit Image Segmentation Ensemblescited by 198
  31. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  32. 2020GRF: Learning a General Radiance Field for 3D Representation and Renderingcited by 183
  33. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  34. 2021COTR: Correspondence Transformer for Matching Across Imagescited by 182
  35. 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 166
  36. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  37. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  38. 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 140

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 23; 2025: 16; 2026: 3.

47 papers.

Sort:

By area:

  1. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 361
  2. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  3. 2024Large Spatial Model: End-to-end Unposed Images to Semantic 3Dcited by 73
  4. 2024LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenescited by 36
  5. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  6. 2024DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Featurescited by 33
  7. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  8. 2024textitRevelio: Interpreting and leveraging semantic information in diffusion modelscited by 29
  9. 2024Feat2GS: Probing Visual Foundation Models with Gaussian Splattingcited by 26
  10. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  11. 2025GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flowcited by 24
  12. 2025LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Modelscited by 23
35 more
  1. 2023GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understandingcited by 20
  2. 2024Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3Dcited by 14
  3. 2024iSeg: Interactive 3D Segmentation via Interactive Attentioncited by 14
  4. 2024Customizing Text-to-Image Diffusion with Object Viewpoint Controlcited by 13
  5. 2025SemanticSplat: Feed-Forward 3D Scene Understanding with Language-Aware Gaussian Fieldscited by 13
  6. 2024Lightning NeRF: Efficient Hybrid Scene Representation for Autonomous Drivingcited by 12
  7. 2024Neural Radiance Field-based Visual Rendering: A Comprehensive Reviewcited by 12
  8. 2025LongSplat: Online Generalizable 3D Gaussian Splatting from Long Sequence Imagescited by 12
  9. 2024GaussianStego: A Generalizable Stenography Pipeline for Generative 3D Gaussians Splattingcited by 11
  10. 2024ConDense: Consistent 2D/3D Pre-training for Dense and Sparse Features from Multi-View Imagescited by 9
  11. 2025DWTNeRF: Boosting Few-shot Neural Radiance Fields via Discrete Wavelet Transformcited by 6
  12. 2025Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Modelscited by 6
  13. 2025DINO-SLAM: DINO-informed RGB-D SLAM for Neural Implicit and Explicit Representationscited by 6
  14. 2023Learning Any-View 6DoF Robotic Grasping in Cluttered Scenes via Neural Surface Renderingcited by 5
  15. 2024Query3D: LLM-Powered Open-Vocabulary Scene Segmentation with Language Embedded 3D Gaussiancited by 5
  16. 2024Data-Efficient Inference of Neural Fluid Fields via SciML Foundation Modelcited by 4
  17. 2025Semantic Consistent Language Gaussian Splatting for Point-Level Open-vocabulary Queryingcited by 4
  18. 2024Generalizable Neural Human Renderercited by 3
  19. 2025Symmetry Understanding of 3D Shapes via Chirality Disentanglementcited by 3
  20. 2024GHNeRF: Learning Generalizable Human Features with Efficient Neural Radiance Fieldscited by 2
  21. 2025Latent Guidance in Diffusion Models for Perceptual Evaluationscited by 2
  22. 2025PCR-GS: COLMAP-Free 3D Gaussian Splatting via Pose Co-Regularizationscited by 2
  23. 2025Point2Act: Efficient 3D Distillation of Multimodal LLMs for Zero-Shot Context-Aware Graspingcited by 2
  24. 2025Multi-View Foundation Modelscited by 2
  25. 2026Symmetry Informative and Agnostic Feature Disentanglement for 3D Shapescited by 2
  26. 2026FF3R: Feedforward Feature 3D Reconstruction from Unconstrained viewscited by 2
  27. 2024NeFF-BioNet: Crop Biomass Prediction from Point Cloud to Drone Imagerycited by 1
  28. 2025Structurally Disentangled Feature Fields Distillation for 3D Understanding and Editingcited by 1
  29. 2025Next Best View Selections for Semantic and Dynamic 3D Gaussian Splattingcited by 1
  30. 2024Exploring 3D-aware Latent Spaces for Efficiently Learning Numerous Scenescited by 0
  31. 2024HFNeRF: Learning Human Biomechanic Features with Neural Radiance Fieldscited by 0
  32. 2024M2DNeRF: Multi-Modal Decomposition NeRF with 3D Feature Fieldscited by 0
  33. 2024Fused-Planes: Why Train a Thousand Tri-Planes When You Can Share?cited by 0
  34. 2025UniFField: A Generalizable Unified Neural Feature Field for Visual, Semantic, and Spatial Uncertainties in Any Scenecited by 0
  35. 2026Planetary Feature Fields are Scalable Earth Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20243D Feature Distillation with Object-Centric Priors
  2. 2024ID-NeRF: Indirect Diffusion-guided Neural Radiance Fields for Generalizable View Synthesis
  3. 2023Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts
  4. 2025Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Training
  5. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion
  6. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors
  7. 2024GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields
  8. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views