mlsanityData to 7 Oct 2026

Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations

arXiv 2209.03494 ↗First posted 7 Sep 2022cs.CV cs.GRcited by 195

Not on the map. Why?

Abstract (arXiv)

We present Neural Feature Fusion Fields (N3F), a method that improves dense 2D image feature extractors when the latter are applied to the analysis of multiple images reconstructible as a 3D scene. Given an image feature extractor, for example pre-trained using self-supervision, N3F uses it as a teacher to learn a student network defined in 3D space. The 3D student network is similar to a neural radiance field that distills said features and can be trained with the usual differentiable rendering machinery. As a consequence, N3F is readily applicable to most neural rendering formulations, including vanilla NeRF and its extensions to complex dynamic scenes. We show that our method not only enables semantic understanding in the context of scene-specific neural fields without the use of manual labels, but also consistently improves over the self-supervised 2D baselines. This is demonstrated by considering various tasks, such as 2D object retrieval, 3D segmentation, and scene editing, in diverse sequences, including long egocentric videos in the EPIC-KITCHENS benchmark.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). Builds on 70 of its 83 references that are in the map. First posted on arXiv 7 Sep 2022.

Builds on

70 of 83 references.

References · 70

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  12. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
38 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  4. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  5. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  6. 2017Deep Image Priorcited by 1,097
  7. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  9. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  10. 2019Relational Knowledge Distillationcited by 873
  11. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  12. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  13. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  14. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  15. 2021Going deeper with Image Transformerscited by 759
  16. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  17. 2019Contrastive Representation Distillationcited by 734
  18. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  19. 2021Neural Fields in Visual Computing and Beyondcited by 539
  20. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  21. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 470
  22. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  23. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  24. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  25. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  26. 2021Advances in Neural Renderingcited by 365
  27. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  28. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  29. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  30. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  31. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  32. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  33. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  34. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  35. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  36. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  37. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  38. 2019The VIA Annotation Software for Images, Audio and Videocited by 166

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 59; 2024: 70; 2025: 44; 2026: 11.

195 papers.

Sort:

By area:

  1. 2023LERF: Language Embedded Radiance Fieldscited by 538
  2. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  3. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  4. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
  5. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 361
  6. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  7. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  8. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 171
  9. 2023GaussianEditor: Editing 3D Gaussians Delicately with Text Instructionscited by 166
  10. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  11. 2023Segment Any 3D Gaussianscited by 158
  12. 2022NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fieldscited by 151
38 more
  1. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  2. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  3. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  4. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  5. 2022MagicPony: Learning Articulated 3D Animals in the Wildcited by 111
  6. 2024GARField: Group Anything with Radiance Fieldscited by 108
  7. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  8. 2023SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Fieldcited by 93
  9. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 92
  10. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  11. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  12. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 81
  13. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
  14. 2023FocalDreamer: Text-driven 3D Editing via Focal-fusion Assemblycited by 75
  15. 2022NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenescited by 71
  16. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 70
  17. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 68
  18. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  19. 2023Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusioncited by 62
  20. 2023Segment Anything in 3D with Radiance Fieldscited by 61
  21. 2022Interactive Segmentation of Radiance Fieldscited by 60
  22. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 60
  23. 2024Click-Gaussian: Interactive Segmentation to Any 3D Gaussianscited by 58
  24. 2024FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimallycited by 57
  25. 2024GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplanecited by 53
  26. 2023EPIC Fields: Marrying 3D Geometry and Video Understandingcited by 49
  27. 2023FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Modelscited by 47
  28. 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 46
  29. 2024SplatLoc: 3D Gaussian Splatting-based Visual Localization for Augmented Realitycited by 42
  30. 2025LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPScited by 40
  31. 2023ObjectSDF++: Improved Object-Compositional Neural Implicit Surfacescited by 38
  32. 2022Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understandingcited by 37
  33. 2023Neural Refinement for Absolute Pose Regression with Feature Synthesiscited by 37
  34. 2023Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Drivingcited by 37
  35. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  36. 2023SHAP-EDITOR: Instruction-guided Latent 3D Editing in Secondscited by 36
  37. 2024LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenescited by 36
  38. 20254D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Modelscited by 35

50 most cited of 195 All 195 (JSON)