mlsanityData to 7 Oct 2026

LERF: Language Embedded Radiance Fields

arXiv 2303.09553 ↗First posted 16 Mar 2023cs.CV cs.GRcited by 538

Not on the map. Why?

Cited by 38 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Humans describe the physical world using natural language to refer to specific 3D locations based on a vast range of properties: visual appearance, semantics, abstract associations, or actionable affordances. In this work we propose Language Embedded Radiance Fields (LERFs), a method for grounding language embeddings from off-the-shelf models like CLIP into NeRF, which enable these types of open-ended language queries in 3D. LERF learns a dense, multi-scale language field inside NeRF by volume rendering CLIP embeddings along training rays, supervising these embeddings across training views to provide multi-view consistency and smooth the underlying language field. After optimization, LERF can extract 3D relevancy maps for a broad range of language prompts interactively in real-time, which has potential use cases in robotics, understanding vision-language models, and interacting with 3D scenes. LERF enables pixel-aligned, zero-shot queries on the distilled 3D CLIP embeddings without relying on region proposals or masks, supporting long-tail open-vocabulary queries hierarchically across the volume. The project website can be found at https://lerf.io .

Key facts. Cited by 538 papers in the map (data through 7 Oct 2026). Builds on 39 of its 43 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

39 of 43 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  7. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  8. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  11. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  12. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
27 more
  1. 2022Language-driven Semantic Segmentationcited by 711
  2. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  3. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  4. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  5. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  6. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  7. 2021Image Segmentation Using Text and Image Promptscited by 509
  8. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  9. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  10. 2022Visual Language Maps for Robot Navigationcited by 362
  11. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  12. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  13. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  14. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  15. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  16. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  17. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  18. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 171
  19. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  20. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  21. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  22. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  23. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  24. 2022NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenescited by 71
  25. 2022Neural Volumetric Object Selectioncited by 61
  26. 2021Language Grounding with 3D Objectscited by 43
  27. 2022Voxel-informed Language Groundingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 53; 2024: 177; 2025: 172; 2026: 136.

538 papers.

Sort:

By area:

  1. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  2. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  3. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
  4. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 361
  5. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  6. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  7. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  8. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  9. 2024gsplat: An Open-Source Library for Gaussian Splattingcited by 279
  10. 2023FSGS: Real-Time Few-shot View Synthesis using Gaussian Splattingcited by 278
  11. 2025AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Viewscited by 234
  12. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
38 more
  1. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  2. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  3. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  4. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  5. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  6. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  7. 2023Segment Any 3D Gaussianscited by 158
  8. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 141
  9. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  10. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  11. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  12. 2024GARField: Group Anything with Radiance Fieldscited by 108
  13. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  14. 20243D Gaussian as a New Era: A Surveycited by 102
  15. 2023On Bringing Robots Homecited by 101
  16. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  17. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  18. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  19. 2025PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyondcited by 91
  20. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  21. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  22. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  23. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  24. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 81
  25. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  26. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
  27. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  28. 2023OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learningcited by 75
  29. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  30. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  31. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 70
  32. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 68
  33. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  34. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  35. 2023Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusioncited by 62
  36. 2023Segment Anything in 3D with Radiance Fieldscited by 61
  37. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 60
  38. 2023Language Models as Zero-Shot Trajectory Generatorscited by 59

50 most cited of 538 All 538 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Diverse Language Generation for 3D Visual Grounding
  2. 2023Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model
  3. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
  4. 2026Language-Conditioned Visual Grounding with CLIP Multilingual
  5. 2025Lang3D-XL: Language Embedded 3D Gaussians for Large-scale Scenes
  6. 2024Grounded 3D-LLM with Referent Tokens
  7. 2026An overview of 3D Vision-Language Models
  8. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models