mlsanityData to 7 Oct 2026

CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory

arXiv 2210.05663 ↗First posted 11 Oct 2022cs.RO cs.CVcited by 164

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization. CLIP-Fields learns a mapping from spatial locations to semantic embedding vectors. Importantly, we show that this mapping can be trained with supervision coming only from web-image and web-text trained models such as CLIP, Detic, and Sentence-BERT; and thus uses no direct human supervision. When compared to baselines like Mask-RCNN, our method outperforms on few-shot instance identification or semantic segmentation on the HM3D dataset with only a fraction of the examples. Finally, we show that using CLIP-Fields as a scene memory, robots can perform semantic navigation in real-world environments. Our code and demonstration videos are available here: https://mahis.life/clip-fields

Key facts. Cited by 164 papers in the map (data through 7 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

32 of 51 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  4. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  6. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  7. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  8. 2021Neural Fields in Visual Computing and Beyondcited by 539
  9. 2021iMAP: Implicit Mapping and Positioning in Real-Timecited by 476
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  11. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  12. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
20 more
  1. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  2. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  3. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  4. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  5. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  6. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  7. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  8. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  9. 2022iSDF: Real-Time Neural Signed Distance Fields for Robot Perceptioncited by 107
  10. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  11. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  12. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  13. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  14. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  15. 2021SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistencycited by 47
  16. 2021Language Grounding with 3D Objectscited by 43
  17. 2022Continuous Scene Representations for Embodied AIcited by 39
  18. 2022Episodic Memory Question Answeringcited by 32
  19. 2021ILabel: Interactive Neural Scene Labellingcited by 28
  20. 2022Neural Motion Fields: Encoding Grasp Trajectories as Implicit Value Functionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 45; 2024: 52; 2025: 40; 2026: 25.

164 papers.

Sort:

By area:

  1. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  2. 2023LERF: Language Embedded Radiance Fieldscited by 538
  3. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  4. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 361
  5. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  6. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  7. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  8. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  9. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  10. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  11. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  12. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
38 more
  1. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  2. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  3. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  4. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  5. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  6. 2023On Bringing Robots Homecited by 101
  7. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  8. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  9. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  10. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  11. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  12. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  13. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  14. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  15. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  16. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  17. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representationcited by 54
  18. 20223D-Aware Object Goal Navigation via Simultaneous Exploration and Identificationcited by 46
  19. 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 46
  20. 2024Physical Property Understanding from Language-Embedded Feature Fieldscited by 45
  21. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  22. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  23. 2023Neural Refinement for Absolute Pose Regression with Feature Synthesiscited by 37
  24. 2023Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Drivingcited by 37
  25. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  26. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  27. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  28. 2023Zero-Shot 3D Shape Correspondencecited by 33
  29. 2023Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphscited by 33
  30. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 33
  31. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  32. 2023UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenescited by 31
  33. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  34. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  35. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  36. 2023Active Neural Mappingcited by 30
  37. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  38. 2024ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matchingcited by 25

50 most cited of 164 All 164 (JSON)