mlsanityData to 14 Oct 2026

ConceptFusion: Open-set Multimodal 3D Mapping

arXiv 2302.07241 ↗First posted 14 Feb 2023cs.CV cs.AI cs.ROcited by 304

Not on the map. Why?

Cited by 38 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building 3D maps of the environment is central to robot navigation, planning, and interaction with objects in a scene. Most existing approaches that integrate semantic concepts with 3D maps largely remain confined to the closed-set setting: they can only reason about a finite set of concepts, pre-defined at training time. Further, these maps can only be queried using class labels, or in recent work, using text prompts. We address both these issues with ConceptFusion, a scene representation that is (1) fundamentally open-set, enabling reasoning beyond a closed set of concepts and (ii) inherently multimodal, enabling a diverse range of possible queries to the 3D map, from language, to images, to audio, to 3D geometry, all working in concert. ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio. We demonstrate that pixel-aligned open-set features can be fused into 3D maps via traditional SLAM and multi-view fusion approaches. This enables effective zero-shot spatial reasoning, not needing any additional training or finetuning, and retains long-tailed concepts better than supervised approaches, outperforming them by more than 40% margin on 3D IoU. We extensively evaluate ConceptFusion on a number of real-world datasets, simulated home environments, a real-world tabletop manipulation task, and an autonomous driving platform. We showcase new avenues for blending foundation models with 3D open-set multimodal mapping. For more information, visit our project page https://concept-fusion.github.io or watch our 5-minute explainer video https://www.youtube.com/watch?v=rkXgws8fiDs

Key facts. Cited by 304 papers in the map (data through 14 Oct 2026). Builds on 58 of its 95 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

58 of 95 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  11. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  3. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  4. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  5. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  6. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  8. 2022Language-driven Semantic Segmentationcited by 717
  9. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  10. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  11. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  12. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  13. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  14. 2021Image Segmentation Using Text and Image Promptscited by 512
  15. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  16. 2020Artificial Intelligence, Values and Alignmentcited by 448
  17. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  18. 2022Visual Language Maps for Robot Navigationcited by 365
  19. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  20. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  21. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  22. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  23. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  24. 2020Experience Grounds Languagecited by 242
  25. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  26. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  27. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  28. 2020D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometrycited by 206
  29. 2021Decoupling Zero-Shot Semantic Segmentationcited by 199
  30. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  31. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  32. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  33. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  34. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 144
  35. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 135
  36. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 120
  37. 2016BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integrationcited by 116
  38. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 102

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 57; 2024: 94; 2025: 87; 2026: 66.

304 papers.

Sort:

By area:

  1. 2023LERF: Language Embedded Radiance Fieldscited by 543
  2. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  3. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  4. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  5. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  6. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  7. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  8. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  9. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  10. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  11. 2023AnyLoc: Towards Universal Visual Place Recognitioncited by 163
  12. 2023Segment Any 3D Gaussianscited by 161
38 more
  1. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  2. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  3. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  4. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  5. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 106
  6. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  7. 2023On Bringing Robots Homecited by 101
  8. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  9. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  10. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 94
  11. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  12. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  13. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 89
  14. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 89
  15. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  16. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  17. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 82
  18. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  19. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 80
  20. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 80
  21. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  22. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  23. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 75
  24. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  25. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 71
  26. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 69
  27. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
  28. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 65
  29. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 63
  30. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  31. 2023Segment Anything in 3D with Radiance Fieldscited by 62
  32. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  33. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  34. 2024Part123: Part-aware 3D Reconstruction from a Single-view Imagecited by 55
  35. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representationcited by 54
  36. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 54
  37. 2024POP-3D: Open-Vocabulary 3D Occupancy Prediction from Imagescited by 52
  38. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51

50 most cited of 304 All 304 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Open-Set 3D Scene Graphs for Field Robotics: An Outdoor Case Study ↗
  2. 2026A Scene Graph Backed Approach to Open Set Semantic Mapping
  3. 2025SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning
  4. 2026Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps
  5. 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
  6. 2026From Pixels to Concepts: Growing Rich 3D Semantic Scene Graph Forests utilizing Foundation Models ↗
  7. 2026Vision-Language-Motion Maps: An Open-Vocabulary, Uncertainty-Aware, Queryable Motion Attribute for 3D Scene Maps ↗
  8. 2026Relationship-Aware Hierarchical 3D Scene Graph for Task Reasoning