# LatentBKI: Open-Dictionary Continuous Mapping in Visual-Language Latent Spaces with Quantifiable Uncertainty

arXiv: https://arxiv.org/abs/2410.11783 · HTML: https://mlsanity.com/paper/2410.11783/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation](https://mlsanity.com/paper/2203.10421/) (2022)
- [Open-vocabulary Queryable Scene Representations for Real World Planning](https://mlsanity.com/paper/2209.09874/) (2022)
- [2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds](https://mlsanity.com/paper/2207.04397/) (2022)
- [Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow](https://mlsanity.com/paper/1802.07095/) (2018)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [Learning Navigation Behaviors End-to-End with AutoRL](https://mlsanity.com/paper/1809.10124/) (2018)
- [A Survey on Active Simultaneous Localization and Mapping: State of the Art and New Frontiers](https://mlsanity.com/paper/2207.00254/) (2022)
- [Learning to Map for Active Semantic Goal Navigation](https://mlsanity.com/paper/2106.15648/) (2021)
- [MotionSC: Data Set and Network for Real-Time Semantic Mapping in Dynamic Environments](https://mlsanity.com/paper/2203.07060/) (2022)
- [Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping](https://mlsanity.com/paper/1909.04631/) (2019)
- [Uncertainty-driven Planner for Exploration and Navigation](https://mlsanity.com/paper/2202.11907/) (2022)
- [Meaningful Maps With Object-Oriented Semantic Mapping](https://mlsanity.com/paper/1609.07849/) (2016)
- [A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing](https://mlsanity.com/paper/2207.10614/) (2022)
- [These Maps Are Made For Walking: Real-Time Terrain Property Estimation for Mobile Robots](https://mlsanity.com/paper/2205.12925/) (2022)
- [ConvBKI: Real-Time Probabilistic Semantic Mapping Network with Quantifiable Uncertainty](https://mlsanity.com/paper/2310.16020/) (2023)

## Built on by (most cited first, 2 of 2)

- [SLIM-VDB: A Real-Time 3D Probabilistic Semantic Mapping Framework](https://mlsanity.com/paper/2512.12945/) (2025)
- [LatentAM: Real-Time, Large-Scale Latent Gaussian Attention Mapping via Online Dictionary Learning](https://mlsanity.com/paper/2602.12314/) (2026)

## Abstract (arXiv)

This paper introduces a novel probabilistic mapping algorithm, LatentBKI, which enables open-vocabulary mapping with quantifiable uncertainty. Traditionally, semantic mapping algorithms focus on a fixed set of semantic categories which limits their applicability for complex robotic tasks. Vision-Language (VL) models have recently emerged as a technique to jointly model language and visual features in a latent space, enabling semantic recognition beyond a predefined, fixed set of semantic classes. LatentBKI recurrently incorporates neural embeddings from VL models into a voxel map with quantifiable uncertainty, leveraging the spatial correlations of nearby observations through Bayesian Kernel Inference (BKI). LatentBKI is evaluated against similar explicit semantic mapping and VL mapping frameworks on the popular Matterport3D and Semantic KITTI datasets, demonstrating that LatentBKI maintains the probabilistic benefits of continuous mapping with the additional benefit of open-dictionary queries. Real-world experiments demonstrate applicability to challenging indoor environments.