# ConvBKI: Real-Time Probabilistic Semantic Mapping Network with Quantifiable Uncertainty

arXiv: https://arxiv.org/abs/2310.16020 · HTML: https://mlsanity.com/paper/2310.16020/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 73 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving](https://mlsanity.com/paper/2003.03653/) (2020)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2007.00258/) (2020)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization](https://mlsanity.com/paper/2201.13360/) (2022)
- [Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planning](https://mlsanity.com/paper/1611.03631/) (2016)
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](https://mlsanity.com/paper/2011.12954/) (2020)
- [Translating Images into Maps](https://mlsanity.com/paper/2110.00966/) (2021)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [SuMa++: Efficient LiDAR-based Semantic SLAM](https://mlsanity.com/paper/2105.11320/) (2021)
- [Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views](https://mlsanity.com/paper/2010.01191/) (2020)
- [MotionSC: Data Set and Network for Real-Time Semantic Mapping in Dynamic Environments](https://mlsanity.com/paper/2203.07060/) (2022)
- [Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency](https://mlsanity.com/paper/2109.10165/) (2021)
- [Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping](https://mlsanity.com/paper/1909.04631/) (2019)

## Built on by (most cited first, 11 of 11)

- [POp-GS: Next Best View in 3D-Gaussian Splatting with P-Optimality](https://mlsanity.com/paper/2503.07819/) (2025)
- [Modeling Uncertainty in 3D Gaussian Splatting through Continuous Semantic Splatting](https://mlsanity.com/paper/2411.02547/) (2024)
- [Evidential Semantic Mapping in Off-road Environments with Uncertainty-aware Bayesian Kernel Inference](https://mlsanity.com/paper/2403.14138/) (2024)
- [These Magic Moments: Differentiable Uncertainty Quantification of Radiance Field Models](https://mlsanity.com/paper/2503.14665/) (2025)
- [Understanding while Exploring: Semantics-driven Active Mapping](https://mlsanity.com/paper/2506.00225/) (2025)
- [Uncertainty-aware Semantic Mapping in Off-road Environments with Dempster-Shafer Theory of Evidence](https://mlsanity.com/paper/2405.06265/) (2024)
- [LatentBKI: Open-Dictionary Continuous Mapping in Visual-Language Latent Spaces with Quantifiable Uncertainty](https://mlsanity.com/paper/2410.11783/) (2024)
- [SLIM-VDB: A Real-Time 3D Probabilistic Semantic Mapping Framework](https://mlsanity.com/paper/2512.12945/) (2025)
- [Belief Consistency Between Foundation-Model Evidence and Geometric Perception in Persistent Robotic Maps](https://mlsanity.com/paper/2606.00318/) (2026)
- [VoxelFix: Post-Hoc Semantic Correction of Completed 3D Voxel Maps](https://mlsanity.com/paper/2609.05114/) (2026)
- [DIGS: Distribution-Informed Gaussian Splatting for Training-Free Open-Vocabulary 3D Segmentation](https://mlsanity.com/paper/2610.16177/) (2026)

## Abstract (arXiv)

In this paper, we develop a modular neural network for real-time colorblack(> 10 Hz) semantic mapping in uncertain environments, which explicitly updates per-voxel probabilistic distributions within a neural network layer. Our approach combines the reliability of classical probabilistic algorithms with the performance and efficiency of modern neural networks. Although robotic perception is often divided between modern differentiable methods and classical explicit methods, a union of both is necessary for real-time and trustworthy performance. We introduce a novel Convolutional Bayesian Kernel Inference (ConvBKI) layer which incorporates semantic segmentation predictions online into a 3D map through a depthwise convolution layer by leveraging conjugate priors. We compare ConvBKI against state-of-the-art deep learning approaches and probabilistic algorithms for mapping to evaluate reliability and performance. We also create a Robot Operating System (ROS) package of ConvBKI and test it on real-world perceptually challenging off-road driving data.