# Real-Time Metric-Semantic Mapping for Autonomous Navigation in Outdoor Environments

arXiv: https://arxiv.org/abs/2412.00291 · HTML: https://mlsanity.com/paper/2412.00291/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 30 Nov 2024.

## Builds on (most cited first, 30 of 33)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planning](https://mlsanity.com/paper/1611.03631/) (2016)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation](https://mlsanity.com/paper/2008.01550/) (2020)
- [MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects](https://mlsanity.com/paper/1804.09194/) (2018)
- [Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery](https://mlsanity.com/paper/1903.00268/) (2019)
- [SNI-SLAM: Semantic Neural Implicit SLAM](https://mlsanity.com/paper/2311.11016/) (2023)
- [Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Data](https://mlsanity.com/paper/2105.08971/) (2021)
- [Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems](https://mlsanity.com/paper/2106.14386/) (2021)
- [R3LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package](https://mlsanity.com/paper/2109.07982/) (2021)
- [Elevation Mapping for Locomotion and Navigation using GPU](https://mlsanity.com/paper/2204.12876/) (2022)
- [A Benchmark for LiDAR-based Panoptic Segmentation based on KITTI](https://mlsanity.com/paper/2003.02371/) (2020)
- [Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency](https://mlsanity.com/paper/2109.10165/) (2021)
- [Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping](https://mlsanity.com/paper/1909.04631/) (2019)
- [nvblox: GPU-Accelerated Incremental Signed Distance Field Mapping](https://mlsanity.com/paper/2311.00626/) (2023)
- [TNS: Terrain Traversability Mapping and Navigation System for Autonomous Excavators](https://mlsanity.com/paper/2109.06250/) (2021)
- [FusionPortable: A Multi-Sensor Campus-Scene Dataset for Evaluation of Localization and Mapping Accuracy on Diverse Platforms](https://mlsanity.com/paper/2208.11865/) (2022)
- [Robust Odometry and Mapping for Multi-LiDAR Systems with Online Extrinsic Calibration](https://mlsanity.com/paper/2010.14294/) (2020)
- [LCE-Calib: Automatic LiDAR-Frame/Event Camera Extrinsic Calibration With A Globally Optimal Solution](https://mlsanity.com/paper/2303.09825/) (2023)
- [FAR Planner: Fast, Attemptable Route Planner using Dynamic Visibility Update](https://mlsanity.com/paper/2110.09460/) (2021)
- [ImMesh: An Immediate LiDAR Localization and Meshing Framework](https://mlsanity.com/paper/2301.05206/) (2023)
- [AutoMerge: A Framework for Map Assembling and Smoothing in City-scale Environments](https://mlsanity.com/paper/2207.06965/) (2022)
- [Multitask Learning for Scalable and Dense Multilayer Bayesian Map Inference](https://mlsanity.com/paper/2106.14986/) (2021)

## Built on by (most cited first, 2 of 2)

- [ST-Booster: An Iterative SpatioTemporal Perception Booster for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2504.09843/) (2025)
- [Single-Frame Point-Pixel Registration via Supervised Cross-Modal Feature Matching](https://mlsanity.com/paper/2506.22784/) (2025)

## Abstract (arXiv)

The creation of a metric-semantic map, which encodes human-prior knowledge, represents a high-level abstraction of environments. However, constructing such a map poses challenges related to the fusion of multi-modal sensor data, the attainment of real-time mapping performance, and the preservation of structural and semantic information consistency. In this paper, we introduce an online metric-semantic mapping system that utilizes LiDAR-Visual-Inertial sensing to generate a global metric-semantic mesh map of large-scale outdoor environments. Leveraging GPU acceleration, our mapping process achieves exceptional speed, with frame processing taking less than 7ms, regardless of scenario scale. Furthermore, we seamlessly integrate the resultant map into a real-world navigation system, enabling metric-semantic-based terrain assessment and autonomous point-to-point navigation within a campus environment. Through extensive experiments conducted on both publicly available and self-collected datasets comprising 24 sequences, we demonstrate the effectiveness of our mapping and navigation methodologies. Code has been publicly released: https://github.com/gogojjh/cobra