# R3LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package

arXiv: https://arxiv.org/abs/2109.07982 · HTML: https://mlsanity.com/paper/2109.07982/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021)
- [FAST-LIO: A Fast, Robust LiDAR-inertial Odometry Package by Tightly-Coupled Iterated Kalman Filter](https://mlsanity.com/paper/2010.08196/) (2020)
- [Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments](https://mlsanity.com/paper/2103.01627/) (2021)
- [LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2104.10831/) (2021)
- [Loam_livox: A fast, robust, high-precision LiDAR odometry and mapping package for LiDARs of small FoV](https://mlsanity.com/paper/1909.06700/) (2019)
- [BALM: Bundle Adjustment for Lidar Mapping](https://mlsanity.com/paper/2010.08215/) (2020)
- [Online Temporal Calibration for Monocular Visual-Inertial Systems](https://mlsanity.com/paper/1808.00692/) (2018)
- [LIC-Fusion: LiDAR-Inertial-Camera Odometry](https://mlsanity.com/paper/1909.04102/) (2019)
- [LIC-Fusion 2.0: LiDAR-Inertial-Camera Odometry with Sliding-Window Plane-Feature Tracking](https://mlsanity.com/paper/2008.07196/) (2020)
- [R2LIVE: A Robust, Real-time, LiDAR-Inertial-Visual tightly-coupled state Estimator and mapping](https://mlsanity.com/paper/2102.12400/) (2021)
- [Stereo Visual Inertial LiDAR Simultaneous Localization and Mapping](https://mlsanity.com/paper/1902.10741/) (2019)
- [Low-cost Retina-like Robotic Lidars Based on Incommensurable Scanning](https://mlsanity.com/paper/2006.11034/) (2020)
- [A decentralized framework for simultaneous calibration, localization and mapping with multiple LiDARs](https://mlsanity.com/paper/2007.01483/) (2020)
- [Avoiding dynamic small obstacles with onboard sensing and computating on aerial robots](https://mlsanity.com/paper/2103.00406/) (2021)

## Built on by (most cited first, 30 of 50)

- [FAST-LIVO2: Fast, Direct LiDAR-Inertial-Visual Odometry](https://mlsanity.com/paper/2408.14035/) (2024)
- [RGB-Event Fusion for Moving Object Detection in Autonomous Driving](https://mlsanity.com/paper/2209.08323/) (2022)
- [Spectrally Pruned Gaussian Fields with Neural Compensation](https://mlsanity.com/paper/2405.00676/) (2024)
- [UAVScenes: A Multi-Modal Dataset for UAVs](https://mlsanity.com/paper/2507.22412/) (2025)
- [LetsGo: Large-Scale Garage Modeling and Rendering via LiDAR-Assisted Gaussian Primitives](https://mlsanity.com/paper/2404.09748/) (2024)
- [STD: Stable Triangle Descriptor for 3D place recognition](https://mlsanity.com/paper/2209.12435/) (2022)
- [Microsaccade-inspired Event Camera for Robotics](https://mlsanity.com/paper/2405.17769/) (2024)
- [High-Definition Map Generation Technologies For Autonomous Driving](https://mlsanity.com/paper/2206.05400/) (2022)
- [SiLVR: Scalable Lidar-Visual Reconstruction with Neural Radiance Fields for Robotic Inspection](https://mlsanity.com/paper/2403.06877/) (2024)
- [LCE-Calib: Automatic LiDAR-Frame/Event Camera Extrinsic Calibration With A Globally Optimal Solution](https://mlsanity.com/paper/2303.09825/) (2023)
- [LF-VIO: A Visual-Inertial-Odometry Framework for Large Field-of-View Cameras with Negative Plane](https://mlsanity.com/paper/2202.12613/) (2022)
- [LF-VISLAM: A SLAM Framework for Large Field-of-View Cameras with Negative Imaging Plane on Mobile Agents](https://mlsanity.com/paper/2209.05167/) (2022)
- [ImMesh: An Immediate LiDAR Localization and Meshing Framework](https://mlsanity.com/paper/2301.05206/) (2023)
- [CalibFormer: A Transformer-based Automatic LiDAR-Camera Calibration Network](https://mlsanity.com/paper/2311.15241/) (2023)
- [GS-LIVO: Real-Time LiDAR, Inertial, and Visual Multi-sensor Fused Odometry with Gaussian Mapping](https://mlsanity.com/paper/2501.08672/) (2025)
- [Register Any Point: Scaling 3D Point Cloud Registration by Flow Matching](https://mlsanity.com/paper/2512.01850/) (2025)
- [GS-SDF: LiDAR-Augmented Gaussian Splatting and Neural SDF for Geometrically Consistent Rendering and Reconstruction](https://mlsanity.com/paper/2503.10170/) (2025)
- [SR-LIVO: LiDAR-Inertial-Visual Odometry and Mapping with Sweep Reconstruction](https://mlsanity.com/paper/2312.16800/) (2023)
- [Autonomous Driving in Unstructured Environments: How Far Have We Come?](https://mlsanity.com/paper/2410.07701/) (2024)
- [MM-Gaussian: 3D Gaussian-based Multi-modal Fusion for Localization and Reconstruction in Unbounded Scenes](https://mlsanity.com/paper/2404.04026/) (2024)
- [LiDAR-3DGS: LiDAR Reinforced 3D Gaussian Splatting for Multimodal Radiance Field Rendering](https://mlsanity.com/paper/2409.16296/) (2024)
- [OmniColor: A Global Camera Pose Optimization Approach of LiDAR-360Camera Fusion for Colorizing Point Clouds](https://mlsanity.com/paper/2404.04693/) (2024)
- [Mode-GS: Monocular Depth Guided Anchored 3D Gaussian Splatting for Robust Ground-View Scene Rendering](https://mlsanity.com/paper/2410.04646/) (2024)
- [LiDAR-VGGT: Cross-Modal Coarse-to-Fine Fusion for Globally Consistent and Metric-Scale Dense Mapping](https://mlsanity.com/paper/2511.01186/) (2025)
- [SALT: A Flexible Semi-Automatic Labeling Tool for General LiDAR Point Clouds with Cross-Scene Adaptability and 4D Consistency](https://mlsanity.com/paper/2503.23980/) (2025)
- [Astra: Toward General-Purpose Mobile Robots via Hierarchical Multimodal Learning](https://mlsanity.com/paper/2506.06205/) (2025)
- [R3LIVE++: A Robust, Real-time, Radiance reconstruction package with a tightly-coupled LiDAR-Inertial-Visual state Estimator](https://mlsanity.com/paper/2209.03666/) (2022)
- [Neural Surface Reconstruction and Rendering for LiDAR-Visual Systems](https://mlsanity.com/paper/2409.05310/) (2024)
- [Environment-Driven Online LiDAR-Camera Extrinsic Calibration](https://mlsanity.com/paper/2502.00801/) (2025)
- [SiLVR: Scalable Lidar-Visual Radiance Field Reconstruction with Uncertainty Quantification](https://mlsanity.com/paper/2502.02657/) (2025)

## Abstract (arXiv)

In this letter, we propose a novel LiDAR-Inertial-Visual sensor fusion framework termed R3LIVE, which takes advantage of measurement of LiDAR, inertial, and visual sensors to achieve robust and accurate state estimation. R3LIVE is contained of two subsystems, the LiDAR-inertial odometry (LIO) and visual-inertial odometry (VIO). The LIO subsystem (FAST-LIO) takes advantage of the measurement from LiDAR and inertial sensors and builds the geometry structure of (i.e. the position of 3D points) global maps. The VIO subsystem utilizes the data of visual-inertial sensors and renders the map's texture (i.e. the color of 3D points). More specifically, the VIO subsystem fuses the visual data directly and effectively by minimizing the frame-to-map photometric error. The developed system R3LIVE is developed based on our previous work R2LIVE, with careful architecture design and implementation. Experiment results show that the resultant system achieves more robustness and higher accuracy in state estimation than current counterparts (see our attached video). R3LIVE is a versatile and well-engineered system toward various possible applications, which can not only serve as a SLAM system for real-time robotic applications, but can also reconstruct the dense, precise, RGB-colored 3D maps for applications like surveying and mapping. Moreover, to make R3LIVE more extensible, we develop a series of offline utilities for reconstructing and texturing meshes, which further minimizes the gap between R3LIVE and various of 3D applications such as simulators, video games and etc (see our demos video). To share our findings and make contributions to the community, we open source R3LIVE on our Github, including all of our codes, software utilities, and the mechanical design of our device.