# R3LIVE++: A Robust, Real-time, Radiance reconstruction package with a tightly-coupled LiDAR-Inertial-Visual state Estimator

arXiv: https://arxiv.org/abs/2209.03666 · HTML: https://mlsanity.com/paper/2209.03666/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 75 references that are in the map.
- First posted on arXiv 8 Sep 2022.

## Builds on (most cited first, 30 of 33)

- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2007.00258/) (2020)
- [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021)
- [FAST-LIO: A Fast, Robust LiDAR-inertial Odometry Package by Tightly-Coupled Iterated Kalman Filter](https://mlsanity.com/paper/2010.08196/) (2020)
- [R3LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package](https://mlsanity.com/paper/2109.07982/) (2021)
- [Stereo DSO: Large-Scale Direct Sparse Visual Odometry with Stereo Cameras](https://mlsanity.com/paper/1708.07878/) (2017)
- [LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2104.10831/) (2021)
- [AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithms](https://mlsanity.com/paper/1901.07849/) (2019)
- [Tightly Coupled 3D Lidar Inertial Odometry and Mapping](https://mlsanity.com/paper/1904.06993/) (2019)
- [A Photometrically Calibrated Benchmark For Monocular Visual Odometry](https://mlsanity.com/paper/1607.02555/) (2016)
- [Loam_livox: A fast, robust, high-precision LiDAR odometry and mapping package for LiDARs of small FoV](https://mlsanity.com/paper/1909.06700/) (2019)
- [Online Temporal Calibration for Monocular Visual-Inertial Systems](https://mlsanity.com/paper/1808.00692/) (2018)
- [VILENS: Visual, Inertial, Lidar, and Leg Odometry for All-Terrain Legged Robots](https://mlsanity.com/paper/2107.07243/) (2021)
- [FAST-LIVO: Fast and Tightly-coupled Sparse-Direct LiDAR-Inertial-Visual Odometry](https://mlsanity.com/paper/2203.00893/) (2022)
- [LIC-Fusion: LiDAR-Inertial-Camera Odometry](https://mlsanity.com/paper/1909.04102/) (2019)
- [LINS: A Lidar-Inertial State Estimator for Robust and Efficient Navigation](https://mlsanity.com/paper/1907.02233/) (2019)
- [CamVox: A Low-cost and Accurate Lidar-assisted Visual SLAM System](https://mlsanity.com/paper/2011.11357/) (2020)
- [High-Definition Map Generation Technologies For Autonomous Driving](https://mlsanity.com/paper/2206.05400/) (2022)
- [LIC-Fusion 2.0: LiDAR-Inertial-Camera Odometry with Sliding-Window Plane-Feature Tracking](https://mlsanity.com/paper/2008.07196/) (2020)
- [R2LIVE: A Robust, Real-time, LiDAR-Inertial-Visual tightly-coupled state Estimator and mapping](https://mlsanity.com/paper/2102.12400/) (2021)
- [Towards High-Performance Solid-State-LiDAR-Inertial Odometry and Mapping](https://mlsanity.com/paper/2010.13150/) (2020)
- [Online Photometric Calibration for Auto Exposure Video for Realtime Visual Odometry and SLAM](https://mlsanity.com/paper/1710.02081/) (2017)
- [A fast, complete, point cloud based loop closure for LiDAR odometry and mapping](https://mlsanity.com/paper/1909.11811/) (2019)
- [Stereo Visual Inertial LiDAR Simultaneous Localization and Mapping](https://mlsanity.com/paper/1902.10741/) (2019)

## Built on by (most cited first, 3 of 3)

- [STD: Stable Triangle Descriptor for 3D place recognition](https://mlsanity.com/paper/2209.12435/) (2022)
- [ImMesh: An Immediate LiDAR Localization and Meshing Framework](https://mlsanity.com/paper/2301.05206/) (2023)
- [OmniColor: A Global Camera Pose Optimization Approach of LiDAR-360Camera Fusion for Colorizing Point Clouds](https://mlsanity.com/paper/2404.04693/) (2024)

## Abstract (arXiv)

Simultaneous localization and mapping (SLAM) are crucial for autonomous robots (e.g., self-driving cars, autonomous drones), 3D mapping systems, and AR/VR applications. This work proposed a novel LiDAR-inertial-visual fusion framework termed R3LIVE++ to achieve robust and accurate state estimation while simultaneously reconstructing the radiance map on the fly. R3LIVE++ consists of a LiDAR-inertial odometry (LIO) and a visual-inertial odometry (VIO), both running in real-time. The LIO subsystem utilizes the measurements from a LiDAR for reconstructing the geometric structure (i.e., the positions of 3D points), while the VIO subsystem simultaneously recovers the radiance information of the geometric structure from the input images. R3LIVE++ is developed based on R3LIVE and further improves the accuracy in localization and mapping by accounting for the camera photometric calibration (e.g., non-linear response function and lens vignetting) and the online estimation of camera exposure time. We conduct more extensive experiments on both public and our private datasets to compare our proposed system against other state-of-the-art SLAM systems. Quantitative and qualitative results show that our proposed system has significant improvements over others in both accuracy and robustness. In addition, to demonstrate the extendability of our work, we developed several applications based on our reconstructed radiance maps, such as high dynamic range (HDR) imaging, virtual environment exploration, and 3D video gaming. Lastly, to share our findings and make contributions to the community, we make our codes, hardware design, and dataset publicly available on our Github: github.com/hku-mars/r3live