# Virtually increasing the measurement frequency of LIDAR sensor utilizing a single RGB camera

arXiv: https://arxiv.org/abs/2302.05192 · HTML: https://mlsanity.com/paper/2302.05192/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 10 Feb 2023.

## Builds on (most cited first, 21 of 21)

- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/1906.06310/) (2019)
- [PENet: Towards Precise and Efficient Image Guided Depth Completion](https://mlsanity.com/paper/2103.00783/) (2021)
- [HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching](https://mlsanity.com/paper/2007.12140/) (2020)
- [CubeSLAM: Monocular 3D Object SLAM](https://mlsanity.com/paper/1806.00557/) (2018)
- [ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2012.05258/) (2020)
- [Adaptive Context-Aware Multi-Modal Network for Depth Completion](https://mlsanity.com/paper/2008.10833/) (2020)
- [Learning 3D Shape Completion under Weak Supervision](https://mlsanity.com/paper/1805.07290/) (2018)
- [MoNet: Motion-based Point Cloud Prediction Network](https://mlsanity.com/paper/2011.10812/) (2020)
- [PointINet: Point Cloud Frame Interpolation Network](https://mlsanity.com/paper/2012.10066/) (2020)
- [Semantically Guided Depth Upsampling](https://mlsanity.com/paper/1608.00753/) (2016)
- [Learning Scene Dynamics from Point Cloud Sequences](https://mlsanity.com/paper/2111.08755/) (2021)
- [Pseudo-LiDAR Point Cloud Interpolation Based on 3D Motion Representation and Spatial Supervision](https://mlsanity.com/paper/2006.11481/) (2020)
- [High-resolution LIDAR-based Depth Mapping using Bilateral Filter](https://mlsanity.com/paper/1606.05614/) (2016)
- [PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation](https://mlsanity.com/paper/1909.07137/) (2019)

## Built on by (most cited first, 1 of 1)

- [A Method for the Runtime Validation of AI-based Environment Perception in Automated Driving System](https://mlsanity.com/paper/2412.16762/) (2024)

## Abstract (arXiv)

The frame rates of most 3D LIDAR sensors used in intelligent vehicles are substantially lower than current cameras installed in the same vehicle. This research suggests using a mono camera to virtually enhance the frame rate of LIDARs, allowing the more frequent monitoring of dynamic objects in the surroundings that move quickly. As a first step, dynamic object candidates are identified and tracked in the camera frames. Following that, the LIDAR measurement points of these items are found by clustering in the frustums of 2D bounding boxes. Projecting these to the camera and tracking them to the next camera frame can be used to create 3D-2D correspondences between different timesteps. These correspondences between the last LIDAR frame and the actual camera frame are used to solve the PnP (Perspective-n-Point) problem. Finally, the estimated transformations are applied to the previously measured points to generate virtual measurements. With the proposed estimation, if the ego movement is known, not just static object position can be determined at timesteps where camera measurement is available, but positions of dynamic objects as well. We achieve state-of-the-art performance on large public datasets in terms of accuracy and similarity to real measurements.