# ISETAuto: Detecting vehicles with depth and radiance information

arXiv: https://arxiv.org/abs/2101.01843 · HTML: https://mlsanity.com/paper/2101.01843/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/1906.06310/) (2019)
- [LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World](https://mlsanity.com/paper/2006.09348/) (2020)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [LIDAR-Camera Fusion for Road Detection Using Fully Convolutional Neural Networks](https://mlsanity.com/paper/1809.07941/) (2018)
- [Augmented LiDAR Simulator for Autonomous Driving](https://mlsanity.com/paper/1811.07112/) (2018)
- [Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-driving Images](https://mlsanity.com/paper/2002.10570/) (2020)
- [Gated2Depth: Real-time Dense Lidar from Gated Images](https://mlsanity.com/paper/1902.04997/) (2019)
- [Neural Network Generalization: The impact of camera parameters](https://mlsanity.com/paper/1912.03604/) (2019)
- [Exploiting Depth from Single Monocular Images for Object Detection and Semantic Segmentation](https://mlsanity.com/paper/1610.01706/) (2016)
- [Soft Prototyping Camera Designs for Car Detection Based on a Convolutional Neural Network](https://mlsanity.com/paper/1910.10916/) (2019)
- [A system for generating complex physically accurate sensor images for automotive applications](https://mlsanity.com/paper/1902.04258/) (2019)

## Built on by (most cited first, 3 of 3)

- [Neural Camera Simulators](https://mlsanity.com/paper/2104.05237/) (2021)
- [Using simulation to quantify the performance of automotive perception systems](https://mlsanity.com/paper/2303.00983/) (2023)
- [ISETHDR: A Physics-based Synthetic Radiance Dataset for High Dynamic Range Driving Scenes](https://mlsanity.com/paper/2408.12048/) (2024)

## Abstract (arXiv)

Autonomous driving applications use two types of sensor systems to identify vehicles - depth sensing LiDAR and radiance sensing cameras. We compare the performance (average precision) of a ResNet for vehicle detection in complex, daytime, driving scenes when the input is a depth map (D = d(x,y)), a radiance image (L = r(x,y)), or both [D,L]. (1) When the spatial sampling resolution of the depth map and radiance image are equal to typical camera resolutions, a ResNet detects vehicles at higher average precision from depth than radiance. (2) As the spatial sampling of the depth map declines to the range of current LiDAR devices, the ResNet average precision is higher for radiance than depth. (3) For a hybrid system that combines a depth map and radiance image, the average precision is higher than using depth or radiance alone. We established these observations in simulation and then confirmed them using realworld data. The advantage of combining depth and radiance can be explained by noting that the two type of information have complementary weaknesses. The radiance data are limited by dynamic range and motion blur. The LiDAR data have relatively low spatial resolution. The ResNet combines the two data sources effectively to improve overall vehicle detection.