# Dataset and Benchmark: Novel Sensors for Autonomous Vehicle Perception

arXiv: https://arxiv.org/abs/2401.13853 · HTML: https://mlsanity.com/paper/2401.13853/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 60 references that are in the map.
- First posted on arXiv 24 Jan 2024.

## Builds on (most cited first, 25 of 25)

- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [A Large Scale Event-based Detection Dataset for Automotive](https://mlsanity.com/paper/2001.08499/) (2020)
- [EV-SegNet: Semantic Segmentation for Event-based Cameras](https://mlsanity.com/paper/1811.12039/) (2018)
- [DDD17: End-To-End DAVIS Driving Dataset](https://mlsanity.com/paper/1711.01458/) (2017)
- [Perception and Sensing for Autonomous Vehicles Under Adverse Weather Conditions: A Survey](https://mlsanity.com/paper/2112.08936/) (2021)
- [ESS: Learning Event-based Semantic Segmentation from Still Images](https://mlsanity.com/paper/2203.10016/) (2022)
- [DDD20 End-to-End Event Camera Driving Dataset: Fusing Frames and Events with Deep Learning for Improved Steering Prediction](https://mlsanity.com/paper/2005.08605/) (2020)
- [How to Calibrate Your Event Camera](https://mlsanity.com/paper/2105.12362/) (2021)
- [There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge](https://mlsanity.com/paper/2103.01353/) (2021)
- [HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images](https://mlsanity.com/paper/2003.04645/) (2020)
- [Ford Multi-AV Seasonal Dataset](https://mlsanity.com/paper/2003.07969/) (2020)
- [CED: Color Event Camera Dataset](https://mlsanity.com/paper/1904.10772/) (2019)
- [Event-based visual place recognition with ensembles of temporal windows](https://mlsanity.com/paper/2006.02826/) (2020)
- [LoST? Appearance-Invariant Place Recognition for Opposite Viewpoints using Visual Semantics](https://mlsanity.com/paper/1804.05526/) (2018)
- [Brno Urban Dataset -- The New Data for Self-Driving Agents and Mapping Tasks](https://mlsanity.com/paper/1909.06897/) (2019)
- [Feature-based Event Stereo Visual Odometry](https://mlsanity.com/paper/2107.04921/) (2021)
- [InfraParis: A multi-modal and multi-task autonomous driving dataset](https://mlsanity.com/paper/2309.15751/) (2023)
- [ECMD: An Event-Centric Multisensory Driving Dataset for SLAM](https://mlsanity.com/paper/2311.02327/) (2023)
- [Evaluation of Thermal Imaging on Embedded GPU Platforms for Application in Vehicular Assistance Systems](https://mlsanity.com/paper/2201.01661/) (2022)

## Built on by (most cited first, 15 of 15)

- [CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillation](https://mlsanity.com/paper/2403.19104/) (2024)
- [Event-based Stereo Depth Estimation: A Survey](https://mlsanity.com/paper/2409.17680/) (2024)
- [EventFly: Event Camera Perception from Ground to the Sky](https://mlsanity.com/paper/2503.19916/) (2025)
- [R-LiViT: A LiDAR-Visual-Thermal Dataset Enabling Vulnerable Road User Focused Roadside Perception](https://mlsanity.com/paper/2503.17122/) (2025)
- [Fieldscale: Locality-Aware Field-based Adaptive Rescaling for Thermal Infrared Image](https://mlsanity.com/paper/2405.15395/) (2024)
- [Ensemble-Based Event Camera Place Recognition Under Varying Illumination](https://mlsanity.com/paper/2509.01968/) (2025)
- [TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation](https://mlsanity.com/paper/2602.19430/) (2026)
- [Event-Driven Neuromorphic Vision Enables Energy-Efficient Visual Place Recognition](https://mlsanity.com/paper/2604.03277/) (2026)
- [Spike-EVPR: Deep Spiking Residual Networks with SNN-Tailored Representations for Event-Based Visual Place Recognition](https://mlsanity.com/paper/2402.10476/) (2024)
- [EventGeM: Global-to-Local Feature Matching for Event-Based Visual Place Recognition](https://mlsanity.com/paper/2603.05807/) (2026)
- [SPOT: Point Cloud Based Stereo Visual Place Recognition for Similar and Opposing Viewpoints](https://mlsanity.com/paper/2404.12339/) (2024)
- [Event-Based Visual Teach-and-Repeat via Fast Fourier-Domain Cross-Correlation](https://mlsanity.com/paper/2509.17287/) (2025)
- [TIDY: Thermal Infrared Image Denoising via Wavelet Domain Entropy and Directional Stripe Index](https://mlsanity.com/paper/2606.19813/) (2026)
- [Multi-viewpoint Geo-localization with Event Cameras](https://mlsanity.com/paper/2609.21219/) (2026)
- [Removing the Compute Bottleneck for Event-based Visual Place Recognition with Spiking Neural Networks](https://mlsanity.com/paper/2610.16022/) (2026)

## Abstract (arXiv)

Conventional cameras employed in autonomous vehicle (AV) systems support many perception tasks, but are challenged by low-light or high dynamic range scenes, adverse weather, and fast motion. Novel sensors, such as event and thermal cameras, offer capabilities with the potential to address these scenarios, but they remain to be fully exploited. This paper introduces the Novel Sensors for Autonomous Vehicle Perception (NSAVP) dataset to facilitate future research on this topic. The dataset was captured with a platform including stereo event, thermal, monochrome, and RGB cameras as well as a high precision navigation system providing ground truth poses. The data was collected by repeatedly driving two ~8 km routes and includes varied lighting conditions and opposing viewpoint perspectives. We provide benchmarking experiments on the task of place recognition to demonstrate challenges and opportunities for novel sensors to enhance critical AV perception tasks. To our knowledge, the NSAVP dataset is the first to include stereo thermal cameras together with stereo event and monochrome cameras. The dataset and supporting software suite is available at: https://umautobots.github.io/nsavp