# TUMTraf Event: Calibration and Fusion Resulting in a Dataset for Roadside Event-Based and RGB Cameras

arXiv: https://arxiv.org/abs/2401.08474 · HTML: https://mlsanity.com/paper/2401.08474/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 74 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Builds on (most cited first, 27 of 27)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM](https://mlsanity.com/paper/1610.08336/) (2016)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation](https://mlsanity.com/paper/2001.00309/) (2020)
- [A Large Scale Event-based Detection Dataset for Automotive](https://mlsanity.com/paper/2001.08499/) (2020)
- [DDD17: End-To-End DAVIS Driving Dataset](https://mlsanity.com/paper/1711.01458/) (2017)
- [DDD20 End-to-End Event Camera Driving Dataset: Fusing Frames and Events with Deep Learning for Improved Steering Prediction](https://mlsanity.com/paper/2005.08605/) (2020)
- [How to Calibrate Your Event Camera](https://mlsanity.com/paper/2105.12362/) (2021)
- [Frame-Event Alignment and Fusion Network for High Frame Rate Tracking](https://mlsanity.com/paper/2305.15688/) (2023)
- [A9-Dataset: Multi-Sensor Infrastructure-Based Dataset for Mobility Research](https://mlsanity.com/paper/2204.06527/) (2022)
- [RGB-Event Fusion for Moving Object Detection in Autonomous Driving](https://mlsanity.com/paper/2209.08323/) (2022)
- [Pushing the Limits of Asynchronous Graph-based Object Detection with Event Cameras](https://mlsanity.com/paper/2211.12324/) (2022)
- [InfraDet3D: Multi-Modal 3D Object Detection based on Roadside Infrastructure Camera and LiDAR Sensors](https://mlsanity.com/paper/2305.00314/) (2023)
- [Providentia -- A Large-Scale Sensor System for the Assistance of Autonomous Vehicles and Its Evaluation](https://mlsanity.com/paper/1906.06789/) (2019)
- [Intelligent Transportation Systems Using External Infrastructure: A Literature Survey](https://mlsanity.com/paper/2112.05615/) (2021)
- [Dynamic Event Camera Calibration](https://mlsanity.com/paper/2107.06749/) (2021)
- [RGB-D And Thermal Sensor Fusion: A Systematic Literature Review](https://mlsanity.com/paper/2305.11427/) (2023)
- [Extrinsic Camera Calibration with Semantic Segmentation](https://mlsanity.com/paper/2208.03949/) (2022)
- [Targetless Extrinsic Calibration of Stereo Cameras, Thermal Cameras, and Laser Sensors in the Wild](https://mlsanity.com/paper/2109.13414/) (2021)

## Built on by (most cited first, 13 of 13)

- [ActiveAnno3D -- An Active Learning Framework for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2402.03235/) (2024)
- [EF-Calib: Spatiotemporal Calibration of Event- and Frame-Based Cameras Using Continuous-Time Trajectories](https://mlsanity.com/paper/2405.17278/) (2024)
- [PointCompress3D: A Point Cloud Compression Framework for Roadside LiDARs in Intelligent Transportation Systems](https://mlsanity.com/paper/2405.01750/) (2024)
- [Enhancing Highway Safety: Accident Detection on the A9 Test Stretch Using Roadside Sensors](https://mlsanity.com/paper/2502.00402/) (2025)
- [RGB-Event Fusion with Self-Attention for Collision Prediction](https://mlsanity.com/paper/2505.04258/) (2025)
- [Safety-Critical Learning for Long-Tail Events: The TUM Traffic Accident Dataset](https://mlsanity.com/paper/2508.14567/) (2025)
- [One target to align them all: LiDAR, RGB and event cameras extrinsic calibration for Autonomous Driving](https://mlsanity.com/paper/2511.12291/) (2025)
- [TUMTraf EMOT: Event-Based Multi-Object Tracking Dataset and Baseline for Traffic Scenarios](https://mlsanity.com/paper/2512.14595/) (2025)
- [Label Correction for Road Segmentation Using Road-side Cameras](https://mlsanity.com/paper/2502.01281/) (2025)
- [Temporal and Rotational Calibration for Event-Centric Multi-Sensor Systems](https://mlsanity.com/paper/2508.12564/) (2025)
- [Event-RGB Adaptive Tracking for Nighttime Highway Perception](https://mlsanity.com/paper/2607.11646/) (2026)
- [Neuromorphic Object Detection: An In-Depth Study and Future Directions](https://mlsanity.com/paper/2607.23576/) (2026)
- [Simplified Cross-Modal Calibration for Heterogeneous Event-RGB Stereo Systems](https://mlsanity.com/paper/2608.22965/) (2026)

## Abstract (arXiv)

Event-based cameras are predestined for Intelligent Transportation Systems (ITS). They provide very high temporal resolution and dynamic range, which can eliminate motion blur and improve detection performance at night. However, event-based images lack color and texture compared to images from a conventional RGB camera. Considering that, data fusion between event-based and conventional cameras can combine the strengths of both modalities. For this purpose, extrinsic calibration is necessary. To the best of our knowledge, no targetless calibration between event-based and RGB cameras can handle multiple moving objects, nor does data fusion optimized for the domain of roadside ITS exist. Furthermore, synchronized event-based and RGB camera datasets considering roadside perspective are not yet published. To fill these research gaps, based on our previous work, we extended our targetless calibration approach with clustering methods to handle multiple moving objects. Furthermore, we developed an early fusion, simple late fusion, and a novel spatiotemporal late fusion method. Lastly, we published the TUMTraf Event Dataset, which contains more than 4,111 synchronized event-based and RGB images with 50,496 labeled 2D boxes. During our extensive experiments, we verified the effectiveness of our calibration method with multiple moving objects. Furthermore, compared to a single RGB camera, we increased the detection performance of up to +9 % mAP in the day and up to +13 % mAP during the challenging night with our presented event-based sensor fusion methods. The TUMTraf Event Dataset is available at https://innovation-mobility.com/tumtraf-dataset.