# RGB-Event Fusion with Self-Attention for Collision Prediction

arXiv: https://arxiv.org/abs/2505.04258 · HTML: https://mlsanity.com/paper/2505.04258/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 26 of 26)

- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [LSQ+: Improving low-bit quantization through learnable offsets and better initialization](https://mlsanity.com/paper/2004.09576/) (2020)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [Forward and Backward Information Retention for Accurate Binary Neural Networks](https://mlsanity.com/paper/1909.10788/) (2019)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction](https://mlsanity.com/paper/2102.09320/) (2021)
- [From Chaos Comes Order: Ordering Event Representations for Object Recognition and Detection](https://mlsanity.com/paper/2304.13455/) (2023)
- [EVDodgeNet: Deep Dynamic Obstacle Dodging with Event Cameras](https://mlsanity.com/paper/1906.02919/) (2019)
- [RGB-Event Fusion for Moving Object Detection in Autonomous Driving](https://mlsanity.com/paper/2209.08323/) (2022)
- [Fully neuromorphic vision and control for autonomous drone flight](https://mlsanity.com/paper/2303.08778/) (2023)
- [NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems](https://mlsanity.com/paper/2304.04640/) (2023)
- [Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware](https://mlsanity.com/paper/2312.00425/) (2023)
- [Event-based Agile Object Catching with a Quadrupedal Robot](https://mlsanity.com/paper/2303.17479/) (2023)
- [FAST-Dynamic-Vision: Detection and Tracking Dynamic Objects with Event and Depth Sensing](https://mlsanity.com/paper/2103.05903/) (2021)
- [Monocular Event-Based Vision for Obstacle Avoidance with a Quadrotor](https://mlsanity.com/paper/2411.03303/) (2024)
- [Deep Visual Odometry with Events and Frames](https://mlsanity.com/paper/2309.09947/) (2023)
- [TUMTraf Event: Calibration and Fusion Resulting in a Dataset for Roadside Event-Based and RGB Cameras](https://mlsanity.com/paper/2401.08474/) (2024)
- [Fast Motion Understanding with Spatiotemporal Neural Networks and Dynamic Vision Sensors](https://mlsanity.com/paper/2011.09427/) (2020)
- [EVReflex: Dense Time-to-Impact Prediction for Event-based Obstacle Avoidance](https://mlsanity.com/paper/2109.00405/) (2021)
- [Fast Trajectory End-Point Prediction with Event Cameras for Reactive Robot Control](https://mlsanity.com/paper/2302.13796/) (2023)
- [A real-time dynamic obstacle tracking and mapping system for UAV navigation and collision avoidance with an RGB-D camera](https://mlsanity.com/paper/2209.08258/) (2022)
- [ColibriUAV: An Ultra-Fast, Energy-Efficient Neuromorphic Edge Processing UAV-Platform with Event-Based and Frame-Based Cameras](https://mlsanity.com/paper/2305.18371/) (2023)
- [Free as a Bird: Event-based Dynamic Sense-and-Avoid for Ornithopter Robot Flight](https://mlsanity.com/paper/2310.09875/) (2023)
- [Night vision obstacle detection and avoidance based on Bio-Inspired Vision Sensors](https://mlsanity.com/paper/2010.15509/) (2020)
- [Towards Low-Latency Event-based Obstacle Avoidance on a FPGA-Drone](https://mlsanity.com/paper/2504.10400/) (2025)

## Built on by (most cited first, 1 of 1)

- [SkyEV: RGB-Event UAV detection and tracking dataset and baseline](https://mlsanity.com/paper/2607.18747/) (2026)

## Abstract (arXiv)

Ensuring robust and real-time obstacle avoidance is critical for the safe operation of autonomous robots in dynamic, real-world environments. This paper proposes a neural network framework for predicting the time and collision position of an unmanned aerial vehicle with a dynamic object, using RGB and event-based vision sensors. The proposed architecture consists of two separate encoder branches, one for each modality, followed by fusion by self-attention to improve prediction accuracy. To facilitate benchmarking, we leverage the ABCD [8] dataset collected that enables detailed comparisons of single-modality and fusion-based approaches. At the same prediction throughput of 50Hz, the experimental results show that the fusion-based model offers an improvement in prediction accuracy over single-modality approaches of 1% on average and 10% for distances beyond 0.5m, but comes at the cost of +71% in memory and + 105% in FLOPs. Notably, the event-based model outperforms the RGB model by 4% for position and 26% for time error at a similar computational cost, making it a competitive alternative. Additionally, we evaluate quantized versions of the event-based models, applying 1- to 8-bit quantization to assess the trade-offs between predictive performance and computational efficiency. These findings highlight the trade-offs of multi-modal perception using RGB and event-based cameras in robotic applications.