# Is That Rain? Understanding Effects on Visual Odometry Performance for Autonomous UAVs and Efficient DNN-based Rain Classification at the Edge arXiv: https://arxiv.org/abs/2407.12663 ยท HTML: https://mlsanity.com/paper/2407.12663/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 55 references that are in the map. - First posted on arXiv 17 Jul 2024. ## Builds on (most cited first, 25 of 25) - [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018) - [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019) - [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016) - [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017) - [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020) - [Perception and Sensing for Autonomous Vehicles Under Adverse Weather Conditions: A Survey](https://mlsanity.com/paper/2112.08936/) (2021) - [Boreas: A Multi-Season Autonomous Driving Dataset](https://mlsanity.com/paper/2203.10168/) (2022) - [4Seasons: A Cross-Season Dataset for Multi-Weather SLAM in Autonomous Driving](https://mlsanity.com/paper/2009.06364/) (2020) - [A General Optimization-based Framework for Local Odometry Estimation with Multiple Sensors](https://mlsanity.com/paper/1901.03638/) (2019) - [A General Optimization-based Framework for Global Pose Estimation with Multiple Sensors](https://mlsanity.com/paper/1901.03642/) (2019) - [Object Detection Under Rainy Conditions for Autonomous Vehicles: A Review of State-of-the-Art and Emerging Techniques](https://mlsanity.com/paper/2006.16471/) (2020) - [Real-time Kinematic Ground Truth for the Oxford RobotCar Dataset](https://mlsanity.com/paper/2002.10152/) (2020) - [Online Temporal Calibration for Monocular Visual-Inertial Systems](https://mlsanity.com/paper/1808.00692/) (2018) - [Navion: A 2mW Fully Integrated Real-Time Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones](https://mlsanity.com/paper/1809.05780/) (2018) - [Worsening Perception: Real-time Degradation of Autonomous Vehicle Perception Performance for Simulation of Adverse Weather Conditions](https://mlsanity.com/paper/2103.02760/) (2021) - [GTAV-NightRain: Photometric Realistic Large-scale Dataset for Night-time Rain Streak Removal](https://mlsanity.com/paper/2210.04708/) (2022) - [UAV Assisted Data Collection for Internet of Things: A Survey](https://mlsanity.com/paper/2211.09555/) (2022) - [Trajectory Design for UAV-Based Internet-of-Things Data Collection: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2107.11015/) (2021) - [QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking](https://mlsanity.com/paper/2207.07789/) (2022) - [Constrained Reinforcement Learning using Distributional Representation for Trustworthy Quadrotor UAV Tracking Control](https://mlsanity.com/paper/2302.11694/) (2023) - [KinoJGM: A framework for efficient and accurate quadrotor trajectory generation and tracking in dynamic environments](https://mlsanity.com/paper/2202.12419/) (2022) - [Localization with Anticipation for Autonomous Urban Driving in Rain](https://mlsanity.com/paper/2306.09134/) (2023) - [Flexible Trinocular: Non-rigid Multi-Camera-IMU Dense Reconstruction for UAV Navigation and Mapping](https://mlsanity.com/paper/1908.08891/) (2019) - [Probabilistic Constrained Reinforcement Learning with Formal Interpretability](https://mlsanity.com/paper/2307.07084/) (2023) - [WiFi-Inertial Indoor Pose Estimation for Micro Aerial Vehicles](https://mlsanity.com/paper/2003.07240/) (2020) ## Built on by (most cited first, 1 of 1) - [AI and Vision based Autonomous Navigation of Nano-Drones in Partially-Known Environments](https://mlsanity.com/paper/2505.04972/) (2025) ## Abstract (arXiv) The development of safe and reliable autonomous unmanned aerial vehicles relies on the ability of the system to recognise and adapt to changes in the local environment based on sensor inputs. State-of-the-art local tracking and trajectory planning are typically performed using camera sensor input to the flight control algorithm, but the extent to which environmental disturbances like rain affect the performance of these systems is largely unknown. In this paper, we first describe the development of an open dataset comprising ~335k images to examine these effects for seven different classes of precipitation conditions and show that a worst-case average tracking error of 1.5 m is possible for a state-of-the-art visual odometry system (VINS-Fusion). We then use the dataset to train a set of deep neural network models suited to mobile and constrained deployment scenarios to determine the extent to which it may be possible to efficiently and accurately classify these `rainy' conditions. The most lightweight of these models (MobileNetV3 small) can achieve an accuracy of 90% with a memory footprint of just 1.28 MB and a frame rate of 93 FPS, which is suitable for deployment in resource-constrained and latency-sensitive systems. We demonstrate a classification latency in the order of milliseconds using typical flight computer hardware. Accordingly, such a model can feed into the disturbance estimation component of an autonomous flight controller. In addition, data from unmanned aerial vehicles with the ability to accurately determine environmental conditions in real time may contribute to developing more granular timely localised weather forecasting.