# A real-time dynamic obstacle tracking and mapping system for UAV navigation and collision avoidance with an RGB-D camera

arXiv: https://arxiv.org/abs/2209.08258 · HTML: https://mlsanity.com/paper/2209.08258/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 17 Sep 2022.

## Builds on (most cited first, 12 of 12)

- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [Recursive Social Behavior Graph for Trajectory Prediction](https://mlsanity.com/paper/2004.10402/) (2020)
- [Towards Monocular Vision based Obstacle Avoidance through Deep Reinforcement Learning](https://mlsanity.com/paper/1706.09829/) (2017)
- [DPMPC-Planner: A real-time UAV trajectory planning framework for complex static environments with dynamic obstacles](https://mlsanity.com/paper/2109.07024/) (2021)
- [Autonomous UAV Exploration of Dynamic Environments via Incremental Sampling and Probabilistic Roadmap](https://mlsanity.com/paper/2010.07429/) (2020)
- [Large-scale Autonomous Flight with Real-time Semantic SLAM under Dense Forest Canopy](https://mlsanity.com/paper/2109.06479/) (2021)
- [Robust Vision-based Obstacle Avoidance for Micro Aerial Vehicles in Dynamic Environments](https://mlsanity.com/paper/2002.04920/) (2020)
- [Vision-aided UAV navigation and dynamic obstacle avoidance using gradient-based B-spline trajectory optimization](https://mlsanity.com/paper/2209.07003/) (2022)
- [Leveraging Stereo-Camera Data for Real-Time Dynamic Obstacle Detection and Tracking](https://mlsanity.com/paper/2007.10743/) (2020)
- [Autonomous Flights in Dynamic Environments with Onboard Vision](https://mlsanity.com/paper/2103.05870/) (2021)
- [Continuous Occupancy Mapping in Dynamic Environments Using Particles](https://mlsanity.com/paper/2202.06273/) (2022)

## Built on by (most cited first, 5 of 5)

- [UAVScenes: A Multi-Modal Dataset for UAVs](https://mlsanity.com/paper/2507.22412/) (2025)
- [A Safer Vision-based Autonomous Planning System for Quadrotor UAVs with Dynamic Obstacle Trajectory Prediction and Its Application with LLMs](https://mlsanity.com/paper/2311.12893/) (2023)
- [Towards Low-Latency Event-based Obstacle Avoidance on a FPGA-Drone](https://mlsanity.com/paper/2504.10400/) (2025)
- [RGB-Event Fusion with Self-Attention for Collision Prediction](https://mlsanity.com/paper/2505.04258/) (2025)
- [ODTFormer: Efficient Obstacle Detection and Tracking with Stereo Cameras Based on Transformer](https://mlsanity.com/paper/2403.14626/) (2024)

## Abstract (arXiv)

The real-time dynamic environment perception has become vital for autonomous robots in crowded spaces. Although the popular voxel-based mapping methods can efficiently represent 3D obstacles with arbitrarily complex shapes, they can hardly distinguish between static and dynamic obstacles, leading to the limited performance of obstacle avoidance. While plenty of sophisticated learning-based dynamic obstacle detection algorithms exist in autonomous driving, the quadcopter's limited computation resources cannot achieve real-time performance using those approaches. To address these issues, we propose a real-time dynamic obstacle tracking and mapping system for quadcopter obstacle avoidance using an RGB-D camera. The proposed system first utilizes a depth image with an occupancy voxel map to generate potential dynamic obstacle regions as proposals. With the obstacle region proposals, the Kalman filter and our continuity filter are applied to track each dynamic obstacle. Finally, the environment-aware trajectory prediction method is proposed based on the Markov chain using the states of tracked dynamic obstacles. We implemented the proposed system with our custom quadcopter and navigation planner. The simulation and physical experiments show that our methods can successfully track and represent obstacles in dynamic environments in real-time and safely avoid obstacles. Our software is available on GitHub as an open-source ROS package.