# Collaborative Reinforcement Learning Based Unmanned Aerial Vehicle (UAV) Trajectory Design for 3D UAV Tracking

arXiv: https://arxiv.org/abs/2401.12079 · HTML: https://mlsanity.com/paper/2401.12079/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 22 Jan 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 22 of 22)

- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [Joint Trajectory and Communication Design for Multi-UAV Enabled Wireless Networks](https://mlsanity.com/paper/1705.02723/) (2017)
- [Distributed Multi-agent Meta Learning for Trajectory Design in Wireless Drone Networks](https://mlsanity.com/paper/2012.03158/) (2020)
- [DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning](https://mlsanity.com/paper/2102.07936/) (2021)
- [Onboard dynamic-object detection and tracking for autonomous robot navigation with RGB-D camera](https://mlsanity.com/paper/2303.00132/) (2023)
- [Distributed Reinforcement Learning for Age of Information Minimization in Real-Time IoT Systems](https://mlsanity.com/paper/2104.01527/) (2021)
- [Optimal 3D-Trajectory Design and Resource Allocation for Solar-Powered UAV Communication Systems](https://mlsanity.com/paper/1808.00101/) (2018)
- [Environment Sensing Considering the Occlusion Effect: A Multi-View Approach](https://mlsanity.com/paper/2207.00776/) (2022)
- [Sim-to-Real Deep Reinforcement Learning based Obstacle Avoidance for UAVs under Measurement Uncertainty](https://mlsanity.com/paper/2303.07243/) (2023)
- [Robust Reinforcement Learning Algorithm for Vision-based Ship Landing of UAVs](https://mlsanity.com/paper/2209.08381/) (2022)
- [UAV Tracking with Lidar as a Camera Sensors in GNSS-Denied Environments](https://mlsanity.com/paper/2303.00277/) (2023)
- [Experimental Study of Outdoor UAV Localization and Tracking using Passive RF Sensing](https://mlsanity.com/paper/2108.07857/) (2021)
- [PAPIR: Practical RIS-aided Localization via Statistical User Information](https://mlsanity.com/paper/2109.08532/) (2021)
- [Energy-Efficient Cellular-Connected UAV Swarm Control Optimization](https://mlsanity.com/paper/2303.10398/) (2023)

## Abstract (arXiv)

In this paper, the problem of using one active unmanned aerial vehicle (UAV) and four passive UAVs to localize a 3D target UAV in real time is investigated. In the considered model, each passive UAV receives reflection signals from the target UAV, which are initially transmitted by the active UAV. The received reflection signals allow each passive UAV to estimate the signal transmission distance which will be transmitted to a base station (BS) for the estimation of the position of the target UAV. Due to the movement of the target UAV, each active/passive UAV must optimize its trajectory to continuously localize the target UAV. Meanwhile, since the accuracy of the distance estimation depends on the signal-to-noise ratio of the transmission signals, the active UAV must optimize its transmit power. This problem is formulated as an optimization problem whose goal is to jointly optimize the transmit power of the active UAV and trajectories of both active and passive UAVs so as to maximize the target UAV positioning accuracy. To solve this problem, a Z function decomposition based reinforcement learning (ZD-RL) method is proposed. Compared to value function decomposition based RL (VD-RL), the proposed method can find the probability distribution of the sum of future rewards to accurately estimate the expected value of the sum of future rewards thus finding better transmit power of the active UAV and trajectories for both active and passive UAVs and improving target UAV positioning accuracy. Simulation results show that the proposed ZD-RL method can reduce the positioning errors by up to 39.4% and 64.6%, compared to VD-RL and independent deep RL methods, respectively.