# Model-aided Deep Reinforcement Learning for Sample-efficient UAV Trajectory Design in IoT Networks

arXiv: https://arxiv.org/abs/2104.10403 · HTML: https://mlsanity.com/paper/2104.10403/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 21 Apr 2021.

## Builds on (most cited first, 6 of 6)

- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [Continuous Deep Q-Learning with Model-based Acceleration](https://mlsanity.com/paper/1603.00748/) (2016)
- [Accessing From The Sky: A Tutorial on UAV Communications for 5G and Beyond](https://mlsanity.com/paper/1903.05289/) (2019)
- [Distributed Multi-agent Meta Learning for Trajectory Design in Wireless Drone Networks](https://mlsanity.com/paper/2012.03158/) (2020)
- [UAV Path Planning for Wireless Data Harvesting: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2007.00544/) (2020)

## Built on by (most cited first, 1 of 1)

- [Model-aided Federated Reinforcement Learning for Multi-UAV Trajectory Planning in IoT Networks](https://mlsanity.com/paper/2306.02029/) (2023)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) is gaining attention as a potential approach to design trajectories for autonomous unmanned aerial vehicles (UAV) used as flying access points in the context of cellular or Internet of Things (IoT) connectivity. DRL solutions offer the advantage of on-the-go learning hence relying on very little prior contextual information. A corresponding drawback however lies in the need for many learning episodes which severely restricts the applicability of such approach in real-world time- and energy-constrained missions. Here, we propose a model-aided deep Q-learning approach that, in contrast to previous work, considerably reduces the need for extensive training data samples, while still achieving the overarching goal of DRL, i.e to guide a battery-limited UAV on an efficient data harvesting trajectory, without prior knowledge of wireless channel characteristics and limited knowledge of wireless node locations. The key idea consists in using a small subset of nodes as anchors (i.e. with known location) and learning a model of the propagation environment while implicitly estimating the positions of regular nodes. Interaction with the model allows us to train a deep Q-network (DQN) to approximate the optimal UAV control policy. We show that in comparison with standard DRL approaches, the proposed model-aided approach requires at least one order of magnitude less training data samples to reach identical data collection performance, hence offering a first step towards making DRL a viable solution to the problem.