# Deep Echo State Q-Network (DEQN) and Its Application in Dynamic Spectrum Sharing for 5G and Beyond

arXiv: https://arxiv.org/abs/2010.05449 · HTML: https://mlsanity.com/paper/2010.05449/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 7 of 7)

- [Survey of Graph Neural Network for Internet of Things and NextG Networks](https://mlsanity.com/paper/2405.17309/) (2024)
- [Deep Q-network using reservoir computing with multi-layered readout](https://mlsanity.com/paper/2203.01465/) (2022)
- [Towards Explainable Machine Learning: The Effectiveness of Reservoir Computing in Wireless Receive Processing](https://mlsanity.com/paper/2310.04956/) (2023)
- [Distributed Learning Meets 6G: A Communication and Computing Perspective](https://mlsanity.com/paper/2303.12802/) (2023)
- [Chaos-based reinforcement learning with TD3](https://mlsanity.com/paper/2405.09086/) (2024)
- [Federated Dynamic Spectrum Access](https://mlsanity.com/paper/2106.14976/) (2021)
- [Universal Approximation of Linear Time-Invariant (LTI) Systems through RNNs: Power of Randomness in Reservoir Computing](https://mlsanity.com/paper/2308.02464/) (2023)

## Abstract (arXiv)

Deep reinforcement learning (DRL) has been shown to be successful in many application domains. Combining recurrent neural networks (RNNs) and DRL further enables DRL to be applicable in non-Markovian environments by capturing temporal information. However, training of both DRL and RNNs is known to be challenging requiring a large amount of training data to achieve convergence. In many targeted applications, such as those used in the fifth generation (5G) cellular communication, the environment is highly dynamic while the available training data is very limited. Therefore, it is extremely important to develop DRL strategies that are capable of capturing the temporal correlation of the dynamic environment requiring limited training overhead. In this paper, we introduce the deep echo state Q-network (DEQN) that can adapt to the highly dynamic environment in a short period of time with limited training data. We evaluate the performance of the introduced DEQN method under the dynamic spectrum sharing (DSS) scenario, which is a promising technology in 5G and future 6G networks to increase the spectrum utilization. Compared to conventional spectrum management policy that grants a fixed spectrum band to a single system for exclusive access, DSS allows the secondary system to share the spectrum with the primary system. Our work sheds light on the application of an efficient DRL framework in highly dynamic environments with limited available training data.