# Continual Deep Reinforcement Learning for Decentralized Satellite Routing

arXiv: https://arxiv.org/abs/2405.12308 · HTML: https://mlsanity.com/paper/2405.12308/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 56 references that are in the map.
- First posted on arXiv 20 May 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 24 of 24)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Federated Learning in Satellite Constellations](https://mlsanity.com/paper/2206.00307/) (2022)
- [Artificial Intelligence for Satellite Communication: A Review](https://mlsanity.com/paper/2101.10899/) (2021)
- [Revolutionizing Future Connectivity: A Contemporary Survey on AI-empowered Satellite-based Non-Terrestrial Networks in 6G](https://mlsanity.com/paper/2303.01633/) (2023)
- [Inter-Plane Inter-Satellite Connectivity in Dense LEO Constellations](https://mlsanity.com/paper/2005.07965/) (2020)
- [Learning to Continuously Optimize Wireless Resource in a Dynamic Environment: A Bilevel Optimization Perspective](https://mlsanity.com/paper/2105.01696/) (2021)
- [Satellite edge computing for real-time and very-high resolution Earth observation](https://mlsanity.com/paper/2212.12912/) (2022)
- [Dynamic Routing for Integrated Satellite-Terrestrial Networks: A Constrained Multi-Agent Reinforcement Learning Approach](https://mlsanity.com/paper/2401.09455/) (2023)
- [MaxEVA: Maximizing the Efficiency of Matrix Multiplication on Versal AI Engine](https://mlsanity.com/paper/2311.04980/) (2023)
- [Continual Learning-Based MIMO Channel Estimation: A Benchmarking Study](https://mlsanity.com/paper/2211.10753/) (2022)
- [NGSO Constellation Design for Global Connectivity](https://mlsanity.com/paper/2203.16597/) (2022)
- [Exploiting topology awareness for routing in LEO satellite constellations](https://mlsanity.com/paper/2006.12242/) (2020)
- [Deep Learning Aided Routing for Space-Air-Ground Integrated Networks Relying on Real Satellite, Flight, and Shipping Data](https://mlsanity.com/paper/2110.15138/) (2021)
- [Scheduling for On-Board Federated Learning with Satellite Clusters](https://mlsanity.com/paper/2402.09105/) (2024)
- [Multi-Agent Deep Reinforcement Learning for Distributed Satellite Routing](https://mlsanity.com/paper/2402.17666/) (2024)
- [Latency and timeliness in multi-hop satellite networks](https://mlsanity.com/paper/1910.12767/) (2019)

## Built on by (most cited first, 6 of 6)

- [Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey](https://mlsanity.com/paper/2412.13840/) (2024)
- [An open source Multi-Agent Deep Reinforcement Learning Routing Simulator for satellite networks](https://mlsanity.com/paper/2407.11047/) (2024)
- [Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systems](https://mlsanity.com/paper/2506.05577/) (2025)
- [Continual Learning to Generalize Forwarding Strategies for Diverse Mobile Wireless Networks](https://mlsanity.com/paper/2509.23913/) (2025)
- [Learning Decentralized Routing Policies via Graph Attention-based Multi-Agent Reinforcement Learning in Lunar Delay-Tolerant Networks](https://mlsanity.com/paper/2510.20436/) (2025)
- [Asynchronous Risk-Aware Multi-Agent Packet Routing for Ultra-Dense LEO Satellite Networks](https://mlsanity.com/paper/2510.27506/) (2025)

## Abstract (arXiv)

This paper introduces a full solution for decentralized routing in Low Earth Orbit satellite constellations based on continual Deep Reinforcement Learning (DRL). This requires addressing multiple challenges, including the partial knowledge at the satellites and their continuous movement, and the time-varying sources of uncertainty in the system, such as traffic, communication links, or communication buffers. We follow a multi-agent approach, where each satellite acts as an independent decision-making agent, while acquiring a limited knowledge of the environment based on the feedback received from the nearby agents. The solution is divided into two phases. First, an offline learning phase relies on decentralized decisions and a global Deep Neural Network (DNN) trained with global experiences. Then, the online phase with local, on-board, and pre-trained DNNs requires continual learning to evolve with the environment, which can be done in two different ways: (1) Model anticipation, where the predictable conditions of the constellation are exploited by each satellite sharing local model with the next satellite; and (2) Federated Learning (FL), where each agent's model is merged first at the cluster level and then aggregated in a global Parameter Server. The results show that, without high congestion, the proposed Multi-Agent DRL framework achieves the same E2E performance as a shortest-path solution, but the latter assumes intensive communication overhead for real-time network-wise knowledge of the system at a centralized node, whereas ours only requires limited feedback exchange among first neighbour satellites. Importantly, our solution adapts well to congestion conditions and exploits less loaded paths. Moreover, the divergence of models over time is easily tackled by the synergy between anticipation, applied in short-term alignment, and FL, utilized for long-term alignment.