# Dynamic Routing for Integrated Satellite-Terrestrial Networks: A Constrained Multi-Agent Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/2401.09455 · HTML: https://mlsanity.com/paper/2401.09455/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 51 references that are in the map.
- First posted on arXiv 23 Dec 2023.

## Builds on (most cited first, 11 of 11)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Penalized Proximal Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2205.11814/) (2022)
- [Multi-Agent Constrained Policy Optimisation](https://mlsanity.com/paper/2110.02793/) (2021)
- [CubeSat Communications: Recent Advances and Future Challenges](https://mlsanity.com/paper/1908.09501/) (2019)
- [Integrating LEO Satellites and Multi-UAV Reinforcement Learning for Hybrid FSO/RF Non-Terrestrial Networks](https://mlsanity.com/paper/2010.10138/) (2020)

## Built on by (most cited first, 9 of 9)

- [Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directions](https://mlsanity.com/paper/2407.04581/) (2024)
- [Continual Deep Reinforcement Learning for Decentralized Satellite Routing](https://mlsanity.com/paper/2405.12308/) (2024)
- [Online Learning for Dynamic Constellation Topologies](https://mlsanity.com/paper/2603.25954/) (2026)
- [Asynchronous Risk-Aware Multi-Agent Packet Routing for Ultra-Dense LEO Satellite Networks](https://mlsanity.com/paper/2510.27506/) (2025)
- [Blockchain-Enabled Routing for Zero-Trust Low-Altitude Intelligent Networks](https://mlsanity.com/paper/2602.23667/) (2026)
- [MASK: Multi-Agent Semantic K-Scheduling for Risk-Sensitive 6G Robotics](https://mlsanity.com/paper/2606.11249/) (2026)
- [ASTRA: ADMM-Accelerated Topology Reconfiguration for Dynamic Satellite Constellations](https://mlsanity.com/paper/2609.33993/) (2026)
- [VERA: Verifiable Feasibility Representations with Counterfactual Credit for Constrained Multi-Agent Control](https://mlsanity.com/paper/2609.38889/) (2026)
- [Goal-Oriented Bundle Management and Flow Control for Deep-Space Communications via Reinforcement Learning](https://mlsanity.com/paper/2610.13923/) (2026)

## Abstract (arXiv)

The integrated satellite-terrestrial network (ISTN) system has experienced significant growth, offering seamless communication services in remote areas with limited terrestrial infrastructure. However, designing a routing scheme for ISTN is exceedingly difficult, primarily due to the heightened complexity resulting from the inclusion of additional ground stations, along with the requirement to satisfy various constraints related to satellite service quality. To address these challenges, we study packet routing with ground stations and satellites working jointly to transmit packets, while prioritizing fast communication and meeting energy efficiency and packet loss requirements. Specifically, we formulate the problem of packet routing with constraints as a max-min problem using the Lagrange method. Then we propose a novel constrained Multi-Agent reinforcement learning (MARL) dynamic routing algorithm named CMADR, which efficiently balances objective improvement and constraint satisfaction during the updating of policy and Lagrange multipliers. Finally, we conduct extensive experiments and an ablation study using the OneWeb and Telesat mega-constellations. Results demonstrate that CMADR reduces the packet delay by a minimum of 21% and 15%, while meeting stringent energy consumption and packet loss rate constraints, outperforming several baseline algorithms.