# Learning Sub-Second Routing Optimization in Computer Networks requires Packet-Level Dynamics

arXiv: https://arxiv.org/abs/2410.10377 · HTML: https://mlsanity.com/paper/2410.10377/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 85 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [A Study on Overfitting in Deep Reinforcement Learning](https://mlsanity.com/paper/1804.06893/) (2018)
- [Age of Information: An Introduction and Survey](https://mlsanity.com/paper/2007.08564/) (2020)
- [RouteNet-Fermi: Network Modeling with Graph Neural Networks](https://mlsanity.com/paper/2212.12070/) (2022)
- [Swarm Reinforcement Learning For Adaptive Mesh Refinement](https://mlsanity.com/paper/2304.00818/) (2023)
- [A Deep-Reinforcement Learning Approach for Software-Defined Networking Routing Optimization](https://mlsanity.com/paper/1709.07080/) (2017)
- [Teal: Learning-Accelerated Optimization of WAN Traffic Engineering](https://mlsanity.com/paper/2210.13763/) (2022)
- [MAGNNETO: A Graph Neural Network-based Multi-Agent system for Traffic Engineering](https://mlsanity.com/paper/2303.18157/) (2023)
- [Knowledge-Defined Networking](https://mlsanity.com/paper/1606.06222/) (2016)
- [Packet Routing with Graph Attention Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2107.13181/) (2021)
- [REPETITA: Repeatable Experiments for Performance Evaluation of Traffic-Engineering Algorithms](https://mlsanity.com/paper/1710.08665/) (2017)
- [Impact of Evolving Protocols and COVID-19 on Internet Traffic Shares](https://mlsanity.com/paper/2201.00142/) (2022)
- [Updating the Theory of Buffer Sizing](https://mlsanity.com/paper/2109.11693/) (2021)

## Built on by (most cited first, 2 of 2)

- [Can Neural Networks Provide Latent Embeddings for Telemetry-Aware Greedy Routing?](https://mlsanity.com/paper/2602.12798/) (2026)
- [Towards Near-Real-Time Telemetry-Aware Routing with Neural Routing Algorithms](https://mlsanity.com/paper/2604.02927/) (2026)

## Abstract (arXiv)

Finding efficient routes for data packets is an essential task in computer networking. The optimal routes depend greatly on the current network topology, state and traffic demand, and they can change within milliseconds. Reinforcement Learning can help to learn network representations that provide routing decisions for possibly novel situations. So far, this has commonly been done using fluid network models. We investigate their suitability for millisecond-scale adaptations with a range of traffic mixes and find that packet-level network models are necessary to capture true dynamics, in particular in the presence of TCP traffic. To this end, we present textitPackeRL, the first packet-level Reinforcement Learning environment for routing in generic network topologies. Our experiments confirm that learning-based strategies that have been trained in fluid environments do not generalize well to this more realistic, but more challenging setup. Hence, we also introduce two new algorithms for learning sub-second Routing Optimization. We present textitM-Slim, a dynamic shortest-path algorithm that excels at high traffic volumes but is computationally hard to scale to large network topologies, and textitFieldLines, a novel next-hop policy design that re-optimizes routing for any network topology within milliseconds without requiring any re-training. Both algorithms outperform current learning-based approaches as well as commonly used static baseline protocols in scenarios with high-traffic volumes. All findings are backed by extensive experiments in realistic network conditions in our fast and versatile training and evaluation framework.