# Deploying RDMA for Distributed Inference and Training At Scale

arXiv: https://arxiv.org/abs/2610.12815 · HTML: https://mlsanity.com/paper/2610.12815/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 50 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Builds on (most cited first, 5 of 5)

- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems](https://mlsanity.com/paper/2003.09518/) (2020)
- [Homa: A Receiver-Driven Low-Latency Transport Protocol Using Network Priorities (Complete Version)](https://mlsanity.com/paper/1803.09615/) (2018)
- [007: Democratically Finding The Cause of Packet Drops](https://mlsanity.com/paper/1802.07222/) (2018)
- [Collie: Finding Performance Anomalies in RDMA Subsystems](https://mlsanity.com/paper/2304.11467/) (2023)

## Abstract (arXiv)

RDMA can reduce communication overhead in distributed machine learning systems, but realizing these benefits in production requires more than replacing TCP. Variable-size messages, concurrent connections, bursty incasts, and service failures introduce challenges that hardware offload alone cannot address. This paper reports our experience deploying ByteExpress, a software communication layer over hardware RDMA, in ByteDance's parameter-server-based inference and training systems. We present three lessons from the deployment: efficient RDMA requires buffer management across application and transport layers; scalable communication requires coordinated management of connections, receive capacity, and outstanding data; and service availability requires recovery above the transport layer. ByteExpress combines an RDMA-aware allocator, zero-copy transfers, QP multiplexing, receive-credit flow control, a software send window, and RPC-assisted recovery. In production inference, ByteExpress reduces mean and P99 model parameter fetch latency by 35.7% and 43.7%, respectively, and total CPU usage by 10.69% compared with kernel TCP. In training, ByteExpress reduces P99 model parameter push latency by up to 74.0% compared with TCP, with no PFC pauses observed at runtime.