# A Study of Skews, Imbalances, and Pathological Conditions in LLM Inference Deployment on GPU Clusters detectable from DPU

arXiv: https://arxiv.org/abs/2509.18114 · HTML: https://mlsanity.com/paper/2509.18114/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 9 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Builds on (most cited first, 6 of 6)

- [Dynamic Space-Time Scheduling for GPU Inference](https://mlsanity.com/paper/1901.00041/) (2018)
- [An Exhaustive Survey on P4 Programmable Data Plane Switches: Taxonomy, Applications, Challenges, and Future Trends](https://mlsanity.com/paper/2102.00643/) (2021)
- [Direct Telemetry Access](https://mlsanity.com/paper/2202.02270/) (2022)
- [D-STACK: High Throughput DNN Inference by Effective Multiplexing and Spatio-Temporal Scheduling of GPUs](https://mlsanity.com/paper/2304.13541/) (2023)
- [BCEdge: SLO-Aware DNN Inference Services with Adaptive Batching on Edge Platforms](https://mlsanity.com/paper/2305.01519/) (2023)
- [Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scaling](https://mlsanity.com/paper/2404.00704/) (2024)

## Abstract (arXiv)

Autoregressive inference in large transformer-based language models (LLMs) presents significant challenges for runtime efficiency, particularly during the decode phase where load imbalance across GPU shards can cause throughput degradation and latency spikes. A DPU-assisted framework leveraged by BlueField-3 Data Processing Units can enable real-time detection and mitigation of load imbalance in multi-node tensor-parallel inference. By offloading monitoring tasks to the DPU and analyzing GPU telemetry and inter-node communication patterns, the resulting system can provide actionable feedback to inference controllers and schedulers. The goal of this study is three-fold i) identify the reported skews/imbalances/pathological conditions that arise in muti-GPU execution of a) LLM tensor computing (both during training and inference), b) identify their impact on computational performance, and c) make a critical assessment if those can be tracked for potential mitigation from a DPU network.