# Moirai: Towards Optimal Placement for Distributed Inference on Heterogeneous Devices

arXiv: https://arxiv.org/abs/2312.04025 · HTML: https://mlsanity.com/paper/2312.04025/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Horovod: fast and easy distributed deep learning in TensorFlow](https://mlsanity.com/paper/1802.05799/) (2018)
- [Memory-Efficient Pipeline-Parallel DNN Training](https://mlsanity.com/paper/2006.09503/) (2020)
- [Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training](https://mlsanity.com/paper/2110.14883/) (2021)
- [Device Placement Optimization with Reinforcement Learning](https://mlsanity.com/paper/1706.04972/) (2017)
- [A Runtime-Based Computational Performance Predictor for Deep Neural Network Training](https://mlsanity.com/paper/2102.00527/) (2021)
- [Placeto: Learning Generalizable Device Placement Algorithms for Distributed Machine Learning](https://mlsanity.com/paper/1906.08879/) (2019)
- [Efficient Algorithms for Device Placement of DNN Graph Operators](https://mlsanity.com/paper/2006.16423/) (2020)
- [On Optimizing Operator Fusion Plans for Large-Scale Machine Learning in SystemML](https://mlsanity.com/paper/1801.00829/) (2018)
- [SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstraction](https://mlsanity.com/paper/2301.08984/) (2023)
- [Dynamic DNN Decomposition for Lossless Synergistic Inference](https://mlsanity.com/paper/2101.05952/) (2021)
- [Baechi: Fast Device Placement of Machine Learning Graphs](https://mlsanity.com/paper/2301.08695/) (2023)
- [Communication-Aware Scheduling of Precedence-Constrained Tasks on Related Machines](https://mlsanity.com/paper/2004.14639/) (2020)
- [Celeritas: Fast Optimizer for Large Dataflow Graphs](https://mlsanity.com/paper/2208.00184/) (2022)

## Built on by (most cited first, 1 of 1)

- [gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling](https://mlsanity.com/paper/2504.14775/) (2025)

## Abstract (arXiv)

The escalating size of Deep Neural Networks (DNNs) has spurred a growing research interest in hosting and serving DNN models across multiple devices. A number of studies have been reported to partition a DNN model across devices, providing device placement solutions. The methods appeared in the literature, however, either suffer from poor placement performance due to the exponential search space or miss an optimal placement as a consequence of the reduced search space with limited heuristics. Moreover, these methods have ignored the runtime inter-operator optimization of a computation graph when coarsening the graph, which degrades the end-to-end inference performance. This paper presents Moirai that better exploits runtime inter-operator fusion in a model to render a coarsened computation graph, reducing the search space while maintaining the inter-operator optimization provided by inference backends. Moirai also generalizes the device placement algorithm from multiple perspectives by considering inference constraints and device heterogeneity.Extensive experimental evaluation with 11 large DNNs demonstrates that Moirai outperforms the state-of-the-art counterparts, i.e., Placeto, m-SCT, and GETF, up to 4.28× in reduction of the end-to-end inference latency. Moirai code is anonymously released at urlhttps://github.com/moirai-placement/moirai.