# Slicing Is All You Need: Towards A Universal One-Sided Algorithm for Distributed Matrix Multiplication

arXiv: https://arxiv.org/abs/2510.08874 · HTML: https://mlsanity.com/paper/2510.08874/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 38 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 10 of 10)

- [Mesh-TensorFlow: Deep Learning for Supercomputers](https://mlsanity.com/paper/1811.02084/) (2018)
- [Red-blue pebbling revisited: near optimal parallel matrix-matrix multiplication](https://mlsanity.com/paper/1908.09606/) (2019)
- [On Optimizing the Communication of Model Parallelism](https://mlsanity.com/paper/2211.05322/) (2022)
- [Exploiting Multiple Levels of Parallelism in Sparse Matrix-Matrix Multiplication](https://mlsanity.com/paper/1510.00844/) (2015)
- [RDMA-Based Algorithms for Sparse Matrix Multiplication on GPUs](https://mlsanity.com/paper/2311.18141/) (2023)
- [A sparsity-aware distributed-memory algorithm for sparse-sparse matrix multiplication](https://mlsanity.com/paper/2408.14558/) (2024)
- [Intel(R) SHMEM: GPU-initiated OpenSHMEM using SYCL](https://mlsanity.com/paper/2409.20476/) (2024)
- [Optimizing ML Concurrent Computation and Communication with GPU DMA Engines](https://mlsanity.com/paper/2412.14335/) (2024)
- [RDMA vs. RPC for Implementing Distributed Data Structures](https://mlsanity.com/paper/1910.02158/) (2019)
- [Distributed Ranges: A Model for Distributed Data Structures, Algorithms, and Views](https://mlsanity.com/paper/2406.00158/) (2024)

## Abstract (arXiv)

Many important applications across science, data analytics, and AI workloads depend on distributed matrix multiplication. Prior work has developed a large array of algorithms suitable for different problem sizes and partitionings including 1D, 2D, 1.5D, and 2.5D algorithms. A limitation of current work is that existing algorithms are limited to a subset of partitionings. Multiple algorithm implementations are required to support the full space of possible partitionings. If no algorithm implementation is available for a particular set of partitionings, one or more operands must be redistributed, increasing communication costs. This paper presents a universal one-sided algorithm for distributed matrix multiplication that supports all combinations of partitionings and replication factors. Our algorithm uses slicing (index arithmetic) to compute the sets of overlapping tiles that must be multiplied together. This list of local matrix multiplies can then either be executed directly, or reordered and lowered to an optimized IR to maximize overlap. We implement our algorithm using a high-level C++-based PGAS programming framework that performs direct GPU-to-GPU communication using intra-node interconnects. We evaluate performance for a wide variety of partitionings and replication factors, finding that our work is competitive with PyTorch DTensor, a highly optimized distributed tensor library targeting AI models.