# Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism

arXiv: https://arxiv.org/abs/2509.08309 · HTML: https://mlsanity.com/paper/2509.08309/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2025.

## Built on by (most cited first, 9 of 9)

- [Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter](https://mlsanity.com/paper/2604.15039/) (2026)
- [AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs](https://mlsanity.com/paper/2511.00796/) (2025)
- [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](https://mlsanity.com/paper/2512.13525/) (2025)
- [FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving](https://mlsanity.com/paper/2602.22593/) (2026)
- [PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving](https://mlsanity.com/paper/2604.12171/) (2026)
- [Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation](https://mlsanity.com/paper/2604.10180/) (2026)
- [RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Serving](https://mlsanity.com/paper/2604.10907/) (2026)
- [Lodestar: An Online-Learning LLM Inference Router](https://mlsanity.com/paper/2606.00946/) (2026)
- [Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving](https://mlsanity.com/paper/2607.16488/) (2026)

## Abstract (arXiv)

The significant resource demands in LLM serving prompts production clusters to fully utilize heterogeneous hardware by partitioning LLM models across a mix of high-end and low-end GPUs. However, existing parallelization approaches often struggle to scale efficiently in heterogeneous environments due to their coarse-grained and static parallelization strategies. In this paper, we introduce Hetis, a new LLM system tailored for heterogeneous GPU clusters. Hetis addresses two critical challenges: (1) memory inefficiency caused by the mismatch between memory capacity and computational power in heterogeneous devices, and (2) computational inefficiency arising from performance gaps across different LLM modules. To tackle these issues, Hetis employs a fine-grained and dynamic parallelism design. Specifically, it selectively parallelizes compute-intensive operations to reduce latency and dynamically distributes Attention computations to low-end GPUs at a head granularity, leveraging the distinct characteristics of each module. Additionally, Hetis features an online load dispatching policy that continuously optimizes serving performance by carefully balancing network latency, computational load, and memory intensity. Evaluation results demonstrate that Hetis can improve serving throughput by up to 2.25× and reduce latency by 1.49× compared to existing systems.