# HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment

arXiv: https://arxiv.org/abs/2502.07903 · HTML: https://mlsanity.com/paper/2502.07903/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2025.

## Built on by (most cited first, 20 of 20)

- [Taming the Titans: A Survey of Efficient LLM Inference Serving](https://mlsanity.com/paper/2504.19720/) (2025)
- [Parallax: Efficient LLM Inference Service over Decentralized Environment](https://mlsanity.com/paper/2509.26182/) (2025)
- [UCCL-EP: Portable Expert-Parallel Communication](https://mlsanity.com/paper/2512.19849/) (2025)
- [Beyond the Buzz: A Pragmatic Take on Inference Disaggregation](https://mlsanity.com/paper/2506.05508/) (2025)
- [AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs](https://mlsanity.com/paper/2511.00796/) (2025)
- [FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel](https://mlsanity.com/paper/2508.18224/) (2025)
- [BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2602.10729/) (2026)
- [Cascadia: An Efficient Cascade Serving System for Large Language Models](https://mlsanity.com/paper/2506.04203/) (2025)
- [Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs](https://mlsanity.com/paper/2509.17542/) (2025)
- [OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration](https://mlsanity.com/paper/2602.12151/) (2026)
- [Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods](https://mlsanity.com/paper/2605.05227/) (2026)
- [Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving](https://mlsanity.com/paper/2606.01839/) (2026)
- [TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference](https://mlsanity.com/paper/2602.05145/) (2026)
- [Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamics](https://mlsanity.com/paper/2604.07144/) (2026)
- [RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving](https://mlsanity.com/paper/2606.17949/) (2026)
- [Trust-Aware Routing for Distributed Generative AI Inference at the Edge](https://mlsanity.com/paper/2603.28622/) (2026)
- [Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation](https://mlsanity.com/paper/2604.10180/) (2026)
- [HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling](https://mlsanity.com/paper/2605.16637/) (2026)
- [E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments](https://mlsanity.com/paper/2606.03770/) (2026)
- [OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving](https://mlsanity.com/paper/2609.14237/) (2026)

## Abstract (arXiv)

Disaggregating the prefill and decoding phases represents an effective new paradigm for generative inference of large language models (LLM), which eliminates prefill-decoding interference and optimizes resource allocation. However, it is still an open problem about how to deploy the disaggregated inference paradigm across a group of heterogeneous GPUs, which can be an economical alternative to deployment over homogeneous high-performance GPUs. Towards this end, we introduce HexGen-2, a distributed system for efficient and economical LLM serving on heterogeneous GPUs following the disaggregated paradigm. Built on top of HexGen, the core component of HexGen-2 is a scheduling algorithm that formalizes the allocation of disaggregated LLM inference computations and communications over heterogeneous GPUs and network connections as a constraint optimization problem. We leverage the graph partitioning and max-flow algorithms to co-optimize resource allocation, parallel strategies for distinct inference phases, and the efficiency of inter-phase key-value (KV) cache communications. We conduct extensive experiments to evaluate HexGen-2, i.e., on OPT (30B) and Llama-2 (70B) models in various real-world settings, the results reveal that HexGen-2 delivers up to a 2.0 times and on average a 1.3 times improvement in serving throughput, reduces the average inference latency by 1.5 times compared with state-of-the-art systems given the same price budget, and achieves comparable inference performance with a 30% lower price budget.