mlsanityData to 7 Oct 2026

HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment

arXiv 2502.07903 ↗First posted 11 Feb 2025cs.DCcited by 20

Not on the map. Why?

Abstract (arXiv)

Disaggregating the prefill and decoding phases represents an effective new paradigm for generative inference of large language models (LLM), which eliminates prefill-decoding interference and optimizes resource allocation. However, it is still an open problem about how to deploy the disaggregated inference paradigm across a group of heterogeneous GPUs, which can be an economical alternative to deployment over homogeneous high-performance GPUs. Towards this end, we introduce HexGen-2, a distributed system for efficient and economical LLM serving on heterogeneous GPUs following the disaggregated paradigm. Built on top of HexGen, the core component of HexGen-2 is a scheduling algorithm that formalizes the allocation of disaggregated LLM inference computations and communications over heterogeneous GPUs and network connections as a constraint optimization problem. We leverage the graph partitioning and max-flow algorithms to co-optimize resource allocation, parallel strategies for distinct inference phases, and the efficiency of inter-phase key-value (KV) cache communications. We conduct extensive experiments to evaluate HexGen-2, i.e., on OPT (30B) and Llama-2 (70B) models in various real-world settings, the results reveal that HexGen-2 delivers up to a 2.0 times and on average a 1.3 times improvement in serving throughput, reduces the average inference latency by 1.5 times compared with state-of-the-art systems given the same price budget, and achieves comparable inference performance with a 30% lower price budget.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Feb 2025.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 12.

20 papers.

Sort:

By area:

  1. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  2. 2025Parallax: Efficient LLM Inference Service over Decentralized Environmentcited by 12
  3. 2025UCCL-EP: Portable Expert-Parallel Communicationcited by 11
  4. 2025Beyond the Buzz: A Pragmatic Take on Inference Disaggregationcited by 9
  5. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 8
  6. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
  7. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  8. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  9. 2025Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUscited by 3
  10. 2026Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methodscited by 3
  11. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 2
  12. 2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Servingcited by 2
8 more
  1. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  2. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1
  3. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  4. 2026Trust-Aware Routing for Distributed Generative AI Inference at the Edgecited by 0
  5. 2026Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregationcited by 0
  6. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0
  7. 2026E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environmentscited by 0
  8. 2026OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024KVDirect: Distributed Disaggregated LLM Inference
  2. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  3. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
  4. 2023Splitwise: Efficient generative LLM inference using phase splitting
  5. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow
  6. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
  7. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  8. 2024Inference Performance Optimization for Large Language Models on CPUs