# HexGen: Generative Inference of Large Language Model over Heterogeneous Environment

arXiv: https://arxiv.org/abs/2311.11514 · HTML: https://mlsanity.com/paper/2311.11514/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2023.

## Built on by (most cited first, 30 of 31)

- [SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration](https://mlsanity.com/paper/2410.02367/) (2024)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments](https://mlsanity.com/paper/2502.09334/) (2025)
- [Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs](https://mlsanity.com/paper/2502.00722/) (2025)
- [Taming the Titans: A Survey of Efficient LLM Inference Serving](https://mlsanity.com/paper/2504.19720/) (2025)
- [Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices](https://mlsanity.com/paper/2410.01805/) (2024)
- [Seesaw: High-throughput LLM Inference via Model Re-sharding](https://mlsanity.com/paper/2503.06433/) (2025)
- [Parallax: Efficient LLM Inference Service over Decentralized Environment](https://mlsanity.com/paper/2509.26182/) (2025)
- [LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind](https://mlsanity.com/paper/2508.15601/) (2025)
- [AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs](https://mlsanity.com/paper/2511.00796/) (2025)
- [FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel](https://mlsanity.com/paper/2508.18224/) (2025)
- [BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2602.10729/) (2026)
- [Cascadia: An Efficient Cascade Serving System for Large Language Models](https://mlsanity.com/paper/2506.04203/) (2025)
- [Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference](https://mlsanity.com/paper/2508.19559/) (2025)
- [Deploying Foundation Model Powered Agent Services: A Survey](https://mlsanity.com/paper/2412.13437/) (2024)
- [ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching](https://mlsanity.com/paper/2509.16857/) (2025)
- [Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods](https://mlsanity.com/paper/2605.05227/) (2026)
- [Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Framework](https://mlsanity.com/paper/2509.26534/) (2025)
- [Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs](https://mlsanity.com/paper/2605.04357/) (2026)
- [Cronus: Efficient LLM inference on Heterogeneous GPU Clusters via Partially Disaggregated Prefill](https://mlsanity.com/paper/2509.17357/) (2025)
- [RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving](https://mlsanity.com/paper/2606.17949/) (2026)
- [ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters](https://mlsanity.com/paper/2606.18600/) (2026)
- [MoLink: Distributed and Efficient Serving Framework for Large Models](https://mlsanity.com/paper/2507.05043/) (2025)
- [BeLLMan: Controlling LLM Congestion](https://mlsanity.com/paper/2510.15330/) (2025)
- [StreamWise: Serving Multi-Modal Generation in Real-Time at Scale](https://mlsanity.com/paper/2603.05800/) (2026)
- [Trust-Aware Routing for Distributed Generative AI Inference at the Edge](https://mlsanity.com/paper/2603.28622/) (2026)
- [Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization](https://mlsanity.com/paper/2604.21072/) (2026)
- [HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling](https://mlsanity.com/paper/2605.16637/) (2026)
- [Multi-Turn Distributed Inference with Mixture of Experts for 6G Edge--Cloud Networks](https://mlsanity.com/paper/2607.02522/) (2026)

## Abstract (arXiv)

Serving generative inference of the large language model is a crucial component of contemporary AI applications. This paper focuses on deploying such services in a heterogeneous and cross-datacenter setting to mitigate the substantial inference costs typically associated with a single centralized datacenter. Towards this end, we propose HexGen, a flexible distributed inference engine that uniquely supports the asymmetric partition of generative inference computations over both tensor model parallelism and pipeline parallelism and allows for effective deployment across diverse GPUs interconnected by a fully heterogeneous network. We further propose a sophisticated scheduling algorithm grounded in constrained optimization that can adaptively assign asymmetric inference computation across the GPUs to fulfill inference requests while maintaining acceptable latency levels. We conduct an extensive evaluation to verify the efficiency of HexGen by serving the state-of-the-art Llama-2 (70B) model. The results suggest that HexGen can choose to achieve up to 2.3 times lower latency deadlines or tolerate up to 4 times more request rates compared with the homogeneous baseline given the same budget.