mlsanityData to 14 Oct 2026

HexGen: Generative Inference of Large Language Model over Heterogeneous Environment

arXiv 2311.11514 ↗First posted 20 Nov 2023cs.DCcited by 31

Not on the map. Why?

Abstract (arXiv)

Serving generative inference of the large language model is a crucial component of contemporary AI applications. This paper focuses on deploying such services in a heterogeneous and cross-datacenter setting to mitigate the substantial inference costs typically associated with a single centralized datacenter. Towards this end, we propose HexGen, a flexible distributed inference engine that uniquely supports the asymmetric partition of generative inference computations over both tensor model parallelism and pipeline parallelism and allows for effective deployment across diverse GPUs interconnected by a fully heterogeneous network. We further propose a sophisticated scheduling algorithm grounded in constrained optimization that can adaptively assign asymmetric inference computation across the GPUs to fulfill inference requests while maintaining acceptable latency levels. We conduct an extensive evaluation to verify the efficiency of HexGen by serving the state-of-the-art Llama-2 (70B) model. The results suggest that HexGen can choose to achieve up to 2.3 times lower latency deadlines or tolerate up to 4 times more request rates compared with the homogeneous baseline given the same budget.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Nov 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 15; 2026: 11.

31 papers.

Sort:

By area:

  1. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  2. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  3. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  4. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  5. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 22
  6. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  7. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  8. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  9. 2025Parallax: Efficient LLM Inference Service over Decentralized Environmentcited by 12
  10. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMindcited by 10
  11. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 9
  12. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
19 more
  1. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  2. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  3. 2025Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inferencecited by 5
  4. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  5. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  6. 2026Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methodscited by 3
  7. 2025Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Frameworkcited by 2
  8. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 2
  9. 2025Cronus: Efficient LLM inference on Heterogeneous GPU Clusters via Partially Disaggregated Prefillcited by 1
  10. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  11. 2026ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusterscited by 1
  12. 2025MoLink: Distributed and Efficient Serving Framework for Large Modelscited by 0
  13. 2025BeLLMan: Controlling LLM Congestioncited by 0
  14. 2026StreamWise: Serving Multi-Modal Generation in Real-Time at Scalecited by 0
  15. 2026Trust-Aware Routing for Distributed Generative AI Inference at the Edgecited by 0
  16. 2026Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimizationcited by 0
  17. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0
  18. 2026Multi-Turn Distributed Inference with Mixture of Experts for 6G Edge--Cloud Networkscited by 0
  19. 2026OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Servingcited by 0