mlsanityData to 7 Oct 2026

Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow

arXiv 2406.01566 ↗First posted 3 Jun 2024cs.DC cs.CL cs.LGcited by 54

Not on the map. Why?

Cited by 23 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper introduces Helix, a distributed system for high-throughput, low-latency large language model (LLM) serving in heterogeneous GPU clusters. The key idea behind Helix is to formulate inference computation of LLMs over heterogeneous GPUs and network connections as a max-flow problem on directed, weighted graphs, whose nodes represent GPU instances and edges capture both GPU and network heterogeneity through their capacities. Helix then uses a mixed integer linear programming (MILP) algorithm to discover highly optimized strategies to serve LLMs on heterogeneous GPUs. This approach allows Helix to jointly optimize model placement and request scheduling, two highly entangled tasks in heterogeneous LLM serving. Our evaluation on several heterogeneous clusters ranging from 24 to 42 GPU nodes shows that Helix improves serving throughput by up to 3.3x and reduces prompting and decoding latency by up to 66% and 24%, respectively, compared to existing approaches. Helix is available at https://github.com/Thesys-lab/Helix-ASPLOS25.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 28 of its 61 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

28 of 61 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
16 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 529
  2. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  3. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  4. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  6. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  7. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  8. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  9. 2021Distributed Deep Learning in Open Collaborationscited by 58
  10. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  11. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  12. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  13. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  14. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  15. 2024HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesiscited by 8
  16. 2020Interleaved Weighted Round-Robin: A Network Calculus Analysiscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 22; 2026: 31.

54 papers.

Sort:

By area:

  1. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  2. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 14
  3. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  4. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
  5. 2025Parallax: Efficient LLM Inference Service over Decentralized Environmentcited by 12
  6. 2025UCCL-EP: Portable Expert-Parallel Communicationcited by 11
  7. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacentercited by 11
  8. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devicescited by 9
  9. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 8
  10. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
  11. 2025Agentic Services Computingcited by 6
  12. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfigurationcited by 6
38 more
  1. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  2. 2025DeServe: Towards Affordable Offline LLM Inference via Decentralizationcited by 4
  3. 2025TetriServe: Efficiently Serving Mixed DiT Workloadscited by 4
  4. 2025Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Servingcited by 4
  5. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Servingcited by 4
  6. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  7. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  8. 2025TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Networkcited by 2
  9. 2025Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Frameworkcited by 2
  10. 2026Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Modelcited by 2
  11. 2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Servingcited by 2
  12. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  13. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  14. 2025AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decodingcited by 1
  15. 2025Hardwired-Neurons Language Processing Units as General-Purpose Cognitive Substratescited by 1
  16. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  17. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 1
  18. 2026PALS: Power-Aware LLM Serving for Mixture-of-Experts Modelscited by 1
  19. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reusecited by 1
  20. 2026NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inferencecited by 1
  21. 2026Vortex: Efficient and Programmable Sparse Attention Serving for AI Agentscited by 1
  22. 2026ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusterscited by 1
  23. 2025Harmonia: End-to-End RAG Serving Optimizationcited by 0
  24. 2025MoLink: Distributed and Efficient Serving Framework for Large Modelscited by 0
  25. 2025GOGH: Correlation-Guided Orchestration of GPUs in Heterogeneous Clusterscited by 0
  26. 2025GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inferencecited by 0
  27. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 0
  28. 2026StreamWise: Serving Multi-Modal Generation in Real-Time at Scalecited by 0
  29. 2026Trust-Aware Routing for Distributed Generative AI Inference at the Edgecited by 0
  30. 2026Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Cloudscited by 0
  31. 2026LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflowscited by 0
  32. 2026Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregationcited by 0
  33. 2026Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimizationcited by 0
  34. 2026MARLIN: Multi-Agent Game-Theoretic Reinforcement Learning for Sustainable LLM Inference in Cloud Datacenterscited by 0
  35. 2026Lodestar: An Online-Learning LLM Inference Routercited by 0
  36. 2026Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Servingcited by 0
  37. 2026Integrity of peer-to-peer distributed LLM inference under malicious nodescited by 0
  38. 2026HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimizationcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Characterizing Communication Patterns in Distributed Large Language Model Inference
  2. 2026OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters ↗
  3. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines
  4. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  5. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
  6. 2025RAPID-LLM: Resilience-Aware Performance analysis of Infrastructure for Distributed LLM Training and Inference ↗
  7. 2024Inference Performance Optimization for Large Language Models on CPUs
  8. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains