mlsanityData to 14 Oct 2026

M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity

arXiv 2404.14527 ↗First posted 22 Apr 2024cs.DC cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly integrated into many online services, yet they remain cost-prohibitive to deploy due to the requirement of expensive GPU instances. Prior work has addressed the high cost of LLM serving by improving the inference engine, but less attention has been given to selecting the most cost-efficient GPU type(s) for a specific LLM service. There is a large and growing landscape of GPU types and, within these options, higher cost does not always lead to increased performance. Instead, through a comprehensive investigation, we find that three key LLM service characteristics (request size, request rate, SLO) strongly influence GPU cost efficiency, and differing GPU types are most cost efficient for differing LLM service settings. As a result, the most cost-efficient allocation for a given service is typically a mix of heterogeneous GPU types. Based on this analysis, we introduce M'elange, a GPU allocation framework that navigates these diverse LLM service characteristics and heterogeneous GPU option space to automatically and efficiently derive the minimal-cost GPU allocation for a given LLM service. We formulate the GPU allocation task as a cost-aware bin packing problem where GPUs are bins and items are slices of the service workload. Our formulation's constraints account for a service's unique characteristics, allowing M'elange to be flexible to support diverse service settings and heterogeneity-aware to adapt the GPU allocation to a specific service. Compared to using only a single GPU type, M'elange reduces deployment costs by up to 77% in conversational settings, 33% in document-based settings, and 51% in a mixed setting.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 35 of its 61 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

35 of 61 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
23 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  5. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  8. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  9. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  10. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  11. 2023Speculative Decoding with Big Little Decodercited by 164
  12. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  13. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  14. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  15. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  16. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  17. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  18. 2023Fairness in Serving Large Language Modelscited by 78
  19. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 68
  20. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  21. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 53
  22. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  23. 2020Srifty: Swift and Thrifty Distributed Training on the Cloudcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 11; 2026: 14.

31 papers.

Sort:

By area:

  1. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  2. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  3. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  4. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 22
  5. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  6. 2025AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructurecited by 13
  7. 2025UCCL-EP: Portable Expert-Parallel Communicationcited by 12
  8. 2025The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networkscited by 7
  9. 2025An Inquiry into Datacenter TCO for LLM Inference with FP8cited by 5
  10. 2025Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inferencecited by 5
  11. 2024A Strategy to Combine 1stGen Transformers and Open LLMs for Automatic Text Classificationcited by 3
  12. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 3
19 more
  1. 2026PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Servingcited by 3
  2. 2025Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloadingcited by 2
  3. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 2
  4. 2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Servingcited by 2
  5. 2025iServe: An Intent-based Serving System for LLMscited by 1
  6. 2025GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inferencecited by 1
  7. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  8. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 1
  9. 2026Quantization-Aware Collaborative Inference for Large Embodied AI Modelscited by 1
  10. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  11. 2026When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Rerankingcited by 1
  12. 2024Software Performance Engineering for Foundation Model-Powered Softwarecited by 0
  13. 2024Topology-aware Preemptive Scheduling for Co-located LLM Workloadscited by 0
  14. 2025MoLink: Distributed and Efficient Serving Framework for Large Modelscited by 0
  15. 2026Cost-Optimal Foundation Model Deployment Portfolio for Transportation Managementcited by 0
  16. 2026Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Servingcited by 0
  17. 2026OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Servingcited by 0
  18. 2026EMA: Elastic and Performance Transparent Memory Across GPUscited by 0
  19. 2026FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization
  2. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  3. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  4. 2024Inference Performance Optimization for Large Language Models on CPUs
  5. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  6. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management
  7. 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
  8. 2025SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference