mlsanityData to 7 Oct 2026

Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs

arXiv 2605.04357 ↗First posted 5 May 2026cs.DC cs.AI cs.CLcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The usage of large language models (LLMs) has grown increasingly fragmented, with no single model dominating. Meanwhile, cloud providers offer a wide range of mid-tier and older-generation GPUs that enjoy better availability and deliver comparable performance per dollar to top-tier hardware. To efficiently harness these heterogeneous resources for serving multiple LLMs concurrently, we introduce Coral, an adaptive heterogeneity-aware multi-LLM serving system. The key idea behind Coral is to jointly optimize resource allocation and the serving strategy of each model replica across all models. To keep pace with shifting throughput demand and resource availability, Coral applies a lossless two-stage decomposition that preserves joint optimality while cutting online solve time from hours to tens of seconds. Our evaluation across 6 models and 20 GPU configurations shows that Coral reduces serving cost by up to 2.79× over the best baseline, and delivers up to 2.39× higher goodput under scarce resource availability.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 53 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

35 of 53 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization11 Feb
  2. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters13 Oct
  3. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning6 May
  4. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs2 Feb
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  6. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  8. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2024Mixtral of Expertscited by 1,785
15 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2025OpenAI GPT-5 System Cardcited by 910
  6. 2024Phi-4 Technical Reportcited by 675
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  9. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  10. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  11. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  12. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  13. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  14. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 19
  15. 2025Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusterscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments
  2. 2026ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters
  3. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  4. 2026OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters ↗
  5. 2025MoLink: Distributed and Efficient Serving Framework for Large Models
  6. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
  7. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  8. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures