mlsanityData to 7 Oct 2026

BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization

arXiv 2602.10729 ↗First posted 11 Feb 2026cs.DCcited by 6

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The rapid growth of large language model (LLM) deployments has made cost-efficient serving systems essential. Recent efforts to enhance system cost-efficiency adopt two main perspectives: (i) An algorithmic perspective that exploits heterogeneous model capabilities to route simpler queries to lower-cost models and complex queries to higher-cost models (i.e., heterogeneous query routing); and (ii) a systems perspective that utilizes heterogeneous GPU resources as cost-effective alternatives to homogeneous high-end GPUs (i.e., heterogeneous model deployment). However, algorithm-system co-design for cost-efficient LLM serving necessitates sophisticated management: (i) Determining optimal query routing strategies under latency and quality requirements, (ii) configuring model deployment across heterogeneous GPUs with appropriate resource allocation and parallelism strategies, and (iii) co-optimizing routing and deployment decisions to maximize overall system performance. To address these challenges, we present BOute, a quality-aware scheduling system that jointly exploits heterogeneous model and GPU capabilities for cost-efficient LLM serving. BOute employs a multi-objective Bayesian optimization (MOBO) framework to co-optimize the routing strategy and model deployment, thereby maximizing the cost-efficiency of the serving system while guaranteeing response quality. Evaluation results demonstrate that BOute outperforms state-of-the-art LLM serving systems by up to 157% and 59% on average under identical cost budgets and quality requirements, or reducing serving costs by 15%-61% (38% on average) while maintaining the same performance targets, validating its effectiveness in achieving cost-efficient LLM serving.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 58 references that are in the map. First posted on arXiv 11 Feb 2026.

Builds on

40 of 58 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment30 Sep
  2. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind21 Aug
  3. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb
  4. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs2 Feb
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  7. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  8. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  8. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  9. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  11. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  12. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
20 more
  1. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  2. 2015High Dimensional Bayesian Optimisation and Bandits via Additive Modelscited by 137
  3. 2021Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvementcited by 115
  4. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  5. 2013Structure Discovery in Nonparametric Regression through Compositional Kernel Searchcited by 86
  6. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 55
  7. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  8. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  9. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  10. 2025TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Timescited by 31
  11. 2025Adaptive LLM Routing under Budget Constraintscited by 21
  12. 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environmentcited by 20
  13. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 19
  14. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 13
  15. 2022OSDP: Optimal Sharded Data Parallel for Distributed Deep Learningcited by 12
  16. 2025Rerouting LLM Routerscited by 11
  17. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 8
  18. 2024HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardwarecited by 7
  19. 2025Thinking Short and Right Over Thinking Long: Serving LLM Reasoning Efficiently and Accuratelycited by 6
  20. 2025Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 2
  2. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1
  3. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 1
  4. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0
  5. 2026TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Servingcited by 0
  6. 2026OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
  2. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  3. 2026ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters
  4. 2025MoLink: Distributed and Efficient Serving Framework for Large Models
  5. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
  6. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines
  7. 2026OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters ↗
  8. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving