mlsanityData to 14 Oct 2026

Efficient Multi-Model Orchestration for Self-Hosted Large Language Models

arXiv 2512.22402 ↗First posted 26 Dec 2025cs.DC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-hosting large language models (LLMs) is increasingly appealing for organizations seeking privacy, cost control, and customization. Yet deploying and maintaining in-house models poses challenges in GPU utilization, workload routing, and reliability. We introduce Pick and Spin, a practical framework that makes self-hosted LLM orchestration scalable and economical. Built on Kubernetes, it integrates a unified Helm-based deployment system, adaptive scale-to-zero automation, and a hybrid routing module that balances cost, latency, and accuracy using both keyword heuristics and a lightweight DistilBERT classifier. We evaluate four models, Llama-3 (90B), Gemma-3 (27B), Qwen-3 (235B), and DeepSeek-R1 (685B) across eight public benchmark datasets, with five inference strategies, and two routing variants encompassing 31,019 prompts and 163,720 inference runs. Pick and Spin achieves up to 21.6% higher success rates, 30% lower latency, and 33% lower GPU cost per query compared with static deployments of the same models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 29 references that are in the map. First posted on arXiv 26 Dec 2025.

Builds on

16 of 29 references.

References · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  12. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
4 more
  1. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  2. 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
  3. 2025Preserving Privacy and Utility in LLM-Based Product Recommendationscited by 5
  4. 2018InferLine: ML Prediction Pipeline Provisioning and Management for Tight Latency Objectivescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
  2. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  3. 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
  4. 2024PickLLM: Context-Aware RL-Assisted Large Language Model Routing
  5. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  6. 2026A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
  7. 2024Performance Characterization of Expert Router for Scalable LLM Inference
  8. 2026TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition