mlsanityData to 7 Oct 2026

OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration

arXiv 2602.12151 ↗First posted 12 Feb 2026cs.DCcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Serving Large Language Models (LLMs) can benefit immensely from parallelizing both the model and input requests across multiple devices, but incoming workloads exhibit substantial spatial and temporal heterogeneity. Spatially, workloads comprise heterogeneous requests with varying compute and memory demands. Temporally, workload composition varies over time. Nevertheless, existing systems typically assume spatially uniform and temporally stable workloads, employing a homogeneous, static model deployment. This mismatch between the assumption and real-world spatial-temporal heterogeneity results in suboptimal performance. We present OServe, an LLM serving system with heterogeneous and flexible model deployment that addresses both spatial and temporal heterogeneity. First, OServe introduces a novel workload-aware scheduling algorithm that optimizes heterogeneous model deployments according to real-time workload characteristics. Second, OServe proposes an efficient workload-adaptive switching method that migrates model deployments in response to predicted workload changes. Experiments on real-world traces show that OServe improves performance by up to 2× (average: 1.5×) compared to state-of-the-art serving systems.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 50 of its 83 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

50 of 83 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization11 Feb
  2. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment30 Sep
  3. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind21 Aug
  4. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb
  5. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs2 Feb
  6. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving26 Nov
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  10. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems23 Dec
  11. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  12. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2024Mixtral of Expertscited by 1,785
  11. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
26 more
  1. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  3. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  4. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  5. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  6. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  7. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  8. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  9. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  10. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  11. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  12. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  13. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  14. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  15. 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environmentcited by 20
  16. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  17. 2024Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instancescited by 16
  18. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 13
  19. 2022OSDP: Optimal Sharded Data Parallel for Distributed Deep Learningcited by 12
  20. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 8
  21. 2021Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decaycited by 7
  22. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
  23. 2025Thinking Short and Right Over Thinking Long: Serving LLM Reasoning Efficiently and Accuratelycited by 6
  24. 2025Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUscited by 6
  25. 2024Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMscited by 4
  26. 2024Revisiting the Time Cost Model of AllReducecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1
  2. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads
  2. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  3. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing
  4. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling
  5. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  6. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  7. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure
  8. 2026CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving