mlsanityData to 14 Oct 2026

FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving

arXiv 2411.18424 ↗First posted 27 Nov 2024cs.LG cs.DCcited by 5

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Serving numerous users and requests concurrently requires good fairness in Large Language Models (LLMs) serving system. This ensures that, at the same cost, the system can meet the Service Level Objectives (SLOs) of more users , such as time to first token (TTFT) and time between tokens (TBT), rather than allowing a few users to experience performance far exceeding the SLOs. To achieve better fairness, the preemption-based scheduling policy dynamically adjusts the priority of each request to maintain balance during runtime. However, existing systems tend to overly prioritize throughput, overlooking the overhead caused by preemption-induced context switching, which is crucial for maintaining fairness through priority adjustments. In this work, we identify three main challenges that result in this overhead. 1) Inadequate I/O utilization. 2) GPU idleness. 3) Unnecessary I/O transmission during multi-turn conversations. Our key insight is that the block-based KV cache memory policy in existing systems, while achieving near-zero memory waste, leads to discontinuity and insufficient granularity in the KV cache memory. To respond, we introduce FastSwitch, a fairness-aware serving system that not only aligns with existing KV cache memory allocation policy but also mitigates context switching overhead. Our evaluation shows that FastSwitch outperforms the state-of-the-art LLM serving system vLLM with speedups of 1.4-11.2x across different tail TTFT and TBT.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 37 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

25 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Queue management for slo-oriented large language model serving5 Jun
  2. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  5. 2023Fairness in Serving Large Language Models31 Dec
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  7. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2018Self-Attention with Relative Position Representationscited by 1,120
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2024Large Language Models: A Surveycited by 525
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
6 more
  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  2. 2023Generating Images with Multimodal Language Modelscited by 260
  3. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  4. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  5. 2024From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Modelscited by 43
  6. 2020Understanding Power Consumption and Reliability of High-Bandwidth Memory with Voltage Underscalingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inferencecited by 6
  3. 2025Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inferencecited by 1
  4. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemptioncited by 0
  5. 2026Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving
  2. 2026Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving
  3. 2026Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling
  4. 2025Mitigating KV Cache Competition to Enhance User Experience in LLM Inference
  5. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Serving
  6. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  8. 2025Locality-aware Fair Scheduling in LLM Serving