mlsanityData to 14 Oct 2026

SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving

arXiv 2609.37626 ↗First posted 29 Sep 2026cs.DC cs.AIcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling also fits: KV cache optimization

Abstract (arXiv)

No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independent requests favor data-parallel attention, and long prompts favor context parallelism. Reasoning, agentic, and RL-rollout workloads make a fixed choice untenable: a batch that begins as many short requests ends as a few very long ones, so the best layout changes while the same requests run. Serving engines nevertheless fix one layout at launch, because changing it has meant draining requests and restarting workers. We present SPLASH, a serving system that switches the parallel layout of attention while requests are running. It builds on one observation: modern attention, with few or no KV heads, decouples where a request's KV cache lives from how attention weights are sharded. This has two consequences. First, layouts differ only in who owns the weights and the cache, and most of that state already sits where the next layout needs it; SPLASH reuses it, moves the rest in the background of ongoing inference, and hands off at a batch boundary, making a switch nearly free: its median overhead is under 0.51% of the step it runs in. Second, the decoupling exposes a layout that existing engines lack: Decoupled Ownership Parallelism (DOP) shards attention weights as tensor parallelism does while keeping each request's cache on a single owner as data-parallel attention does. DOP replicates neither, offers 27-60% more KV capacity than data-parallel attention, and gives the scheduler a choice when KV memory limits admission. A transition-aware scheduler follows the best of the four layouts as load changes. On B200 GPUs serving GLM-5.3, SPLASH improves end-to-end serving throughput by 1.3-1.73x over fixed-layout deployments, and the same layout regimes appear with DeepSeek-V3.2 on H200 and GLM-5.3-Flash on DCU.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 44 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

30 of 44 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving17 Jun
  2. 2026PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving14 Apr
  3. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving26 Feb
  4. 2025Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads20 Sep
  5. 2025Seesaw: High-throughput LLM Inference via Model Re-sharding9 Mar
  6. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 22

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  8. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
10 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 539
  2. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  3. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  5. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
  6. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  7. 2026Agentic AI Workload Characteristicscited by 4
  8. 2025Amoeba: Runtime Tensor Parallel Transformation for LLM Inference Servicescited by 2
  9. 2026Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelismcited by 1
  10. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switchcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  2. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation
  3. 2024Marconi: Prefix Caching for the Era of Hybrid LLMs
  4. 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching
  5. 2025Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
  6. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  7. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
  8. 2026S-HPLB: Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance