mlsanityData to 13 Oct 2026

KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decoding

arXiv 2606.29207 ↗First posted 28 Jun 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

LLM serving is increasingly dominated by long and dynamic decode workloads from agents, reasoning models, and extended conversations. When bursty long-context demand exceeds deployed capacity, existing serving systems typically scale out by launching additional serving instances with model replicas. This instance-level elasticity increases KV capacity only by provisioning another full copy of the model, inheriting startup latency, memory overhead, and batch fragmentation. We present KernelFlume, a decode-centric architecture that disaggregates the stable projection/FFN path from core-attention computation: weight nodes execute dense projection/FFN kernels, while weightless attention nodes store token-range KV partitions and scale with request-state demand. To make this separation elastic, KernelFlume maintains a routing table that maps token ranges to attention-node endpoints. It updates routes at token boundaries and uses host-visible graph signals to drive pre-registered UCX endpoint communication outside the captured CUDA Graph. To preserve low per-token latency after disaggregation, KernelFlume combines query-first core-attention dispatch with inter-layer kernel pipelining, overlapping remote attention and communication with local projection/FFN work. On real GPU testbeds (intra-node A6000 and cross-node H100), under a dynamic long-context agentic workload serving Llama-3.1-8B, KernelFlume sustains flat p99 TPOTs of ~74 ms on A6000 and ~34 ms on H100, while lowering cost per million output tokens by up to 32% and 61%, respectively, relative to full-instance elastic scaling with ServerlessLLM, a state-of-the-art instance-startup method. Replaying the same trace at larger model scale in simulation projects a 56--66% cost reduction over ServerlessLLM, widening to 80--85% with cheaper heterogeneous attention-node hardware and persisting into the million-token context range.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 36 references that are in the map. First posted on arXiv 28 Jun 2026.

Builds on

17 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 15

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 545
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 514
  5. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 452
  6. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 194
  7. 2018Online normalizer calculation for softmaxcited by 159
  8. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  9. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  10. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  11. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
  12. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
3 more
  1. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferencecited by 15
  2. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13
  3. 2025Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inferencecited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
  2. 2026CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference
  3. 2026SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
  4. 2026PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs
  5. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch
  6. 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
  7. 2026Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding
  8. 2026KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU