mlsanityData to 7 Oct 2026

SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

arXiv 2607.08565 ↗First posted 9 Jul 2026cs.DC cs.AIcited by 2

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

LLM scheduling is critical to serving, yet how well existing designs fit agentic serving--where agents, not humans, issue the requests--remains unclear. Agents shift the workload in two ways: they consume many more tokens than humans, so the cluster must provide high throughput (TPS) at low latency; and their requests reuse far more KV than chat. Existing schedulers still trade off load balance against KV reuse: cache-aware schedulers may crowd requests onto the few instances caching the KV, leaving the rest idle, while balanced schedulers may lose the opportunity for reuse, which is costly at a high reuse ratio. We thus present two key insights: (1) with a global-tier KV store, pursuing load balance need not compromise KV reuse, though the slower global tier must be used with care; and (2) given the agent's intra-session locality, routing requests by their sessions can balance the load with high KV reuse. A key challenge in realizing session-centric scheduling is that the scheduler must identify a request's session statelessly, which is difficult for model providers serving arbitrary agents. SMetric addresses this with differential scheduling based on two indicators derived from the request itself, the session turn and the local KV hit: it schedules first-turn requests for load balance, and sticks follow-ups to the instance with the highest local hit for high KV reuse. As sessions differ widely in size, SMetric sticks a follow-up only if the instance can serve it within its SLO, and otherwise migrates the session to the least-loaded instance to prevent many long sessions from eventually imbalancing the load. Evaluated on real-world traces, SMetric improves the peak TPS by 9-15% under prefill-decode colocation with a provisioned global tier and the peak prefill TPS by 9% under disaggregation over state-of-the-art schedulers, also with lower latency.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 50 of its 80 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

50 of 80 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving28 Apr
  2. 2026FM-Agent: Scaling Formal Methods to Large Systems via LLM-Based Hoare-Style Reasoning13 Apr
  3. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb

Other references · 47

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  10. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  11. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  12. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
35 more
  1. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  3. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  4. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  5. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  6. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  7. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  8. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  9. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  10. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  11. 2023Fairness in Serving Large Language Modelscited by 77
  12. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  13. 2023Stateful Large Language Model Serving with Pensievecited by 59
  14. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  15. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  16. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  17. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  18. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 34
  19. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  20. 2026How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Taskscited by 28
  21. 2024BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Cachingcited by 23
  22. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 21
  23. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  24. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domainscited by 14
  25. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10
  26. 2025SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inferencecited by 9
  27. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  28. 2025PolyServe: Efficient Multi-SLO Serving at Scalecited by 5
  29. 2025Aragog: Just-in-Time Model Routing for Scalable Serving of Agentic Workflowscited by 5
  30. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Servingcited by 4
  31. 2025AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Servingcited by 4
  32. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  33. 2026Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inferencecited by 2
  34. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  35. 2025iServe: An Intent-based Serving System for LLMscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reusecited by 0
  2. 2026LAM: Efficient Lossy Agent Memory Framework With A Retrieval-Score Error Boundcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
  2. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents
  3. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
  4. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
  5. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information
  6. 2025Astraea: A State-Aware Scheduling Engine for LLM-Powered Agents
  7. 2026SARA: SLO-Aware Resource Allocation for Disaggregated Agentic LLM Services
  8. 2026A Policy-Driven Runtime Layer for Agentic LLM Serving