mlsanityData to 14 Oct 2026

Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs

arXiv 2607.17181 ↗First posted 19 Jul 2026cs.DC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Serverless multi-model LLM systems multiplex popularity-skewed model catalogs over shared GPU pools, yet typically schedule each request independently. Tool-using agents break this abstraction: a session repeatedly calls an LLM across short tool gaps, carries a long reusable KV prefix, and is judged by session completion time (SCT). Load-only routing can separate a continuation from both its model and KV state, while round-based model multiplexing can delay even a correctly placed continuation until the target model's next slot. Both failures are especially costly for hundred-billion-parameter models: their weights constrain residency, while long-context KV is expensive to reconstruct or move. We present Talaria, a session-aware serverless multi-model serving system that makes session continuity a joint placement-and-admission decision. Its router ranks placements by model residency, KV locality, and instance pressure, while soft reservations account for likely returns in the last serving instance's admission budget. Session-prefill (SP) admits budget-eligible continuations before the active model slot closes. An instance-local substrate keeps HBM addresses stable, preserves host-restorable KV, and stages weights across model switches. On a single TP=8 server, we replay 30 SWE-Bench model-sessions (960 calls) over three models, each with more than 100B total parameters. Against an otherwise identical round scheduler with SP, host-KV restoration, and D2D staging disabled, Talaria cuts p50 SCT from 1000 s to 189 s and p95 from 2296 s to 867 s, speedups of 5.3x and 2.6x.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 19 Jul 2026.

Builds on

32 of 48 references.

References · 32

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  12. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
20 more
  1. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  3. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  4. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  7. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  8. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  9. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  10. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  11. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  12. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  13. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  14. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  15. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  16. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 73
  17. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  18. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 66
  19. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 14
  20. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8

Built on by

Not cited yet.