mlsanityData to 14 Oct 2026

TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure

arXiv 2608.06007 ↗First posted 6 Aug 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Modern LLM infrastructure increasingly manages tensors not only as computation data, but also as persistent states shared across distributed components. Existing systems optimize individual tensor management tasks, such as model weight loading, KV cache management, and checkpoint synchronization, by deeply integrating task-specific mechanisms with execution engines, networks, or storage backends. However, this specialization creates isolated silos that hinder the reuse and composition of tensor management strategies across evolving LLM workloads. In this paper, we identify tensor lifecycle management as a missing abstraction layer in LLM infrastructure and propose Tensor-as-a-Service (TaaS), which decouples tensor state management from computation logic. We design and build TensorCast, a distributed tensor management layer that provides first-class tensor abstractions, programmable lifecycle primitives, and a runtime that separates tensor management policies from execution mechanisms. This enables developers to write tensor management programs using TensorCast APIs while transparently leveraging distributed execution and data movement. We integrate TensorCast with vLLM and SGLang and evaluate it across diverse tensor lifecycle workloads, including model weight materialization, weight synchronization, KV cache management, and programmable request routing. Our results show that TensorCast achieves competitive performance with specialized tensor management systems while enabling new cross-component optimization policies. A programmable policy implemented with TensorCast improves median TTFT by up to 93.2% under highly concurrent multi-turn agent workloads.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 84 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

43 of 84 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling16 Mar
  2. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving6 Feb
  3. 2025Pie: A Programmable Serving System for Emerging LLM Applications28 Oct
  4. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds21 Feb
  5. 2024A System for Microserving of LLMs17 Dec
  6. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  9. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
22 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  2. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  4. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  5. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  7. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  8. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  9. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  10. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  11. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  12. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  13. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  14. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  15. 2023Stateful Large Language Model Serving with Pensievecited by 61
  16. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  17. 2024BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Cachingcited by 25
  18. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 25
  19. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  20. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacentercited by 11
  21. 2025ToTRL: Unlock LLM Tree-of-Thoughts Reasoning Potential through Puzzles Solvingcited by 8
  22. 2025FlexKV: Flexible Index Offloading for Memory-Disaggregated Key-Value Storecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel
  2. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
  3. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving
  4. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  5. 2026TCL: Enabling Fast and Efficient Cross-Hardware Tensor Program Optimization via Continual Learning
  6. 2025AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism
  7. 2026Write Once, Run Everywhere: The Axon DSL for Shape-Safe and Framework-Agnostic LLM Architectures
  8. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices