mlsanityData to 14 Oct 2026

Pie: A Programmable Serving System for Emerging LLM Applications

arXiv 2510.24051 ↗First posted 28 Oct 2025cs.CLcited by 14

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Emerging large language model (LLM) applications involve diverse reasoning strategies and agentic workflows, straining the capabilities of existing serving systems built on a monolithic token generation loop. This paper introduces Pie, a programmable LLM serving system designed for flexibility and efficiency. Pie decomposes the traditional generation loop into fine-grained service handlers exposed via an API and delegates control of the generation process to user-provided programs, called inferlets. This enables applications to implement new KV cache strategies, bespoke generation logic, and seamlessly integrate computation and I/O-entirely within the application, without requiring modifications to the serving system. Pie executes inferlets using WebAssembly, benefiting from its lightweight sandboxing. Our evaluation shows Pie matches state-of-the-art performance on standard tasks (3-12% latency overhead) while significantly improving latency and throughput (1.3x-3.4x higher) on agentic workflows by enabling application-specific optimizations.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 38 of its 80 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

38 of 80 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023An LLM Compiler for Parallel Function Calling7 Dec
  2. 2023LEVER: Learning to Verify Language-to-Code Generation with Execution16 Feb

Other references · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  12. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
24 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  4. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  7. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  8. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  9. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  10. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  12. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  13. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  14. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  15. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 105
  16. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 101
  17. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  18. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  19. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
  20. 2023Stateful Large Language Model Serving with Pensievecited by 61
  21. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  22. 2024Fast State Restoration in LLM Serving with HCachecited by 33
  23. 2023Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Modelscited by 24
  24. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  2. 2026KernelBlaster: Continual Cross-Task CUDA Optimization via Memory-Augmented In-Context Reinforcement Learningcited by 12
  3. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 10
  4. 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusterscited by 6
  5. 2026VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?cited by 4
  6. 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Managementcited by 2
  7. 2026Benchmarking LLM Serving Systems for Agentic AI Workloads with XPerfcited by 1
  8. 2026SwarmX: Agentic Scheduling for Low-Latency Agentic Systemscited by 0
  9. 2026Scalable LLM Agent Tool Access in the Cloudcited by 0
  10. 2026Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMscited by 0
  11. 2026TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructurecited by 0
  12. 2026Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systemscited by 0
2 more
  1. 2026BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inferencecited by 0
  2. 2026PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
  2. 2025Serve Programs, Not Prompts
  3. 2025LLM Applications: Current Paradigms and the Next Frontier
  4. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
  5. 2024Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels
  6. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  7. 2026Characterizing Large Language Model Agentic Workflows: A Study on N8n Ecosystem
  8. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling