mlsanityData to 7 Oct 2026

Serve Programs, Not Prompts

arXiv 2510.25412 ↗First posted 29 Oct 2025cs.CLcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Current large language model (LLM) serving systems, primarily designed for text completion, are neither efficient nor adaptable for increasingly complex LLM applications due to their inflexible design. We propose a new LLM serving system architecture that serves programs instead of prompts to address this problem. These programs, called LLM Inference Programs (LIPs), allow users to customize token prediction and KV cache management at runtime and to offload parts of their application logic, such as tool execution, to the server. We describe an example of this architecture through a system named Symphony, which functions as an operating system for LIPs. Symphony exposes LLM model computations via system calls and virtualizes KV cache with a dedicated file system, while ensuring GPU efficiency with a two-level process scheduling scheme. Symphony has the potential to open the door to a more efficient and extensible ecosystem for LLM applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 50 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

36 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inference2 Feb

Other references · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
23 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2023A Watermark for Large Language Modelscited by 667
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  7. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  8. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  9. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  10. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
  11. 2023An LLM Compiler for Parallel Function Callingcited by 170
  12. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  13. 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
  14. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 99
  15. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  16. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  17. 2022Validating Large Language Models with ReLMcited by 33
  18. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  19. 2023Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Modelscited by 24
  20. 2023TypeFly: Flying Drones with Large Language Modelcited by 23
  21. 2013Programming Protocol-Independent Packet Processorscited by 21
  22. 2024Asynchronous LLM Function Callingcited by 18
  23. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  2. 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  4. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  5. 2025Pie: A Programmable Serving System for Emerging LLM Applications
  6. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems
  7. 2024A System for Microserving of LLMs
  8. 2025iServe: An Intent-based Serving System for LLMs