mlsanityData to 14 Oct 2026

Parrot: Efficient Serving of LLM-based Applications with Semantic Variable

arXiv 2405.19888 ↗First posted 30 May 2024cs.LG cs.AIcited by 99

Not on the map. Why?

Cited by 46 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rise of large language models (LLMs) has enabled LLM-based applications (a.k.a. AI agents or co-pilots), a new software paradigm that combines the strength of LLM and conventional software. Diverse LLM applications from different tenants could design complex workflows using multiple LLM requests to accomplish one task. However, they have to use the over-simplified request-level API provided by today's public LLM services, losing essential application-level information. Public LLM services have to blindly optimize individual LLM requests, leading to sub-optimal end-to-end performance of LLM applications. This paper introduces Parrot, an LLM service system that focuses on the end-to-end experience of LLM-based applications. Parrot proposes Semantic Variable, a unified abstraction to expose application-level knowledge to public LLM services. A Semantic Variable annotates an input/output variable in the prompt of a request, and creates the data pipeline when connecting multiple LLM requests, providing a natural way to program LLM applications. Exposing Semantic Variables to the public LLM service allows it to perform conventional data flow analysis to uncover the correlation across multiple LLM requests. This correlation opens a brand-new optimization space for the end-to-end performance of LLM-based applications. Extensive evaluations demonstrate that Parrot can achieve up to an order-of-magnitude improvement for popular and practical use cases of LLM applications.

Key facts. Cited by 99 papers in the map (data through 14 Oct 2026). Builds on 17 of its 51 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

17 of 51 references.

References · 17

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  11. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  12. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
5 more
  1. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  2. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  3. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  4. 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 65
  5. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 42; 2026: 46.

99 papers.

Sort:

By area:

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  2. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 109
  3. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 87
  4. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 58
  5. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  6. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  7. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  8. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 47
  9. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  10. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  11. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  12. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
38 more
  1. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 25
  2. 2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 20
  3. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  4. 2025Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platformscited by 18
  5. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  6. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  7. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  8. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  9. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 16
  10. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 14
  11. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 14
  12. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  13. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  14. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  15. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 11
  16. 2026Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scalecited by 11
  17. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  18. 2025GPIoT: Tailoring Small Language Models for IoT Program Synthesis and Developmentcited by 10
  19. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  20. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharingcited by 8
  21. 2025Batch Query Processing and Optimization for Agentic Workflowscited by 6
  22. 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusterscited by 6
  23. 2025Queueing, Predictions, and LLMs: Challenges and Open Problemscited by 5
  24. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inferencecited by 5
  25. 2026DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFScited by 5
  26. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systemscited by 5
  27. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  28. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharingcited by 4
  29. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMscited by 4
  30. 2026Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Managementcited by 4
  31. 2024Ensuring Fair LLM Serving Amid Diverse Applicationscited by 3
  32. 2025PyGraph: Robust Compiler Support for CUDA Graphs in PyTorchcited by 3
  33. 2025FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoningcited by 3
  34. 2025PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMscited by 3
  35. 2025A Plan Reuse Mechanism for LLM-Driven Agentcited by 3
  36. 2026MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systemscited by 3
  37. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  38. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 3

50 most cited of 99 All 99 (JSON)