mlsanityData to 14 Oct 2026

Teola: Towards End-to-End Optimization of LLM-based Applications

arXiv 2407.00326 ↗First posted 29 Jun 2024cs.DC cs.AI cs.NIcited by 32

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM)-based applications consist of both LLM and non-LLM components, each contributing to the end-to-end latency. Despite great efforts to optimize LLM inference, end-to-end workflow optimization has been overlooked. Existing frameworks employ coarse-grained orchestration with task modules, which confines optimizations to within each module and yields suboptimal scheduling decisions. We propose fine-grained end-to-end orchestration, which utilizes task primitives as the basic units and represents each query's workflow as a primitive-level dataflow graph. This explicitly exposes a much larger design space, enables optimizations in parallelization and pipelining across primitives of different modules, and enhances scheduling to improve application-level performance. We build Teola, a novel orchestration framework for LLM-based applications that implements this scheme. Comprehensive experiments show that Teola can achieve up to 2.09x speedup over existing systems across various popular LLM applications. The code is available at https://github.com/NetX-lab/Ayo.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 42 of its 70 references that are in the map. First posted on arXiv 29 Jun 2024.

Builds on

42 of 70 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  10. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
30 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  2. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  4. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  6. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  8. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  9. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  10. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  11. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  12. 2023An LLM Compiler for Parallel Function Callingcited by 179
  13. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  14. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  15. 2023Online Speculative Decodingcited by 113
  16. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  17. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  18. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  19. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  20. 2023Fairness in Serving Large Language Modelscited by 78
  21. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  22. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  23. 2023Query Expansion by Prompting Large Language Modelscited by 57
  24. 2024RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedbackcited by 46
  25. 2024Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMscited by 34
  26. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decodingcited by 23
  27. 2023On Optimal Caching and Model Multiplexing for Large Model Inferencecited by 20
  28. 2021Llama: A Heterogeneous & Serverless Framework for Auto-Tuning Video Analytics Pipelinescited by 16
  29. 2022Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructurescited by 12
  30. 2024Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 18.

32 papers.

Sort:

By area:

  1. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  2. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 39
  3. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  4. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  5. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  6. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 14
  7. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  8. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 11
  9. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusterscited by 9
  10. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  11. 2025Batch Query Processing and Optimization for Agentic Workflowscited by 6
  12. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
20 more
  1. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systemscited by 5
  2. 2025ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inferencecited by 4
  3. 2025Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspectivecited by 4
  4. 2026Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Managementcited by 4
  5. 2026MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systemscited by 3
  6. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  7. 2025OET: Optimization-based prompt injection Evaluation Toolkitcited by 1
  8. 2025Rethinking Query Optimization for Multi-Agent Systems [Vision]cited by 1
  9. 2026Nalar: An agent serving frameworkcited by 1
  10. 2026SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Servingcited by 1
  11. 2026Benchmarking LLM Serving Systems for Agentic AI Workloads with XPerfcited by 1
  12. 2025Harmonia: End-to-End RAG Serving Optimizationcited by 0
  13. 2026Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKITcited by 0
  14. 2026LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflowscited by 0
  15. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
  16. 2026Scalable LLM Agent Tool Access in the Cloudcited by 0
  17. 2026Workload-Aware Caching for Multi-Agent Systemscited by 0
  18. 2026HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimizationcited by 0
  19. 2026PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Servingcited by 0
  20. 2026DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Servingcited by 0