mlsanityData to 7 Oct 2026

InferCept: Efficient Intercept Support for Augmented Large Language Model Inference

arXiv 2402.01869 ↗First posted 2 Feb 2024cs.LG cs.CL cs.DCcited by 32

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models are increasingly integrated with external environments, tools, and agents like ChatGPT plugins to extend their capability beyond language-centric tasks. However, today's LLM inference systems are designed for standalone LLMs. They treat each external interaction as the end of LLM generation and form a new request when the interaction finishes, causing unnecessary recomputation of already computed contexts, which accounts for 37-40% of total model forwarding time. This paper presents InferCept, the first LLM inference framework targeting augmented LLMs and supporting the efficient interception of LLM generation. InferCept minimizes the GPU resource waste caused by LLM interceptions and dedicates saved memory for serving more requests. InferCept improves the overall serving throughput by 1.6x-2x and completes 2x more requests per second compared to the state-of-the-art LLM inference systems.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 50 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

29 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  2. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 27

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
15 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  5. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  9. 2023Tool Learning with Foundation Modelscited by 302
  10. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  11. 2022TALM: Tool Augmented Language Modelscited by 197
  12. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  13. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  14. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  15. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 90

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 10; 2026: 13.

32 papers.

Sort:

By area:

  1. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  2. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  3. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  4. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 21
  5. 2024Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Executioncited by 16
  6. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  7. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  8. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  9. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  10. 2024Fast Inference for Augmented Large Language Modelscited by 10
  11. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10
  12. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
20 more
  1. 2024Alto: Orchestrating Distributed Compound AI Systems with Nested Ancestrycited by 7
  2. 2024BlockLLM: Multi-tenant Finer-grained Serving for Large Language Modelscited by 5
  3. 2025Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agentscited by 5
  4. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  5. 2025Queueing, Predictions, and LLMs: Challenges and Open Problemscited by 4
  6. 2026Diffusion Templates: A Unified Plugin Framework for Controllable Diffusioncited by 3
  7. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  8. 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Managementcited by 2
  9. 2025Efficient Serving of LLM Applications with Probabilistic Demand Modelingcited by 1
  10. 2026AgentServeSim: Serving-System Simulation and Policy Search for LLM Agent Programscited by 1
  11. 2026TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Servingcited by 1
  12. 2025Serve Programs, Not Promptscited by 0
  13. 2026AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agentcited by 0
  14. 2026HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimizationcited by 0
  15. 2026Second Thought: Reasoning in Parallel as LLM Agents Act and Observecited by 0
  16. 2026Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloadscited by 0
  17. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0
  18. 2026Position: It is Time to Virtualize Foundation Models with a Self-evolving Operating System Layercited by 0
  19. 2026EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?cited by 0
  20. 2026Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  2. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  3. 2025Glinthawk: A Two-Tiered Architecture for Offline LLM Inference
  4. 2024Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models
  5. 2023Efficient LLM Inference on CPUs
  6. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  7. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
  8. 2025Seesaw: High-throughput LLM Inference via Model Re-sharding