mlsanityData to 14 Oct 2026

CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control

arXiv 2601.22705 ↗First posted 30 Jan 2026cs.DCcited by 14

On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

Batch inference for agentic workloads stresses the GPU key-value (KV) cache in a sustained and cumulative manner, often causing severe throughput degradation well before memory capacity is exhausted. We identify this phenomenon as middle-phase thrashing, a previously under-characterized pathology in which cache efficiency collapses as long-lived agents accumulate state over time. We argue that mitigating this pathology requires moving beyond reactive, request-level cache management to proactive, agent-level admission control. Drawing inspiration from congestion control in distributed systems, we view the KV cache as a shared resource whose efficient utilization depends on feedback-driven regulation. Based on this insight, we present CONCUR, a lightweight control layer that regulates agent admission to bound aggregate cache pressure while preserving execution continuity. CONCUR adapts a cache-aware control algorithm to dynamically adjust the number of active agents using runtime cache signals. Across large models and real-world agent workloads, CONCUR prevents middle-phase thrashing and improves batch inference throughput by up to 4.09x on Qwen3-32B and 1.9x on DeepSeek-V3, while remaining compatible with existing LLM serving systems.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 32 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

24 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live4 Nov
  2. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications21 Oct
  3. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb

Other references · 21

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2024AFlow: Automating Agentic Workflow Generationcited by 432
  5. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  6. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 212
  7. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  8. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  9. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  10. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
  11. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 73
  12. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
9 more
  1. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  2. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  3. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  4. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
  5. 2025Nex-N1: Agentic Models Trained via a Unified Ecosystem for Large-Scale Environment Constructioncited by 17
  6. 2025Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloudcited by 11
  7. 2024InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Shardingcited by 8
  8. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  9. 2025SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloadingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 20
  2. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferencecited by 15
  3. 2026Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Managementcited by 4
  4. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Schedulingcited by 4
  5. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  6. 2026WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learningcited by 3
  7. 2026TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AIcited by 1
  8. 2026Noesis: Bidirectional Graph-RAG with Adaptive Parallelism and Cross-Knowledge-Base Semantic Discoverycited by 1
  9. 2026KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decodingcited by 0
  10. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
  11. 2026Dynamic HBM Repartitioning for Multi-Turn MoE Servingcited by 0
  12. 2026Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Servingcited by 0
2 more
  1. 2026Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Servingcited by 0
  2. 2026S3R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands
  2. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
  3. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
  4. 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters
  5. 2026ActKV: Efficient LLM Agents through Action-Guided KV Cache Management
  6. 2026A Policy-Driven Runtime Layer for Agentic LLM Serving
  7. 2026Workload-Aware Caching for Multi-Agent Systems
  8. 2024Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies