mlsanityData to 14 Oct 2026

JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs

arXiv 2610.19822 ↗First posted 14 Oct 2026cs.DC cs.LGcited by 0

On the map: LLM agents and tool use → Agent memory systems also fits: General LLM agent frameworks and benchmarks

Abstract (arXiv)

LLM agents are evolving from a single model calling tools to planners that orchestrate multiple expert models whose capabilities live in distinct weights. This shift breaks a hidden assumption of today's serving systems: that the working set of models fits in GPU memory. On the memory-constrained GPUs typical of private and edge deployments, naive whole-model swapping serializes weight transfer with computation and, without host-backed KV restoration, discards session state at a switch. We present JointServe, a serving system that treats weight shards and KV blocks as residency objects competing for one unified memory budget. JointServe exploits three structural properties of agent workloads (directed calls, gapped execution, and dual-state residency) to prepare models before they are called: an intent signal parsed from the caller's own output stream triggers prefetch during generation, partial k* residency reduces switching to incremental layers, and a preemptive transfer scheduler protects live-session KV with a guaranteed bandwidth share. A carve-out control step adjusts the weight/KV allocation as workload demands change. Evaluations show that JointServe reduces session completion time by 21.7% over the strongest whole-model swap baseline and stays usable in a crash regime where every whole-model swap baseline fails to finish (a 46-fold gap). On replayed SWE-bench, tau-bench, and OSWorld traces, it cuts session completion time by up to 49% and lifts SLO attainment from 2.8-24.5% to 76.8-100%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 45 of its 56 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

45 of 56 references.

References · 45

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  8. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  11. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  12. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
33 more
  1. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  7. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  8. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  12. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  13. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  14. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  15. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  16. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  17. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  18. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  19. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  20. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  21. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  22. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  23. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 178
  24. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  25. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  26. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  27. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  28. 2023Stateful Large Language Model Serving with Pensievecited by 61
  29. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  30. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 52
  31. 2024Fast State Restoration in LLM Serving with HCachecited by 33
  32. 2024BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Cachingcited by 25
  33. 2023DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMscited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
  2. 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management
  3. 2026LOCAL: Enabling Learning On-device Contiguously for Agent LLMs
  4. 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters
  5. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
  6. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
  7. 2026A Policy-Driven Runtime Layer for Agentic LLM Serving
  8. 2026INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration