Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

arXiv 2603.04428 ↗First posted 17 Feb 2026cs.LG cs.AIcited by 8

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more

Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without persistence, every eviction forces a full re-prefill through the model -- 15.7 seconds per agent at 4K context. We address this by persisting each agent's KV cache to disk in 4-bit quantized format and reloading it directly into the attention layer, eliminating redundant O(n) prefill computation via direct cache restoration. The system comprises three components: a block pool providing per-agent isolated Q4 KV caches in safetensors format, a BatchQuantizedKVCache for concurrent inference over multiple agents' quantized caches, and cross-phase context injection that accumulates attention state across conversation phases without re-computation. Evaluated on three architectures (Gemma 3 12B, dense GQA, 48 layers; DeepSeek-Coder-V2-Lite 16B, MoE MLA, 27 layers; Llama 3.1 8B, dense GQA, 32 layers), cache restoration reduces time-to-first-token by up to 136x (Gemma: 22--136x at 4K--32K; DeepSeek: 11--76x at 4K--32K; Llama: 24--111x at 4K--16K; 3--10x at 1K). Q4 quantization fits 4x more agent contexts into fixed device memory than FP16. Perplexity measured with actual Q4 KV caches shows -0.7% for Gemma, +2.8% for Llama, and +3.0% for DeepSeek. Open-source at https://github.com/yshk-mxim/agent-memory

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 29 of its 35 references that are in the map. First posted on arXiv 17 Feb 2026.

Builds on

29 of 35 references.

Closest ancestors · 4

Same direction, newest first

Other references · 25

2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507

13 more

2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
2025SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planningcited by 58
2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 25
2025Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computingcited by 5
2025Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPScited by 5
2025KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inferencecited by 5
2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
2026Native LLM and MLLM Inference at Scale on Apple Siliconcited by 5
2025Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

2026Forget to Improve: On-Device LLM-Agent Continual Learning via Budget-Curated Memorycited by 8
2026Beyond tokens: a unified framework for latent communication in LLM-based multi-agent systemscited by 6
2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
2026PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inferencecited by 3
2026Knowledge Packs: Zero-Token Knowledge Delivery via KV Cache Injectioncited by 2
2026Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloadscited by 0
2026BIO-MEMART: Biometric-Aware KV Cache Memory for Multi-User LLM Agentscited by 0
2026JustFit: Just-in-Time State Management for Local LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way