mlsanityData to 7 Oct 2026

FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework

arXiv 2503.08461 ↗First posted 11 Mar 2025cs.MM cs.DCcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Multi-modal Large Language Models (MLLMs) serving systems commonly employ KV-cache compression to reduce memory footprint. However, existing compression methods introduce significant processing overhead and queuing delays, particularly in concurrent serving scenarios. We present textttFastCache, a novel serving framework that effectively addresses these challenges through two key innovations: (1) a dynamic batching strategy that optimizes request scheduling across prefill, compression, and decode stages, and (2) an efficient KV-cache memory pool mechanism that eliminates memory fragmentation while maintaining high GPU utilization. Our comprehensive experiments on the GQA and MileBench datasets demonstrate that textttFastCache achieves up to 19.3× reduction in Time-To-First-Token (TTFT) and 12.1× improvement in throughput compared to state-of-the-art baselines. The system maintains stable performance under high-concurrency scenarios (up to 40 req/s) while reducing average memory consumption by 20%. These results establish textttFastCache as an efficient solution for real-world LLM serving systems with KV-cache compression.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 41 of its 53 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

41 of 53 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving22 Jul
  2. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  4. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun
  5. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  6. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  8. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  9. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  10. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
1 more
  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 28

  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
16 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  2. 2022Efficiently Scaling Transformer Inferencecited by 529
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  4. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  6. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  7. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  8. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  9. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  10. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  11. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  12. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  13. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  14. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15
  15. 2024Optimised Grouped-Query Attention Mechanism for Transformerscited by 11
  16. 2024Weighted Grouped Query Attention in Transformerscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 9
  2. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  3. 2025SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  2. 2025AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
  3. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  4. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2026MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
  7. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  8. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing