mlsanityData to 14 Oct 2026

Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration

arXiv 2505.17787 ↗First posted 23 May 2025cs.ARcited by 2

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Large language models (LLMs) have gained great success in various domains. Existing systems cache Key and Value within the attention block to avoid redundant computations. However, the size of key-value cache (KV cache) is unpredictable and can even be tens of times larger than the weights in the long context length scenario. In this work, we propose Titanus, a software-hardware co-design to efficiently compress the KV cache on-the-fly. We first propose the cascade pruning-quantization (CPQ) method to reduce the KV cache movement. The hierarchical quantization extension strategy is introduced to tackle the non-independent per-channel quantization issue. To further reduce KV cache movement, we transfer only the non-zero KV cache between the accelerator and off-chip memory. Moreover, we customize a two-stage design space exploration framework for the CPQ method. A novel pipeline and parallelism dataflow is designed to reduce the first token generation time. Experiments show that Titanus achieves 159.9x (49.6x) and 34.8x (29.2x) energy efficiency (throughput) compared to Nvidia A100 GPU and FlightLLM respectively. The code for Titanus is available at https://github.com/peilin-chen/Titanus-for-LLM-acceleration.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 40 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

17 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 15

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  6. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  7. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  8. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 101
  9. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  10. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
  11. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 64
  12. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
3 more
  1. 2024KV Prediction for Improved Time to First Tokencited by 11
  2. 2021HERO: Hessian-Enhanced Robust Optimization for Unifying and Improving Generalization and Quantization Performancecited by 8
  3. 2024Addition is All You Need for Energy-efficient Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025End-to-End Transformer Acceleration Through Processing-in-Memory Architecturescited by 1
  2. 2026HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
  2. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  3. 2026InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models
  4. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  5. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations
  6. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving
  7. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
  8. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs