mlsanityData to 7 Oct 2026

ThinK: Thinner Key Cache by Query-Driven Pruning

arXiv 2407.21018 ↗First posted 30 Jul 2024cs.CL cs.AIcited by 66

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications. However, their increased computational and memory demands present significant challenges, especially when handling long sequences. This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference. Unlike existing approaches that optimize the memory based on the sequence length, we identify substantial redundancy in the channel dimension of the KV cache, as indicated by an uneven magnitude distribution and a low-rank structure in the attention weights. In response, we propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels. Our approach not only maintains or enhances model accuracy but also achieves a reduction in KV cache memory costs by over 20% compared with vanilla KV cache eviction and quantization methods. For instance, ThinK integrated with KIVI can achieve a 2.8x reduction in peak memory usage while maintaining nearly the same quality, enabling up to a 5x increase in batch size when using a single GPU. Extensive evaluations on the LLaMA and Mistral models across various long-sequence datasets verified the efficiency of ThinK, establishing a new baseline algorithm for efficient LLM deployment without compromising performance. Our code has been made available at https://github.com/SalesforceAIResearch/ThinK.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 57 of its 66 references that are in the map. First posted on arXiv 30 Jul 2024.

Builds on

57 of 66 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2023Let's Verify Step by Stepcited by 3,598
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2023Mistral 7Bcited by 3,065
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  13. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  14. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  15. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  16. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  17. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  18. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  19. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  20. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  21. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  22. 2020What is the State of Neural Network Pruning?cited by 578
  23. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  24. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  25. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  26. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  27. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  28. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  29. 2023Efficient Large Language Models: A Surveycited by 183
  30. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  31. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 155
  32. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  33. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  34. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  35. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  36. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  37. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  38. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 32; 2026: 25.

66 papers.

Sort:

By area:

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  2. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  3. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  4. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
  5. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 54
  6. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  7. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  8. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
  9. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 38
  10. 2025Scalable Chain of Thoughts via Elastic Reasoningcited by 32
  11. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
  12. 2025Cognitive Memory in Large Language Modelscited by 28
38 more
  1. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  2. 2024SCOPE: Optimizing Key-Value Cache Compression in Long-context Generationcited by 20
  3. 2025The Pitfalls of KV Cache Compressioncited by 15
  4. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  5. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Evictioncited by 14
  6. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Headcited by 12
  7. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 12
  8. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMscited by 8
  9. 2025CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clusteringcited by 8
  10. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  11. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  12. 2025FreqKV: Key-Value Compression in Frequency Domain for Context Window Extensioncited by 7
  13. 2025Fractured Chain-of-Thought Reasoningcited by 7
  14. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  15. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6
  16. 2025DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMscited by 6
  17. 2026On State Reduction in Linear Attentioncited by 6
  18. 2025LeanK: Learnable K Cache Channel Pruning for Efficient Decodingcited by 5
  19. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  20. 2025Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoningcited by 4
  21. 2025Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decodingcited by 4
  22. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cachecited by 4
  23. 2026Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Samplingcited by 4
  24. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMscited by 3
  25. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  26. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  27. 2026WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantizationcited by 3
  28. 2025Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Accelerationcited by 2
  29. 2026RAP: KV-Cache Compression via RoPE-Aligned Pruningcited by 2
  30. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstructioncited by 2
  31. 2026HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compressioncited by 2
  32. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  33. 2024ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillationcited by 1
  34. 2024Boosting Long-Context Management via Query-Guided Activation Refillingcited by 1
  35. 2025PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inferencecited by 1
  36. 2025TAdaRAG: Task Adaptive Retrieval-Augmented Generation via On-the-Fly Knowledge Graph Constructioncited by 1
  37. 2025End-to-End Transformer Acceleration Through Processing-in-Memory Architecturescited by 1
  38. 2026Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selectioncited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
  2. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inference
  3. 2026DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  5. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  7. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference
  8. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review