mlsanityData to 7 Oct 2026

PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference

arXiv 2503.23274 ↗First posted 30 Mar 2025cs.CLcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

As large language models (LLMs) tackle increasingly complex tasks and longer documents, their computational and memory costs during inference become a major bottleneck. To address this, we propose PromptDistill, a novel, training-free method that improves inference efficiency while preserving generation quality. PromptDistill identifies and retains the most informative tokens by leveraging attention interactions in early layers, preserving their hidden states while reducing the computational burden in later layers. This allows the model to focus on essential contextual information without fully processing all tokens. Unlike previous methods such as H2O and SnapKV, which perform compression only after processing the entire input, or GemFilter, which selects a fixed portion of the initial prompt without considering contextual dependencies, PromptDistill dynamically allocates computational resources to the most relevant tokens while maintaining a global awareness of the input. Experiments using our method and baseline approaches with base models such as LLaMA 3.1 8B Instruct, Phi 3.5 Mini Instruct, and Qwen2 7B Instruct on benchmarks including LongBench, InfBench, and Needle in a Haystack demonstrate that PromptDistill significantly improves efficiency while having minimal impact on output quality compared to the original models. With a single-stage selection strategy, PromptDistill effectively balances performance and efficiency, outperforming prior methods like GemFilter, H2O, and SnapKV due to its superior ability to retain essential information. Specifically, compared to GemFilter, PromptDistill achieves an overall 1% to 5% performance improvement while also offering better time efficiency. Additionally, we explore multi-stage selection, which further improves efficiency while maintaining strong generation performance.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 29 references that are in the map. First posted on arXiv 30 Mar 2025.

Builds on

25 of 29 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024RefreshKV: Updating Small KV Cache During Long-form Generation8 Nov
  2. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts2 Oct
  3. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction25 Sep
  4. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 18

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  5. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  6. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  7. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  8. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  9. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  10. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  11. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  12. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
6 more
  1. 2024Needle in the Haystack for Memory Based Large Language Modelscited by 32
  2. 2024Selective Attention: Enhancing Transformer through Principled Context Controlcited by 16
  3. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  4. 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrievalcited by 5
  5. 2024QuickLLaMA: Query-aware Inference Acceleration for Large Language Modelscited by 4
  6. 2024MCSD: An Efficient Language Model with Diverse Fusioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inferencecited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Dynamic Compressing Prompts for Efficient Inference of Large Language Models
  2. 2023Batch Prompting: Efficient Inference with Large Language Model APIs
  3. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
  4. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference
  5. 2023SparQ Attention: Bandwidth-Efficient LLM Inference
  6. 2025Learning a Single Token to Replace Long System Prompts in LLMs
  7. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
  8. 2026Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference