mlsanityData to 14 Oct 2026

Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference

arXiv 2601.07667 ↗First posted 12 Jan 2026cs.CL cs.AI cs.LGcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Due to the prevalence of large language models (LLMs), key-value (KV) cache reduction for LLM inference has received remarkable attention. Among numerous works that have been proposed in recent years, layer-wise token pruning approaches, which select a subset of tokens at particular layers to retain in KV cache and prune others, are one of the most popular schemes. They primarily adopt a set of pre-defined layers, at which tokens are selected. Such design is inflexible in the sense that the accuracy significantly varies across tasks and deteriorates in harder tasks such as KV retrieval. In this paper, we propose ASL, a training-free method that adaptively chooses the selection layer for KV cache reduction, exploiting the variance of token ranks ordered by attention score. The proposed method balances the performance across different tasks while meeting the user-specified KV budget requirement. ASL operates during the prefilling stage and can be jointly used with existing KV cache reduction methods such as SnapKV to optimize the decoding stage. By evaluations on the InfiniteBench, RULER, and NIAH benchmarks, we show that ASL, equipped with one-shot token selection, adaptively trades inference speed for accuracy, outperforming state-of-the-art layer-wise token pruning methods in difficult tasks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 36 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

29 of 36 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025EvolKV: Evolutionary KV Cache Compression for LLM Inference10 Sep
  2. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning8 Aug
  3. 2025PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference30 Mar
  4. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
  5. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration3 Feb
  6. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs19 Dec
  7. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  8. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  9. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation4 Oct
  10. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction25 Sep
  11. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  12. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
5 more
  1. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  2. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  4. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget7 Apr
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct

Other references · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  9. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  10. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  11. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  12. 2024FoldGPT: Simple and Effective Large Language Model Compression Schemecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaborationcited by 5
  2. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 5
  3. 2026EntroPrefill: Renyi-Guided Context Pruning with Conditional Stability Guarantees for Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
  2. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  3. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
  4. 2025Leveraging KV Similarity for Online Structured Pruning in LLMs
  5. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  6. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  7. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  8. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference