mlsanityData to 14 Oct 2026

SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning

arXiv 2508.06447 ↗First posted 8 Aug 2025cs.CLcited by 13

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context inference for Large Language Models (LLMs) is heavily limited by high computational demands. While several existing methods optimize attention computation, they still process the full set of hidden states at each layer, limiting overall efficiency. In this work, we propose SlimInfer, an innovative framework that aims to accelerate inference by directly pruning less critical prompt tokens during the forward pass. Our key insight is an information diffusion phenomenon: As information from critical tokens propagates through layers, it becomes distributed across the entire sequence. This diffusion process suggests that LLMs can maintain their semantic integrity when excessive tokens, even including these critical ones, are pruned in hidden states. Motivated by this, SlimInfer introduces a dynamic fine-grained pruning mechanism that accurately removes redundant tokens of hidden state at intermediate layers. This layer-wise pruning naturally enables an asynchronous KV cache manager that prefetches required token blocks without complex predictors, reducing both memory usage and I/O costs. Extensive experiments show that SlimInfer can achieve up to 2.53× time-to-first-token (TTFT) speedup and 1.88× end-to-end latency reduction for LLaMA3.1-8B-Instruct on a single RTX 4090, without sacrificing performance on LongBench. Our code is available at https://github.com/Longxmas/SlimInfer.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 36 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

26 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference11 Mar
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr

Other references · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  7. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  8. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  9. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  10. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  11. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  12. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
10 more
  1. 2023TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Modelscited by 69
  2. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  3. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  4. 2024TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attentioncited by 29
  5. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 22
  6. 2024How Sparse Attention Approximates Exact Attention? Your Attention is Naturally nC-Sparsecited by 17
  7. 2024HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Accelerationcited by 17
  8. 2025Pre3: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generationcited by 11
  9. 2024Temporal Feature Matters: A Framework for Diffusion Model Quantizationcited by 5
  10. 2024PTSBench: A Comprehensive Post-Training Sparsity Benchmark Towards Algorithms and Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  2. 2025Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuningcited by 8
  3. 2026FBS: Modeling Native Parallel Reading inside a Transformercited by 7
  4. 2026Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusingcited by 7
  5. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  6. 2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inferencecited by 3
  7. 2026UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsificationcited by 2
  8. 2025On-device Semantic Selection Made Low Latency and Memory Efficient with Monolithic Forwardingcited by 1
  9. 2026Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefillingcited by 1
  10. 2026VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMscited by 0
  11. 2026TALSC: Timeliness-Aware Large-Small VLM Collaboration for Infrastructure-Assisted Autonomous Drivingcited by 0
  12. 2026Beyond Sparse Weights: When Is Attention Compressible?cited by 0
1 more
  1. 2026EntroPrefill: Renyi-Guided Context Pruning with Conditional Stability Guarantees for Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
  2. 2026DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
  3. 2026Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference
  4. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  5. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
  6. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
  7. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  8. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads