mlsanityData to 7 Oct 2026

LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference

arXiv 2407.14057 ↗First posted 19 Jul 2024cs.CL cs.AI cs.LGcited by 70

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

The inference of transformer-based large language models consists of two sequential stages: 1) a prefilling stage to compute the KV cache of prompts and generate the first token, and 2) a decoding stage to generate subsequent tokens. For long prompts, the KV cache must be computed for all tokens during the prefilling stage, which can significantly increase the time needed to generate the first token. Consequently, the prefilling stage may become a bottleneck in the generation process. An open question remains whether all prompt tokens are essential for generating the first token. To answer this, we introduce a novel method, LazyLLM, that selectively computes the KV for tokens important for the next token prediction in both the prefilling and decoding stages. Contrary to static pruning approaches that prune the prompt at once, LazyLLM allows language models to dynamically select different subsets of tokens from the context in different generation steps, even though they might be pruned in previous steps. Extensive experiments on standard datasets across various tasks demonstrate that LazyLLM is a generic method that can be seamlessly integrated with existing language models to significantly accelerate the generation without fine-tuning. For instance, in the multi-document question-answering task, LazyLLM accelerates the prefilling stage of the LLama 2 7B model by 2.34x while maintaining accuracy.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 33 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

30 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding25 Apr
  2. 2021Magic Pyramid: Accelerating Inference with Early Exiting and Token Pruning30 Oct

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  12. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
16 more
  1. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  2. 2022Efficiently Scaling Transformer Inferencecited by 529
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  5. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  6. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  7. 2021Learned Token Pruning for Transformerscited by 138
  8. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  9. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  10. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  11. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  12. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 37
  13. 2023Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Promptcited by 28
  14. 2024Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generationcited by 15
  15. 2022Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformerscited by 11
  16. 2023XGen-7B Technical Reportcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 36; 2026: 22.

70 papers.

Sort:

By area:

  1. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  2. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  3. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  4. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
  5. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  6. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  7. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  8. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  9. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  10. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  11. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  12. 2025FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruningcited by 34
38 more
  1. 2025PENCIL: Long Thoughts with Short Memorycited by 31
  2. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  3. 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 20
  4. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  5. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  6. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 16
  7. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  8. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  9. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  10. 2025Adaptive GoGI-Skip: Coupling Goal-Gradient Importance with Dynamic Uncertainty for Efficient Reasoningcited by 12
  11. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similaritycited by 11
  12. 2025Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inferencecited by 10
  13. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 10
  14. 2024KV Prediction for Improved Time to First Tokencited by 9
  15. 2025CipherPrune: Efficient and Scalable Private Transformer Inferencecited by 9
  16. 2025Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuningcited by 8
  17. 2025InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPUcited by 7
  18. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  19. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfigurationcited by 6
  20. 2026FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inferencecited by 5
  21. 2025AdaFV: Rethinking of Visual-Language alignment for VLM accelerationcited by 4
  22. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniquescited by 4
  23. 2025METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understandingcited by 4
  24. 2025T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganizationcited by 4
  25. 2024CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuningcited by 3
  26. 2025Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Modelscited by 3
  27. 2025Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMscited by 3
  28. 2025ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Modelscited by 3
  29. 2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inferencecited by 3
  30. 2024TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generationcited by 2
  31. 2025Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Accelerationcited by 2
  32. 2025Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Informationcited by 2
  33. 2025OverFill: Two-Stage Models for Efficient Language Model Decodingcited by 2
  34. 2025Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contextscited by 2
  35. 2025Learning a Single Token to Replace Long System Prompts in LLMscited by 2
  36. 2025HEART-VIT: Hessian-Guided Efficient Dynamic Attention and Token Pruning in Vision Transformercited by 2
  37. 2026LLM-enabled Applications Require System-Level Threat Monitoringcited by 2
  38. 2026UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsificationcited by 2

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Dynamic Vocabulary Pruning in Early-Exit LLMs
  2. 2025Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?
  3. 2024RefreshKV: Updating Small KV Cache During Long-form Generation
  4. 2026POP: Prefill-Only Pruning for Efficient Large Model Inference
  5. 2024An Efficient Inference Framework for Early-exit Large Language Models
  6. 2024Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models
  7. 2026DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
  8. 2025HAMburger: Accelerating LLM Inference via Token Smashing