mlsanityData to 14 Oct 2026

Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers

arXiv 2305.15805 ↗First posted 25 May 2023cs.CL cs.LGcited by 67

Not on the map. Why?

Cited by 24 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Autoregressive Transformers adopted in Large Language Models (LLMs) are hard to scale to long sequences. Despite several works trying to reduce their computational cost, most of LLMs still adopt attention layers between all pairs of tokens in the sequence, thus incurring a quadratic cost. In this study, we present a novel approach that dynamically prunes contextual information while preserving the model's expressiveness, resulting in reduced memory and computational requirements during inference. Our method employs a learnable mechanism that determines which uninformative tokens can be dropped from the context at any point across the generation process. By doing so, our approach not only addresses performance concerns but also enhances interpretability, providing valuable insight into the model's decision-making process. Our technique can be applied to existing pre-trained models through a straightforward fine-tuning process, and the pruning strength can be specified by a sparsity parameter. Notably, our empirical findings demonstrate that we can effectively prune up to 80% of the context without significant performance degradation on downstream tasks, offering a valuable tool for mitigating inference costs. Our reference implementation achieves up to 2× increase in inference throughput and even greater memory savings.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Builds on 52 of its 66 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

52 of 66 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  11. 2020Longformer: The Long-Document Transformercited by 3,413
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  5. 2020Reformer: The Efficient Transformercited by 1,668
  6. 2020Rethinking Attention with Performerscited by 1,601
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  13. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  14. 2022Token Merging: Your ViT But Fastercited by 1,021
  15. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  16. 2020On Layer Normalization in the Transformer Architecturecited by 922
  17. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  18. 2020Efficient Transformers: A Surveycited by 790
  19. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  20. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  21. 2022Efficiently Scaling Transformer Inferencecited by 539
  22. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  23. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  24. 2020Hopfield Networks is All You Needcited by 475
  25. 2021A Survey of Transformerscited by 401
  26. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  27. 2019Sparse Sequence-to-Sequence Modelscited by 170
  28. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  29. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  30. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  31. 2021Learned Token Pruning for Transformerscited by 139
  32. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 125
  33. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  34. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  35. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  36. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 83
  37. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  38. 2021H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequencescited by 47

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 31; 2025: 22; 2026: 9.

67 papers.

Sort:

By area:

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  4. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  6. 2023Efficient Large Language Models: A Surveycited by 186
  7. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  8. 2024Transformers are Multi-State RNNscited by 120
  9. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  10. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  11. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
  12. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
38 more
  1. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  2. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 52
  3. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  4. 2024A Survey on Transformer Compressioncited by 46
  5. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  6. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  7. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Timecited by 43
  8. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 30
  9. 2025Cognitive Memory in Large Language Modelscited by 29
  10. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 28
  11. 2024Selective Attention Improves Transformercited by 26
  12. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budgetcited by 23
  13. 2024HSR-Enhanced Sparse Attention Accelerationcited by 19
  14. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  15. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  16. 2024In-context KV-Cache Eviction for LLMs via Attention-Gatecited by 13
  17. 2025FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Computecited by 12
  18. 2024Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questionscited by 11
  19. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruningcited by 10
  20. 2025PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attentioncited by 9
  21. 2024A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decodercited by 8
  22. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inferencecited by 7
  23. 2024A-VL: Adaptive Attention for Large Vision-Language Modelscited by 7
  24. 2024Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Modelscited by 7
  25. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
  26. 2025EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Modelscited by 7
  27. 2024QCQA: Quality and Capacity-aware grouped Query Attentioncited by 6
  28. 2025Position: AI Scaling: From Up to Down and Outcited by 5
  29. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniquescited by 5
  30. 2025Saliency-driven Dynamic Token Pruning for Large Language Modelscited by 5
  31. 2025Adaptive Computation Pruning for the Forgetting Transformercited by 5
  32. 2025FastFLUX: Pruning FLUX with Block-wise Replacement and Sandwich Trainingcited by 5
  33. 2023Navigating Scaling Laws: Compute Optimality in Adaptive Model Trainingcited by 4
  34. 2024CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modelingcited by 4
  35. 2024EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importancecited by 4
  36. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  37. 2025Token Level Routing Inference System for Edge Devicescited by 4
  38. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Mini-GPTs: Efficient Large Language Models through Contextual Pruning
  2. 2025Pruning Laws for Large Language Models
  3. 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
  4. 2024Dynamic Vocabulary Pruning in Early-Exit LLMs
  5. 2026When Does Context Help? Error Dynamics of Contextual Information in Large Language Models
  6. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
  7. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
  8. 2025Dynamic Vocabulary Pruning: Stable LLM-RL by Taming the Tail