mlsanityData to 7 Oct 2026

A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder

arXiv 2407.20485 ↗First posted 30 Jul 2024cs.CL cs.LGcited by 7

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Recently, large language models (LLM) based on transformers are facing memory bottleneck issues due to KV cache, especially in long sequence handling. Previous researches proposed KV cache compression techniques that identify insignificant tokens based on Accumulative Attention Scores and removes their items from KV cache, noting that only few tokens play an important role in attention operations. However, we have observed that the existing Accumulative Attention Score is not suitable for the transformer decoder structure. In the decoder model, the number of times the Attention Score accumulates varies depending on the order of token appearance due to the effect of masking, causing an uneven comparison between tokens. To solve this, we propose Accumulative Attention Score with Forgetting Factor (A2SF) technique, which introduces a Forgetting Factor in the Attention Score accumulation process. A2SF applies a penalty to the past Attention Score generated from old tokens by repeatedly multiplying the Forgetting Factor to the Attention Score over time. Therefore, older tokens receive a larger penalty, providing fairness among different ages of tokens. Through the fair comparison among tokens, we can more effectively select important tokens. We have verified the accuracy improvement through A2SF in the OPT and LLaMA models and A2SF improves the accuracy of LLaMA 2 by up to 7.8% and 5.1% on 1-shot and 0-shot.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 27 references that are in the map. First posted on arXiv 30 Jul 2024.

Builds on

22 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  2. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
6 more
  1. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  3. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  4. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  5. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  6. 2022No Token Left Behind: Explainability-Aided Image Classification and Generationcited by 25

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 5.

7 papers.

  1. 2025Cognitive Memory in Large Language Modelscited by 28
  2. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMscited by 27
  3. 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 20
  4. 2024In-context KV-Cache Eviction for LLMs via Attention-Gatecited by 12
  5. 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsitycited by 5
  6. 2024CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuningcited by 3
  7. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
  2. 2025Forgetting Transformer: Softmax Attention with a Forget Gate
  3. 2024FlashMask: Efficient and Rich Mask Extension of FlashAttention
  4. 2026Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
  5. 2026LOOKAT: Lookup-Optimized Key-Attention for Memory-Efficient Transformers
  6. 2026Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration
  7. 2024Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
  8. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression