mlsanityData to 7 Oct 2026

Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling

arXiv 2604.18103 ↗First posted 20 Apr 2026cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Prefilling computational costs pose a significant bottleneck for Large Language Models (LLMs) and Large Multimodal Models (LMMs) in long-context settings. While token pruning reduces sequence length, prior methods rely on heuristics that break compatibility with hardware-efficient kernels like FlashAttention. In this work, we observe that tokens evolve toward textitsemantic fixing points, making further processing redundant. To this end, we introduce Delta Attention Selective Halting (DASH), a training-free policy that monitors the layer-wise update dynamics of the self-attention mechanism to selectively halt stabilized tokens. Extensive evaluation confirms that DASH generalizes across language and vision benchmarks, delivering significant prefill speedups while preserving model accuracy and hardware efficiency. Code will be released at https://github.com/verach3n/DASH.git.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 32 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

19 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration26 Nov

Other references · 18

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  7. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  8. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  9. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  10. 2024What Matters in Transformers? Not All Attention is Neededcited by 83
  11. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  12. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
6 more
  1. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  2. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentationcited by 61
  3. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  4. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  5. 2024Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformerscited by 7
  6. 2025GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Evictioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models
  2. 2026DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models
  3. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
  4. 2026Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models
  5. 2026From Token Importance to Conditional Removability: Rethinking Visual Token Pruning in Multimodal Large Language Models
  6. 2026FASA: Frequency-aware Sparse Attention
  7. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
  8. 2026Sliding-window beats linear attention