mlsanityData to 14 Oct 2026

CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill

arXiv 2602.16054 ↗First posted 17 Feb 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The prefill stage in long-context LLM inference remains a computational bottleneck. Recent token-ranking heuristics accelerate inference by selectively processing a subset of semantically relevant tokens. However, existing methods suffer from unstable token importance estimation, often varying between layers. Evaluating token-ranking quality independently from heuristic-specific architectures is challenging. To address this, we introduce an Answer-Informed Oracle, which defines ground-truth token importance by measuring attention from generated answers back to the prompt. This oracle reveals that existing heuristics exhibit high variance across layers: rankings can degrade sharply at specific layers, a failure mode invisible to end-to-end benchmarks. The diagnosis suggests a simple fix: aggregate scores across layers rather than relying on any single one. We implement this as Cross-Layer Attention Aggregation (CLAA), which closes the gap to the oracle upper bound and reduces Time-to-First-Token (TTFT) by up to 39% compared to the Full KV Cache baseline.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 19 references that are in the map. First posted on arXiv 17 Feb 2026.

Builds on

15 of 19 references.

References · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  8. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  10. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  11. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  12. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
3 more
  1. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  2. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
  3. 2025Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimationcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
  2. 2025OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
  3. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
  4. 2026CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference
  5. 2024Extending Token Computation for LLM Reasoning
  6. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
  7. 2026CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference
  8. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs