mlsanityData to 14 Oct 2026

MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computation

arXiv 2604.00235 ↗First posted 31 Mar 2026cs.LG cs.AI cs.DCcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context decoding in LLMs is IO-bound: each token re-reads an ever-growing KV cache. Prior accelerations cut bytes via compression, which lowers fidelity, or selection/eviction, which restricts what remains accessible, and both can degrade delayed recall and long-form generation. We introduce MAC-Attention, a fidelity- and access-preserving alternative that accelerates decoding by reusing prior attention computations for semantically similar recent queries. It starts with a match stage that performs pre-RoPE L2 matching over a short local window; an amend stage rectifies the reused attention by recomputing a small band near the match boundary; and a complete stage fuses the rectified results with fresh attention computed on the KV tail through a numerically stable merge. On a match hit, the compute and bandwidth complexity is constant regardless of context length. The method is model-agnostic and composes with IO-aware kernels, paged-KV managers, and MQA/GQA. Across LongBench v2 (120K), RULER (120K), and LongGenBench (16K continuous generation), compared to the latest FlashInfer library, MAC-Attention reduces KV accesses by up to 99%, cuts token generation latency by over 60% at 128K, and achieves over 14.3x attention-phase speedups, up to 2.6x end-to-end, while maintaining full-attention quality. By reusing computation, MAC-Attention delivers long-context inference that is both fast and faithful. Code is available here: https://github.com/YJHMITWEB/MAC-Attention.git

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 37 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

30 of 37 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression19 Feb
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  3. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy4 Oct
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference30 Mar
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
8 more
  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  3. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  4. 2018Online normalizer calculation for softmaxcited by 162
  5. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  6. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  7. 2024TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attentioncited by 29
  8. 2025Multipole Attention for Efficient Long Context Reasoningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Context Memorization for Efficient Long Context Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
  2. 2026ART: Attention Run-time Termination for Efficient Large Language Model Decoding
  3. 2025PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel
  4. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
  5. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  6. 2026KVBuffer: IO-aware Serving for Linear Attention
  7. 2026Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding
  8. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding