mlsanityData to 7 Oct 2026

MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning

arXiv 2603.20586 ↗First posted 21 Mar 2026cs.LG cs.AIcited by 6

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

As long-context language modeling becomes increasingly important, the cost of maintaining and attending to large Key/Value (KV) caches grows rapidly, becoming a major bottleneck in both training and inference. While prior works such as Multi-Query Attention (MQA) and Multi-Latent Attention (MLA) reduce memory by sharing or compressing KV features, they often trade off representation quality or incur runtime overhead. We propose Memory-Keyed Attention (MKA), a hierarchical attention mechanism that integrates multi-level KV caches (local, session, and long-term) and learns to route attention across them dynamically. We further introduce Route-Fused MKA (FastMKA), a broadcast-routed variant that fuses memory sources before attention computation for improved efficiency. Experiments on different sequence lengths show that FastMKA achieves a favorable accuracy-efficiency trade-off: comparable perplexity to MLA while achieving up to 5x faster training throughput and 1.8x lower evaluation latency. These results highlight MKA as a practical and extensible framework for efficient long-context attention.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 19 of its 28 references that are in the map. First posted on arXiv 21 Mar 2026.

Builds on

19 of 28 references.

References · 19

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  10. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  11. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  12. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
7 more
  1. 2014Memory Networkscited by 354
  2. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  3. 2024Transformers are Multi-State RNNscited by 120
  4. 2025Scaling Laws for Native Multimodal Modelscited by 47
  5. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  6. 2024Verification of Geometric Robustness of Neural Networks via Piecewise Linear Approximation and Lipschitz Optimisationcited by 7
  7. 2024Prototype Smart Home Environment With Biofeedbackcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compressioncited by 2
  2. 2026Accelerating Frequency Domain Diffusion Models with Error-Feedback Event-Driven Cachingcited by 1
  3. 2026Thoughts-as-Planning: Latent World Models for Chain-of-Thoughts Optimization via Reinforcement Planningcited by 1
  4. 2026RUBRIC: Realism--Utility Balanced Ranking for Imbalanced Classificationcited by 1
  5. 2026OLIVE: Online Low-Rank Incremental Learning for Efficient Adaptive Exoskeletonscited by 0
  6. 2026HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ARM: Attention with Routed-Memory for Learnable Sparse Control
  2. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  3. 2026Kwai Summary Attention Technical Report
  4. 2026Sliding-window beats linear attention
  5. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
  6. 2025Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads
  7. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
  8. 2024Beyond KV Caching: Shared Attention for Efficient LLMs