mlsanityData to 7 Oct 2026

Kimi Linear: An Expressive, Efficient Attention Architecture

arXiv 2510.26692 ↗First posted 30 Oct 2025cs.CL cs.LGcited by 169

Not on the map. Why?

Cited by 27 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule. We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths. To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Builds on 109 of its 125 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

109 of 125 references.

References · 109

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
38 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2019Root Mean Square Layer Normalizationcited by 1,425
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  13. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  14. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  15. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  16. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  17. 2018Self-Attention with Relative Position Representationscited by 1,112
  18. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  19. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  20. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  21. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  22. 2025Kimi K2: Open Agentic Intelligencecited by 523
  23. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  24. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  25. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  26. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  27. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  28. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  29. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  30. 2025Muon is Scalable for LLM Trainingcited by 381
  31. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  32. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  33. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  34. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  35. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  36. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  37. 2022Memorizing Transformerscited by 219
  38. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208

50 most cited of 109 All 109 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 161.

169 papers.

Sort:

By area:

  1. 2026MiMo-V2-Flash Technical Reportcited by 163
  2. 2026Kimi K3: Open Frontier Intelligencecited by 99
  3. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  4. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  5. 2026Attention Residualscited by 57
  6. 2026Helios: Real Real-Time Long Video Generation Modelcited by 56
  7. 2025End-to-End Test-Time Training for Long Contextcited by 53
  8. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 50
  9. 2026MiniMax Sparse Attentioncited by 34
  10. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  11. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 24
  12. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
38 more
  1. 2026Alaya-EVOKE: From Linear-Scaling Supervision to Endless Worldcited by 13
  2. 2026Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scalingcited by 12
  3. 2026δ-mem: Efficient Online Memory for Large Language Modelscited by 12
  4. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacentercited by 11
  5. 2026Arcee Trinity Large Technical Reportcited by 10
  6. 2026Test-Time Training with KV Binding Is Secretly Linear Attentioncited by 10
  7. 2026SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformercited by 10
  8. 2025Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regressioncited by 9
  9. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 9
  10. 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewardscited by 9
  11. 2026FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefillingcited by 9
  12. 2026HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharingcited by 8
  13. 2026SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Normcited by 8
  14. 2026HorizonStream: Long-Horizon Attention for Streaming 3D Reconstructioncited by 8
  15. 2025SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verificationcited by 7
  16. 2026Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Trackingcited by 7
  17. 2026Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillationcited by 7
  18. 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Cachingcited by 7
  19. 2026On State Reduction in Linear Attentioncited by 6
  20. 2026Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMscited by 5
  21. 2026Learning State-Tracking from Code Using Linear RNNscited by 5
  22. 2026Mem3R: Streaming 3D Reconstruction with Hybrid Memory via Test-Time Trainingcited by 5
  23. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  24. 2026Rethinking the Role of Efficient Attention in Hybrid Architecturescited by 5
  25. 2025Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamicscited by 4
  26. 2026Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformerscited by 4
  27. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Trainingcited by 4
  28. 2026End-to-End Context Compression at Scalecited by 4
  29. 2026A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgetscited by 4
  30. 2026Raven: High-Recall Sequence Modeling with Sparse Memory Routingcited by 4
  31. 2026DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantizationcited by 4
  32. 2026STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMscited by 3
  33. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inferencecited by 3
  34. 2026Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implicationscited by 3
  35. 2026Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharingcited by 3
  36. 2026MDN: Parallelizing Stepwise Momentum for Delta Linear Attentioncited by 3
  37. 2026KVBuffer: IO-aware Serving for Linear Attentioncited by 3
  38. 2026Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformerscited by 3

50 most cited of 169 All 169 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learning
  2. 2026ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation
  3. 2026On-Policy Attention Linearization
  4. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  5. 2025Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
  6. 2025GatedFWA: Linear Flash Windowed Attention with Gated Associative Memory
  7. 2026HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing
  8. 2026GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference