mlsanityData to 14 Oct 2026

KV Shifting Attention Enhances Language Modeling

arXiv 2411.19574 ↗First posted 29 Nov 2024cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

The current large language models are mainly based on decode-only structure transformers, which have great in-context learning (ICL) capabilities. It is generally believed that the important foundation of its ICL capability is the induction heads mechanism, which requires at least two layers attention. In order to more efficiently implement the ability of the model's induction, we revisit the induction heads mechanism and proposed a KV shifting attention. We theoretically prove that the KV shifting attention reducing the model's requirements for the depth and width of the induction heads mechanism. Our experimental results demonstrate that KV shifting attention is beneficial to learning induction heads and language modeling, which lead to better performance or faster convergence from toy models to the pre-training models with more than 10 B parameters.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

37 of 54 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  11. 2022In-context Learning and Induction Headscited by 964
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
25 more
  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  2. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  3. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  4. 2023Function Vectors in Large Language Modelscited by 308
  5. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  6. 2024An Empirical Study of Mamba-based Language Modelscited by 186
  7. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  8. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  9. 2024Transformers, parallel computation, and logarithmic depthcited by 79
  10. 2024Base of RoPE Bounds Context Lengthcited by 52
  11. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 50
  12. 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 47
  13. 2024Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learningcited by 46
  14. 2022Sub-Task Decomposition Enables Learning in Sequence to Sequence Taskscited by 41
  15. 2023Learning Transformer Programscited by 41
  16. 2024One-layer transformers fail to solve the induction heads taskcited by 32
  17. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  18. 2024Transformers on Markov Data: Constant Depth Sufficescited by 27
  19. 2024Selective Attention Improves Transformercited by 26
  20. 2023Accelerating Transducers through Adjacent Token Mergingcited by 15
  21. 2024Differential Transformercited by 13
  22. 2024Fundamental Limitations on Subquadratic Alternatives to Transformerscited by 9
  23. 2024How Transformers Get Rich: Approximation and Dynamics Analysiscited by 8
  24. 2023Large Learning Rates Improve Generalization: But How Large Are We Talking About?cited by 1
  25. 2024Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  2. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 43
  3. 2025Understanding Transformer from the Perspective of Associative Memorycited by 34
  4. 2025Multi-Token Attentioncited by 10
  5. 2026Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Designcited by 5
  6. 2025Amplify Adjacent Token Differences: Enhancing Long Chain-of-Thought Reasoning with Shift-FFNcited by 1
  7. 2026Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstructioncited by 0
  8. 2026Why Do Conventional World Models Fail to Learn Cellular Automata?cited by 0