QV May Be Enough: Toward the Essence of Attention in LLMs
Not on the map. Why?
Abstract (arXiv)
Starting from first principles and a linguistic perspective centered on part-of-speech (POS) and syntactic analysis, this paper explores and derives the underlying essence of the Query-Key-Value (QKV) mechanism within the Transformer architecture. Based on this theoretical foundation, we provide a unified explanatory framework for the efficacy of contemporary architectures, including MQA, GQA, and MLA, while identifying their inherent trade-offs and potential optimization trajectories. We introduce the QV paradigm and provide empirical evidence for its validity. Building upon this, we propose the QV-Ka optimization scheme, which is further substantiated through experimental validation. The interpretable theoretical analysis of the QKV mechanism presented in this work establishes a robust foundation for the future evolution of large language model architectures.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 7 of its 7 references that are in the map. First posted on arXiv 11 Mar 2026.
Builds on
7 of 7 references.
References · 7
- 2017Attention Is All You Needcited by 60,361
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
- 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 89
- 2026Attention's Gravitational Field:A Power-Law Interpretation of Positional Correlationcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.