Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers
Not on the map. Why?
Abstract (arXiv)
We theoretically investigate whether the Query, Key, Value weight triplet can be reduced in encoder-only and decoder-only transformers. Under mild assumptions, we prove that one of the Query, Key or Value weights are redundant and can be replaced with the identity matrix, reducing attention parameters by 25%. If applied to the Query or Key weights, this also simplifies optimization: attention logits depend on a single learned weight matrix rather than on a product of two. Validating the Query weight removal on decoder-only GPT-style small models trained from scratch, we find that reduced models match baseline performance despite fewer parameters, and outperform baselines when saved parameters are reallocated. Our analysis has also led us to a structural expressivity boundary: in the mathematically tractable ReLU setting, skip connections push MLPs into a generically disjoint function class at fixed width. These findings motivate investigation across modalities and at scale, where the observed stability and efficiency gains may prove most consequential.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 27 Oct 2025.
Builds on
43 of 59 references.
References · 43
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2025Qwen3 Technical Reportcited by 8,195
- 2016Layer Normalizationcited by 5,579
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
31 more
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2016Identity Mappings in Deep Residual Networkscited by 1,960
- 2020Rethinking Attention with Performerscited by 1,584
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
- 2019Root Mean Square Layer Normalizationcited by 1,425
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2020Efficient Transformers: A Surveycited by 784
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
- 2019Deep Equilibrium Modelscited by 603
- 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
- 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
- 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
- 2016Using the Output Embedding to Improve Language Modelscited by 266
- 2020Query-Key Normalization for Transformerscited by 266
- 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
- 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
- 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
- 2023A mathematical perspective on Transformerscited by 160
- 2023The emergence of clusters in self-attention dynamicscited by 133
- 2024Hyper-Connectionscited by 81
- 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
- 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
- 2023Simplifying Transformer Blockscited by 37
- 2024From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformerscited by 34
- 2025Always Skip Attentioncited by 12
- 2024You can remove GPT2's LayerNorm by fine-tuningcited by 11
- 2025Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretabilitycited by 7
Built on by
Citing papers per year: 2026: 2.
2 papers.