Beyond Linearity in Attention Projections: The Case for Nonlinear Queries
Not on the map. Why?
Abstract (arXiv)
Recent algebraic analysis shows that in decoder-only and encoder-only transformers, the Query projection WQ may be set to identity without noticeable performance deterioration. This is possible because attention depends on X only through the products XWQ, XWK, XWV, allowing basis transformations to be absorbed by adjacent layers and propagated through the network. We replace WQ ∈ Rd × d with a nonlinear residual of the form Q(X) = X + fθ(X), where fθ is a bottleneck MLP with d2 + O(d) parameters. The identity term anchors the nonlinearity to a known-good prior. Experiments on GPT-3 small style models show consistent improvement over the baseline (2.40% lower validation log-loss, 6.81% lower perplexity), comfortably outperforming a model with 12.5% more non-embedding parameters. These results motivate investigation at larger scales and across modalities.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 11 Mar 2026.
Builds on
27 of 34 references.
References · 27
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2020Rethinking Attention with Performerscited by 1,584
- 2022Toy Models of Superpositioncited by 928
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
15 more
- 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
- 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
- 2024Language models scale reliably with over-training and on downstream taskscited by 105
- 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
- 2024Hyper-Connectionscited by 81
- 2022Training Trajectories of Language Models Across Scalescited by 80
- 2025mHC: Manifold-Constrained Hyper-Connectionscited by 76
- 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
- 2024Normalization and effective learning rates in reinforcement learningcited by 55
- 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
- 2024Efficient Learning With Sine-Activated Low-rank Matricescited by 22
- 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
- 2025Always Skip Attentioncited by 12
- 2023Neural Attention: Enhancing QKV Calculation in Self-Attention Mechanism with Neural Networkscited by 3
- 2025Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformerscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.