# Role of Bias Terms in Dot-Product Attention

arXiv: https://arxiv.org/abs/2302.08626 · HTML: https://mlsanity.com/paper/2302.08626/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2023.

## Built on by (most cited first, 5 of 5)

- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers](https://mlsanity.com/paper/2406.02847/) (2024)
- [Symmetry Breaking in Neural Network Optimization: Insights from Input Dimension Expansion](https://mlsanity.com/paper/2409.06402/) (2024)
- [HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoning](https://mlsanity.com/paper/2610.03039/) (2026)

## Abstract (arXiv)

Dot-product attention is a core module in the present generation of neural network models, particularly transformers, and is being leveraged across numerous areas such as natural language processing and computer vision. This attention module is comprised of three linear transformations, namely query, key, and value linear transformations, each of which has a bias term. In this work, we study the role of these bias terms, and mathematically show that the bias term of the key linear transformation is redundant and could be omitted without any impact on the attention module. Moreover, we argue that the bias term of the value linear transformation has a more prominent role than that of the bias term of the query linear transformation. We empirically verify these findings through multiple experiments on language modeling, natural language understanding, and natural language generation tasks.