The Brownian motion in the transformer model
Not on the map. Why?
Abstract (arXiv)
Transformer is the state of the art model for many language and visual tasks. In this paper, we give a deep analysis of its multi-head self-attention (MHSA) module and find that: 1) Each token is a random variable in high dimensional feature space. 2) After layer normalization, these variables are mapped to points on the hyper-sphere. 3) The update of these tokens is a Brownian motion. The Brownian motion has special properties, its second order item should not be ignored. So we present a new second-order optimizer(an iterative K-FAC algorithm) for the MHSA module. In some short words: All tokens are mapped to high dimension hyper-sphere. The Scaled Dot-Product Attention softmax(fracQKT√d) is just the Markov transition matrix for the random walking on the sphere. And the deep learning process would learn proper kernel function to get proper positions of these tokens. The training process in the MHSA module corresponds to a Brownian motion worthy of further study.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 20 of its 39 references that are in the map. First posted on arXiv 12 Jul 2021.
Builds on
20 of 39 references.
References · 20
- 2017Attention Is All You Needcited by 59,932
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
- 2019Attention is not Explanationcited by 916
- 2020On Layer Normalization in the Transformer Architecturecited by 906
- 2020A Primer in BERTology: What we know about how BERT workscited by 807
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2019Attention is not not Explanationcited by 629
8 more
- 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
- 2014New insights and perspectives on the natural gradient methodcited by 435
- 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
- 2019Revealing the Dark Secrets of BERTcited by 300
- 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
- 2016Random walks and diffusion on networkscited by 28
- 2020New Interpretations of Normalization Methods in Deep Learningcited by 8
- 2021An iterative K-FAC algorithm for Deep Learningcited by 2
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 1.
2 papers.