Generalized Attention Mechanism and Relative Position for Transformer
Not on the map. Why?
Abstract (arXiv)
In this paper, we propose generalized attention mechanism (GAM) by first suggesting a new interpretation for self-attention mechanism of Vaswani et al. . Following the interpretation, we provide description for different variants of attention mechanism which together form GAM. Further, we propose a new relative position representation within the framework of GAM. This representation can be easily utilized for cases in which elements next to each other in input sequence can be at random locations in actual dataset/corpus.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 5 references that are in the map. First posted on arXiv 24 Jul 2022.
Builds on
5 of 5 references.
References · 5
- 2017Attention Is All You Needcited by 60,361
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2018Self-Attention with Relative Position Representationscited by 1,120
- 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
- 2021Position Information in Transformers: An Overviewcited by 103
Built on by
Not cited yet.