Improving Transformers using Faithful Positional Encoding
Not on the map. Why?
Abstract (arXiv)
We propose a new positional encoding method for a neural network architecture called the Transformer. Unlike the standard sinusoidal positional encoding, our approach is based on solid mathematical grounds and has a guarantee of not losing information about the positional order of the input sequence. We show that the new encoding approach systematically improves the prediction performance in the time-series classification task.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 11 references that are in the map. First posted on arXiv 15 May 2024.
Builds on
7 of 11 references.
References · 7
- 2017Attention Is All You Needcited by 60,361
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2018Self-Attention with Relative Position Representationscited by 1,120
- 2021Multi-Time Attention Networks for Irregularly Sampled Time Seriescited by 238
- 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
- 2022Formal Algorithms for Transformerscited by 72
- 2023Diagnostic Spatio-temporal Transformer with Faithful Encodingcited by 3
Built on by
Not cited yet.