mlsanityData to 7 Oct 2026

Self-Attention with Relative Position Representations

arXiv 1803.02155 ↗First posted 6 Mar 2018cs.CLcited by 1,112

Not on the map. Why?

Cited by 36 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs.

Key facts. Cited by 1,112 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 221; 2022: 210; 2023: 204; 2024: 189; 2025: 171; 2026: 117.

1,112 papers.

Sort:

By area:

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  5. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
  6. 2021Transformers in Vision: A Surveycited by 1,009
  7. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
  8. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  10. 2024A Survey on Large Language Models for Code Generationcited by 646
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  12. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
38 more
  1. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  2. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  3. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  4. 2023The Falcon Series of Open Language Modelscited by 526
  5. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  6. 2024Large Language Models: A Surveycited by 506
  7. 2022Transformers in Time Series: A Surveycited by 489
  8. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  9. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  10. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  11. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  12. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  13. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  14. 2021A Survey of Transformerscited by 399
  15. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  16. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  17. 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 319
  18. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  19. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  20. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  21. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  22. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  23. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  24. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  25. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  26. 2024Rotary Position Embedding for Vision Transformercited by 226
  27. 2021Associating Objects with Transformers for Video Object Segmentationcited by 225
  28. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  29. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  30. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  31. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  32. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 203
  33. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  34. 2022Structure-Aware Transformer for Graph Representation Learningcited by 191
  35. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 191
  36. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  37. 2021StyTr2: Image Style Transfer with Transformerscited by 186
  38. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186

50 most cited of 1,112 All 1,112 (JSON)