mlsanityData to 7 Oct 2026

CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings

arXiv 2106.03143 ↗First posted 6 Jun 2021cs.LG cs.CL cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

Without positional information, attention-based Transformer neural networks are permutation-invariant. Absolute or relative positional embeddings are the most popular ways to feed Transformer models with positional information. Absolute positional embeddings are simple to implement, but suffer from generalization issues when evaluating on sequences longer than seen at training time. Relative positions are more robust to input length change, but are more complex to implement and yield inferior model throughput due to extra computational and memory costs. In this paper, we propose an augmentation-based approach (CAPE) for absolute positional embeddings, which keeps the advantages of both absolute (simplicity and speed) and relative positional embeddings (better generalization). In addition, our empirical evaluation on state-of-the-art models in machine translation, image and speech recognition demonstrates that CAPE leads to better generalization performance as well as increased stability with respect to training hyper-parameters.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 46 of its 60 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

46 of 60 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
34 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2021ViViT: A Video Vision Transformercited by 1,610
  4. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2017Random Erasing Data Augmentationcited by 1,235
  9. 2018Self-Attention with Relative Position Representationscited by 1,112
  10. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  11. 2018Image Transformercited by 770
  12. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  13. 2018Universal Transformerscited by 613
  14. 2020Recipes for building an open-domain chatbotcited by 593
  15. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  16. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  17. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  18. 2019Attention Augmented Convolutional Networkscited by 363
  19. 2021Video Transformer Networkcited by 271
  20. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  21. 2018Music Transformercited by 207
  22. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  23. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  24. 2021Position Information in Transformers: An Overviewcited by 103
  25. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 86
  26. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  27. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  28. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  29. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 72
  30. 2019Encoding word order in complex embeddingscited by 53
  31. 2020Very Deep Transformers for Neural Machine Translationcited by 50
  32. 2019Transformers with convolutional context for ASRcited by 44
  33. 2018Deep Metric Transfer for Label Propagation with Limited Annotated Datacited by 33
  34. 2019Improving Generalization of Transformer for Speech Recognition with Parallel Schedule Sampling and Relative Positional Embeddingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 16; 2024: 8; 2025: 11; 2026: 1.

47 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  4. 2024Rotary Position Embedding for Vision Transformercited by 226
  5. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  6. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  7. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  8. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  9. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  10. 2023Transformers in Speech Processing: A Surveycited by 38
  11. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  12. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
35 more
  1. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  2. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  3. 2021Pseudo-Labeling for Massively Multilingual Speech Recognitioncited by 23
  4. 2023How to Scale Your EMAcited by 22
  5. 2023Linearized Relative Positional Encodingcited by 17
  6. 2024AMES: Asymmetric and Memory-Efficient Similarity Estimation for Instance-level Retrievalcited by 16
  7. 2022Variable Attention Masking for Configurable Transformer Transducer Speech Recognitioncited by 14
  8. 2024Towards LifeSpan Cognitive Systemscited by 13
  9. 2022Continual Transformers: Redundancy-Free Attention for Online Inferencecited by 12
  10. 2022Continuous Pseudo-Labeling from the Startcited by 11
  11. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  12. 2023Monotonic Location Attention for Length Generalizationcited by 9
  13. 2025LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encodingcited by 9
  14. 2023Extract-and-Adaptation Network for 3D Interacting Hand Mesh Recoverycited by 7
  15. 2022Continuous Soft Pseudo-Labeling in ASRcited by 6
  16. 2024HSViT: Horizontally Scalable Vision Transformercited by 6
  17. 2023AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognitioncited by 5
  18. 2025Group Representational Position Encodingcited by 5
  19. 2022Similarity and Content-based Phonetic Self Attention for Speech Recognitioncited by 3
  20. 2023NeuroCodeBench: a plain C neural network benchmark for software verificationcited by 3
  21. 2023Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysiscited by 3
  22. 2022More Speaking or More Speakers?cited by 2
  23. 2023LEA: Improving Sentence Similarity Robustness to Typos Using Lexical Attention Biascited by 2
  24. 2025Context-aware Biases for Length Extrapolationcited by 2
  25. 2025Floating-Point Neural Network Verification at the Software Levelcited by 2
  26. 2022Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLPcited by 1
  27. 2025Positional Encoding via Token-Aware Phase Attentioncited by 1
  28. 2023Extending Input Contexts of Language Models through Training on Segmented Sequencescited by 0
  29. 2024Knowledge-enhanced Transformer for Multivariate Long Sequence Time-series Forecastingcited by 0
  30. 2025Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Modelscited by 0
  31. 2025Input Resolution Downsizing as a Compression Technique for Vision Deep Learning Systemscited by 0
  32. 2025Learning to Adapt to Position Bias in Vision Transformer Classifierscited by 0
  33. 2025PiPViT: Patch-based Visual Interpretable Prototypes for Retinal Image Analysiscited by 0
  34. 2025Segmental Attention Decoding With Long Form Acoustic Encodingscited by 0
  35. 2026Position Encoding with Random Float Sampling Enhances Length Generalization of Transformerscited by 0