mlsanityData to 14 Oct 2026

Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models

arXiv 2106.05505 ↗First posted 10 Jun 2021cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we detail the relationship between convolutions and self-attention in natural language tasks. We show that relative position embeddings in self-attention layers are equivalent to recently-proposed dynamic lightweight convolutions, and we consider multiple new ways of integrating convolutions into Transformer self-attention. Specifically, we propose composite attention, which unites previous relative position embedding methods under a convolutional framework. We conduct experiments by training BERT with composite attention, finding that convolutions consistently improve performance on multiple downstream tasks, replacing absolute position embeddings. To inform future work, we present results comparing lightweight convolutions, dynamic convolutions, and depthwise-separable convolutions in language model pre-training, considering multiple injection points for convolutions in self-attention layers.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 25 of its 32 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

25 of 32 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2018Self-Attention with Relative Position Representationscited by 1,120
13 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  2. 2018Neural Network Acceptability Judgmentscited by 958
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  4. 2019Attention Augmented Convolutional Networkscited by 363
  5. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  6. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  7. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  8. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  9. 2020ConvBERT: Improving BERT with Span-based Dynamic Convolutioncited by 88
  10. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  11. 2020Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translationcited by 43
  12. 2020Local Self-Attention over Long Text for Efficient Document Retrievalcited by 18
  13. 2019Position-Aware Self-Attention based Neural Sequence Labelingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 2; 2024: 1; 2025: 2; 2026: 2.

11 papers.

  1. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  2. 2021DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understandingcited by 15
  3. 2023Monotonic Location Attention for Length Generalizationcited by 9
  4. 2022Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-trainingcited by 6
  5. 2022Word Order Matters when you Increase Maskingcited by 4
  6. 2025Integral Transformer: Denoising Attention, Not Too Much Not Too Littlecited by 3
  7. 2022SeqNet: An Efficient Neural Network for Automatic Malware Detectioncited by 1
  8. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  9. 2024TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Modelscited by 0
  10. 2025Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generationcited by 0
  11. 2026Performance and Complexity Trade-off Optimization of Speech Models During Trainingcited by 0