mlsanityData to 7 Oct 2026

Relative Position Prediction as Pre-training for Text Encoders

arXiv 2202.01145 ↗First posted 2 Feb 2022cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Meaning is defined by the company it keeps. However, company is two-fold: It's based on the identity of tokens and also on their position (topology). We argue that a position-centric perspective is more general and useful. The classic MLM and CLM objectives in NLP are easily phrased as position predictions over the whole vocabulary. Adapting the relative position encoding paradigm in NLP to create relative labels for self-supervised learning, we seek to show superior pre-training judged by performance on downstream tasks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 2 Feb 2022.

Builds on

16 of 28 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2018Deep contextualized word representationscited by 2,285
4 more
  1. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  2. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2022Rewiring with Positional Encodings for Graph Neural Networkscited by 35

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Position Prediction as an Effective Pretraining Strategycited by 24