# Position Prediction as an Effective Pretraining Strategy

arXiv: https://arxiv.org/abs/2207.07611 · HTML: https://mlsanity.com/paper/2207.07611/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 15 Jul 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [Equivariant Contrastive Learning](https://mlsanity.com/paper/2111.00899/) (2021)
- [Temporal Convolution for Real-time Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1904.03814/) (2019)
- [Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition](https://mlsanity.com/paper/1808.07507/) (2018)
- [PERT: Pre-training BERT with Permuted Language Model](https://mlsanity.com/paper/2203.06906/) (2022)
- [Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup](https://mlsanity.com/paper/2111.15454/) (2021)

## Built on by (most cited first, 24 of 24)

- [Perceptual Grouping in Contrastive Vision-Language Models](https://mlsanity.com/paper/2210.09996/) (2022)
- [Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models](https://mlsanity.com/paper/2401.13537/) (2024)
- [MV-JAR: Masked Voxel Jigsaw and Reconstruction for LiDAR-Based Self-Supervised Pre-Training](https://mlsanity.com/paper/2303.13510/) (2023)
- [Autoregressive Pretraining with Mamba in Vision](https://mlsanity.com/paper/2406.07537/) (2024)
- [Location-Aware Self-Supervised Transformers for Semantic Segmentation](https://mlsanity.com/paper/2212.02400/) (2022)
- [DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions](https://mlsanity.com/paper/2309.03576/) (2023)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [Correlational Image Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2303.12670/) (2023)
- [MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments](https://mlsanity.com/paper/2307.09361/) (2023)
- [Energy-Inspired Self-Supervised Pretraining for Vision Models](https://mlsanity.com/paper/2302.01384/) (2023)
- [CLIP with Quality Captions: A Strong Pretraining for Vision Tasks](https://mlsanity.com/paper/2405.08911/) (2024)
- [SemPPL: Predicting pseudo-labels for better contrastive representations](https://mlsanity.com/paper/2301.05158/) (2023)
- [Soft-Landing Strategy for Alleviating the Task Discrepancy Problem in Temporal Action Localization Tasks](https://mlsanity.com/paper/2211.06023/) (2022)
- [SemanticMIM: Marring Masked Image Modeling with Semantics Compression for General Visual Representation](https://mlsanity.com/paper/2406.10673/) (2024)
- [How PARTs assemble into wholes: Learning the relative composition of images](https://mlsanity.com/paper/2506.03682/) (2025)
- [Label-Efficient Sleep Staging Using Transformers Pre-trained with Position Prediction](https://mlsanity.com/paper/2404.15308/) (2024)
- [Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training](https://mlsanity.com/paper/2509.17816/) (2025)
- [Learning to Perceive "Where": Spatial Pretext Tasks for Robust Self-Supervised Learning](https://mlsanity.com/paper/2605.09963/) (2026)
- [Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations](https://mlsanity.com/paper/2312.00101/) (2023)
- [MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance](https://mlsanity.com/paper/2412.10730/) (2024)
- [SPAR: Self-supervised Placement-Aware Representation Learning for Distributed Sensing](https://mlsanity.com/paper/2505.16936/) (2025)
- [Position Prediction Self-Supervised Learning for Multimodal Satellite Imagery Semantic Segmentation](https://mlsanity.com/paper/2506.06852/) (2025)
- [Learning the relative composition of EEG signals using pairwise relative shift pretraining](https://mlsanity.com/paper/2511.11940/) (2025)
- [Separators in Enhancing Autoregressive Pretraining for Vision Mamba](https://mlsanity.com/paper/2603.03806/) (2026)

## Abstract (arXiv)

Transformers have gained increasing popularity in a wide range of applications, including Natural Language Processing (NLP), Computer Vision and Speech Recognition, because of their powerful representational capacity. However, harnessing this representational capacity effectively requires a large amount of data, strong regularization, or both, to mitigate overfitting. Recently, the power of the Transformer has been unlocked by self-supervised pretraining strategies based on masked autoencoders which rely on reconstructing masked inputs, directly, or contrastively from unmasked content. This pretraining strategy which has been used in BERT models in NLP, Wav2Vec models in Speech and, recently, in MAE models in Vision, forces the model to learn about relationships between the content in different parts of the input using autoencoding related objectives. In this paper, we propose a novel, but surprisingly simple alternative to content reconstruction~-- that of predicting locations from content, without providing positional information for it. Doing so requires the Transformer to understand the positional relationships between different parts of the input, from their content alone. This amounts to an efficient implementation where the pretext task is a classification problem among all possible positions for each input token. We experiment on both Vision and Speech benchmarks, where our approach brings improvements over strong supervised training baselines and is comparable to modern unsupervised/self-supervised pretraining methods. Our method also enables Transformers trained without position embeddings to outperform ones trained with full position information.