mlsanityData to 7 Oct 2026

Stochastic positional embeddings improve masked image modeling

arXiv 2308.00566 ↗First posted 31 Jul 2023cs.CV cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Masked Image Modeling (MIM) is a promising self-supervised learning approach that enables learning from unlabeled images. Despite its recent success, learning good representations through MIM remains challenging because it requires predicting the right semantic content in accurate locations. For example, given an incomplete picture of a dog, we can guess that there is a tail, but we cannot determine its exact location. In this work, we propose to incorporate location uncertainty into MIM by using stochastic positional embeddings (StoP). Specifically, we condition the model on stochastic masked token positions drawn from a Gaussian distribution. StoP reduces overfitting to location features and guides the model toward learning features that are more robust to location uncertainties. Quantitatively, StoP improves downstream MIM performance on a variety of downstream tasks, including +1.7% on ImageNet linear probing using ViT-B, and +2.5% for ViT-H using 1% of the data.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

38 of 46 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
26 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  4. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  7. 2019Contrastive Multiview Codingcited by 1,297
  8. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  11. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  12. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  13. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  14. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  15. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  16. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  17. 2019Attention Augmented Convolutional Networkscited by 363
  18. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  19. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  20. 2020Whitening for Self-Supervised Representation Learningcited by 278
  21. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  22. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  23. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  24. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  25. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 57
  26. 2021Relative Positional Encoding for Transformers with Linear Complexitycited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 7.

12 papers.

  1. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  2. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  3. 2025Scaling Language-Free Visual Representation Learningcited by 62
  4. 2025Cluster and Predict Latent Patches for Improved Masked Image Modelingcited by 17
  5. 2024Sample- and Parameter-Efficient Auto-Regressive Image Modelscited by 5
  6. 2026DeFM: Learning Foundation Representations from Depth for Roboticscited by 3
  7. 2024Enhancing JEPAs with Spatial Conditioning: Robust and Efficient Representation Learningcited by 1
  8. 2026Suppressing Non-Semantic Noise in Masked Image Modeling Representationscited by 0
  9. 2026Joint-Embedding Predictive Architecture for Solar PV Panel Fault Classificationcited by 0
  10. 2026The JEPA Predictor: A Transferable Operator for Occluded Feature Completioncited by 0
  11. 2026What masking geometry works best for EEG foundation models?cited by 0
  12. 2026Image Classifiers are Efficient Self-Supervised Video Representation Learnerscited by 0