mlsanityData to 7 Oct 2026

RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-training

arXiv 2201.06857 ↗First posted 18 Jan 2022cs.CV cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Recently, self-supervised vision transformers have attracted unprecedented attention for their impressive representation learning ability. However, the dominant method, contrastive learning, mainly relies on an instance discrimination pretext task, which learns a global understanding of the image. This paper incorporates local feature learning into self-supervised vision transformers via Reconstructive Pre-training (RePre). Our RePre extends contrastive frameworks by adding a branch for reconstructing raw image pixels in parallel with the existing contrastive objective. RePre is equipped with a lightweight convolution-based decoder that fuses the multi-hierarchy features from the transformer encoder. The multi-hierarchy features provide rich supervisions from low to high semantic information, which are crucial for our RePre. Our RePre brings decent improvements on various contrastive frameworks with different vision transformer architectures. Transfer performance in downstream tasks outperforms supervised pre-training and state-of-the-art (SOTA) self-supervised counterparts.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 18 Jan 2022.

Builds on

29 of 43 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2017Mask R-CNNcited by 7,259
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
17 more
  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  9. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  10. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  12. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  13. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  14. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  15. 2018Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentationcited by 321
  16. 2021DetCo: Unsupervised Contrastive Learning for Object Detectioncited by 205
  17. 2021Self-Supervised Learning with Swin Transformerscited by 121

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 6; 2024: 4; 2025: 3; 2026: 1.

24 papers.

Sort:

  1. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  2. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  3. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  4. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  5. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  6. 2022Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domaincited by 22
  7. 2022SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learnerscited by 21
  8. 2024MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representationscited by 19
  9. 2023Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representationscited by 18
  10. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  11. 2022SERE: Exploring Feature Self-relation for Self-supervised Transformercited by 12
  12. 2023Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimickingcited by 12
12 more
  1. 2022Towards Sustainable Self-supervised Learningcited by 8
  2. 2022Self-Supervised Learning for Building Damage Assessment from Large-scale xBD Satellite Imagery Benchmark Datasetscited by 4
  3. 2024STARS: Self-supervised Tuning for 3D Action Recognition in Skeleton Sequencescited by 3
  4. 2023Enhancing Representations through Heterogeneous Self-Supervised Learningcited by 2
  5. 2024SRA: A Novel Method to Improve Feature Embedding in Self-supervised Learning for Histopathological Imagescited by 2
  6. 2022Improving Dense Contrastive Learning with Dense Negative Pairscited by 1
  7. 2024LoDisc: Learning Global-Local Discriminative Features for Self-Supervised Fine-Grained Visual Recognitioncited by 1
  8. 2023Mask Hierarchical Features For Self-Supervised Learningcited by 0
  9. 2025Driving behavior recognition via self-discovery learningcited by 0
  10. 2025GAIA: A Foundation Model for Operational Atmospheric Dynamicscited by 0
  11. 2025Disentangling Content from Style to Overcome Shortcut Learning: A Hybrid Generative-Discriminative Learning Frameworkcited by 0
  12. 2026Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Frameworkcited by 0