mlsanityData to 14 Oct 2026

Self-Supervised Pre-training of Vision Transformers for Dense Prediction Tasks

arXiv 2205.15173 ↗First posted 30 May 2022cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

We present a new self-supervised pre-training of Vision Transformers for dense prediction tasks. It is based on a contrastive loss across views that compares pixel-level representations to global image representations. This strategy produces better local features suitable for dense prediction tasks as opposed to contrastive pre-training based on global image representation only. Furthermore, our approach does not suffer from a reduced batch size since the number of negative examples needed in the contrastive loss is in the order of the number of local features. We demonstrate the effectiveness of our pre-training strategy on two dense prediction tasks: semantic segmentation and monocular depth estimation.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 34 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

29 of 34 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
17 more
  1. 2020Supervised Contrastive Learningcited by 2,829
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  4. 2021Vision Transformers for Dense Predictioncited by 1,803
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  7. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  8. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  9. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  10. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  11. 2020Contrastive Multi-View Representation Learning on Graphscited by 652
  12. 2021ImageNet-21K Pretraining for the Massescited by 613
  13. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  14. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  15. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 403
  16. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 288
  17. 2021Transformer-Based Attention Networks for Continuous Pixel-Wise Predictioncited by 125

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Image Segmentation-based Unsupervised Multiple Objects Discoverycited by 4