mlsanityData to 14 Oct 2026

Siamese Image Modeling for Self-Supervised Vision Representation Learning

arXiv 2206.01204 ↗First posted 2 Jun 2022cs.CVcited by 78

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has delivered superior performance on a variety of downstream vision tasks. Two main-stream SSL frameworks have been proposed, i.e., Instance Discrimination (ID) and Masked Image Modeling (MIM). ID pulls together representations from different views of the same image, while avoiding feature collapse. It lacks spatial sensitivity, which requires modeling the local structure within each image. On the other hand, MIM reconstructs the original content given a masked image. It instead does not have good semantic alignment, which requires projecting semantically similar views into nearby representations. To address this dilemma, we observe that (1) semantic alignment can be achieved by matching different image views with strong augmentations; (2) spatial sensitivity can benefit from predicting dense representations with masked images. Driven by these analysis, we propose Siamese Image Modeling (SiameseIM), which predicts the dense representations of an augmented view, based on another masked view from the same image but with different augmentations. SiameseIM uses a Siamese network with two branches. The online branch encodes the first view, and predicts the second view's representation according to the relative positions between these two views. The target branch produces the target by encoding the second view. SiameseIM can surpass both ID and MIM on a wide range of downstream tasks, including ImageNet finetuning and linear probing, COCO and LVIS detection, and ADE20k semantic segmentation. The improvement is more significant in few-shot, long-tail and robustness-concerned scenarios. Code shall be released at https://github.com/fundamentalvision/Siamese-Image-Modeling.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 52 of its 64 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

52 of 64 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
38 more
  1. 2016Layer Normalizationcited by 5,669
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  13. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  14. 2019Natural Adversarial Examplescited by 1,408
  15. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  16. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  17. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  18. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  19. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  20. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  21. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  22. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 657
  23. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  24. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  25. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  26. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  27. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  28. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  29. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  30. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  31. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 131
  32. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  33. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  34. 2020Self-EMD: Self-Supervised Object Detection without ImageNetcited by 70
  35. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  36. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  37. 2021Revisiting the Transferability of Supervised Pretraining: an MLP Perspectivecited by 36
  38. 2021Self-Supervised Representation Learning from Flow Equivariancecited by 28

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 15; 2023: 31; 2024: 25; 2025: 2; 2026: 5.

78 papers.

Sort:

By area:

  1. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  2. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
  3. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 220
  4. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  5. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  6. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  7. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  8. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  9. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  10. 2023CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understandingcited by 48
  11. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 43
  12. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
38 more
  1. 2022A Unified View of Masked Image Modelingcited by 36
  2. 2022Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Informationcited by 31
  3. 2023Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysiscited by 26
  4. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representationscited by 25
  5. 2022CAE v2: Context Autoencoder with CLIP Targetcited by 25
  6. 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Modelscited by 24
  7. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23
  8. 2022Attentive Mask CLIPcited by 23
  9. 2024FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learningcited by 23
  10. 2022Extreme Masking for Learning Instance and Distributed Visual Representationscited by 21
  11. 2023ADDP: Learning General Representations for Image Recognition and Generation with Alternating Denoising Diffusion Processcited by 20
  12. 2024Towards Latent Masked Image Modeling for Self-Supervised Visual Representation Learningcited by 20
  13. 2023Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representationscited by 18
  14. 2024Self-Supervised Facial Representation Learning with Facial Region Awarenesscited by 17
  15. 2023DeepMIM: Deep Supervision for Masked Image Modelingcited by 16
  16. 2023Rejuvenating image-GPT as Strong Visual Representation Learnerscited by 16
  17. 2024One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representationcited by 16
  18. 2025Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learningcited by 16
  19. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  20. 2024Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervisioncited by 13
  21. 2024MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognitioncited by 13
  22. 2023FreMIM: Fourier Transform Meets Masked Image Modeling for Medical Image Segmentationcited by 11
  23. 2024SeaMo: A Season-Aware Multimodal Foundation Model for Remote Sensingcited by 11
  24. 2024Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Groundingcited by 10
  25. 2024AnatoMask: Enhancing Medical Image Segmentation with Reconstruction-guided Self-maskingcited by 10
  26. 2024Monocular 3D lane detection for Autonomous Driving: Recent Achievements, Challenges, and Outlookscited by 9
  27. 2022Towards Sustainable Self-supervised Learningcited by 8
  28. 2023CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasetscited by 8
  29. 2024Self-Supervised Pre-training with Symmetric Superimposition Modeling for Scene Text Recognitioncited by 8
  30. 2023Understanding Self-Supervised Pretraining with Part-Aware Representation Learningcited by 7
  31. 2023Inter-Instance Similarity Modeling for Contrastive Learningcited by 6
  32. 2023PersonMAE: Person Re-Identification Pre-Training with Masked AutoEncoderscited by 6
  33. 2024Paving the way toward foundation models for irregular and unaligned Satellite Image Time Seriescited by 6
  34. 2023Delving Deeper into Data Scaling in Masked Image Modelingcited by 5
  35. 2023Asymmetric Patch Sampling for Contrastive Learningcited by 5
  36. 2022Self-Supervised Learning based on Heat Equationcited by 4
  37. 2023MOMA:Distill from Self-Supervised Teacherscited by 4
  38. 2023Masked Momentum Contrastive Learning for Zero-shot Semantic Understandingcited by 4

50 most cited of 78 All 78 (JSON)