mlsanityData to 14 Oct 2026

Siamese Masked Autoencoders

arXiv 2305.14344 ↗First posted 23 May 2023cs.CV cs.LGcited by 76

Not on the map. Why?

Abstract (arXiv)

Establishing correspondence between images or scenes is a significant challenge in computer vision, especially given occlusions, viewpoint changes, and varying object appearances. In this paper, we present Siamese Masked Autoencoders (SiamMAE), a simple extension of Masked Autoencoders (MAE) for learning visual correspondence from videos. SiamMAE operates on pairs of randomly sampled video frames and asymmetrically masks them. These frames are processed independently by an encoder network, and a decoder composed of a sequence of cross-attention layers is tasked with predicting the missing patches in the future frame. By masking a large fraction (95%) of patches in the future frame while leaving the past frame unchanged, SiamMAE encourages the network to focus on object motion and learn object-centric representations. Despite its conceptual simplicity, features learned via SiamMAE outperform state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation tasks. SiamMAE achieves competitive results without relying on data augmentation, handcrafted tracking-based pretext tasks, or other techniques to prevent representational collapse.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Builds on 58 of its 104 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

58 of 104 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,866
38 more
  1. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2017The Kinetics Human Action Video Datasetcited by 1,909
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  5. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  8. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  9. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  10. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  11. 2016Colorful Image Colorizationcited by 1,114
  12. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  13. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  14. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  15. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  16. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  17. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 784
  18. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 568
  19. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  20. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  21. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  22. 2019Tracking without bells and whistlescited by 382
  23. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  24. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  25. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  26. 2022Unifying Flow, Stereo and Depth Estimationcited by 275
  27. 2018Fast Online Object Tracking and Segmentation: A Unifying Approachcited by 272
  28. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  29. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  30. 2020What Should Not Be Contrastive in Contrastive Learningcited by 226
  31. 2017Unsupervised Representation Learning by Sorting Sequencescited by 221
  32. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  33. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 204
  34. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  35. 2018Tracking Emerges by Colorizing Videoscited by 166
  36. 2016Learning Features by Watching Objects Movecited by 138
  37. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  38. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 28; 2025: 24; 2026: 18.

76 papers.

Sort:

By area:

  1. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  2. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  3. 2024Cross-View Completion Models are Zero-shot Correspondence Estimatorscited by 30
  4. 2024Rethinking Patch Dependence for Masked Autoencoderscited by 29
  5. 2024Masked Image Modeling: A Surveycited by 29
  6. 2024VideoMAC: Video Masked Autoencoders Meet ConvNetscited by 20
  7. 2024Efficient Image Pre-Training with Siamese Cropped Masked Autoencoderscited by 17
  8. 2024Pair-VPR: Place-Aware Pre-training and Contrastive Pair Classification for Visual Place Recognition with Vision Transformerscited by 15
  9. 2025V2-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondencecited by 14
  10. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  11. 2025Feed-Forward SceneDINO for Unsupervised Semantic Scene Completioncited by 13
  12. 2024Cross-view Masked Diffusion Transformers for Person Image Synthesiscited by 12
38 more
  1. 2024Harnessing Massive Satellite Imagery with Efficient Masked Image Modelingcited by 12
  2. 20253D Foundation Model for Generalizable Disease Detection in Head Computed Tomographycited by 12
  3. 20233D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimationcited by 11
  4. 2023T-MAE: Temporal Masked Autoencoders for Point Cloud Representation Learningcited by 11
  5. 2024PooDLe: Pooled and dense self-supervised learning from naturalistic videoscited by 11
  6. 2024SeaMo: A Season-Aware Multimodal Foundation Model for Remote Sensingcited by 11
  7. 2024Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Groundingcited by 10
  8. 2024Visual Representation Learning with Stochastic Frame Predictioncited by 10
  9. 2024Self-Supervised Any-Point Tracking by Contrastive Random Walkscited by 10
  10. 2025Scene-Centric Unsupervised Panoptic Segmentationcited by 10
  11. 2025Recurrent Video Masked Autoencoderscited by 10
  12. 2023CtxMIM: Context-Enhanced Masked Image Modeling for Remote Sensing Image Understandingcited by 9
  13. 2025Learning from Streaming Video with Orthogonal Gradientscited by 9
  14. 2025Utilizing Strategic Pre-training to Reduce Overfitting: Baguan -- A Pre-trained Weather Forecasting Modelcited by 7
  15. 2025Towards More Diverse and Challenging Pre-training for Point Cloud Learning: Self-Supervised Cross Reconstruction with Decoupled Viewscited by 7
  16. 2024Self Pre-training with Topology- and Spatiality-aware Masked Autoencoders for 3D Medical Image Segmentationcited by 6
  17. 2025Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognitioncited by 6
  18. 2023Bootstrap Masked Visual Modeling via Hard Patches Miningcited by 5
  19. 2024Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learningcited by 4
  20. 2024Walker: Self-supervised Multiple Object Tracking by Walking on Temporal Appearance Graphscited by 4
  21. 2025Developing a PET/CT Foundation Model for Cross-Modal Anatomical and Functional Imagingcited by 4
  22. 2025Token Bottleneck: One Token to Remember Dynamicscited by 4
  23. 2025InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signalscited by 3
  24. 2025Contextures: Representations from Contextscited by 3
  25. 2025PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learningcited by 3
  26. 2025Self-Guided Masked Autoencodercited by 3
  27. 2025Understanding and Enhancing Mask-Based Pretraining towards Universal Representationscited by 3
  28. 2026TrackMAE: Video Representation Learning via Track Mask and Predictcited by 3
  29. 2024Cohere3D: Exploiting Temporal Coherence for Unsupervised Representation Learning of Vision-based Autonomous Drivingcited by 2
  30. 2024Unsupervised Part Discovery via Dual Representation Alignmentcited by 2
  31. 2024Equivariant Representation Learning for Augmentation-based Self-Supervised Learning via Image Reconstructioncited by 2
  32. 2025FRAME: Pre-Training Video Feature Representations via Anticipation and Memorycited by 2
  33. 2025Deep Space Weather Model: Long-Range Solar Flare Prediction from Multi-Wavelength Imagescited by 2
  34. 2025Unique Lives, Shared World: Learning from Single-Life Videoscited by 2
  35. 2024LoDisc: Learning Global-Local Discriminative Features for Self-Supervised Fine-Grained Visual Recognitioncited by 1
  36. 2024Self-Supervised Learning for Interventional Image Analytics: Towards Robust Device Trackerscited by 1
  37. 2024Self-supervised Pretraining and Finetuning for Monocular Depth and Visual Odometrycited by 1
  38. 2024Decomposition Betters Tracking Everything Everywherecited by 1

50 most cited of 76 All 76 (JSON)