Concatenated Masked Autoencoders as Spatial-Temporal Learner
Not on the map. Why?
Abstract (arXiv)
Learning representations from videos requires understanding continuous motion and visual correspondences between frames. In this paper, we introduce the Concatenated Masked Autoencoders (CatMAE) as a spatial-temporal learner for self-supervised video representation learning. For the input sequence of video frames, CatMAE keeps the initial frame unchanged while applying substantial masking (95%) to subsequent frames. The encoder in CatMAE is responsible for encoding visible patches for each frame individually; subsequently, for each masked frame, the decoder leverages visible patches from both previous and current frames to reconstruct the original image. Our proposed method enables the model to estimate the motion information between visible patches, match the correspondences between preceding and succeeding frames, and ultimately learn the evolution of scenes. Furthermore, we propose a new data augmentation strategy, Video-Reverse (ViRe), which uses reversed video frames as the model's reconstruction targets. This further encourages the model to utilize continuous motion details and correspondences to complete the reconstruction, thereby enhancing the model's capabilities. Compared to the most advanced pre-training methods, CatMAE achieves a leading level in video segmentation tasks and action recognition tasks.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 2 Nov 2023.
Builds on
37 of 52 references.
References · 37
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
- 2020Exploring Simple Siamese Representation Learningcited by 2,866
25 more
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
- 2016Colorful Image Colorizationcited by 1,114
- 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
- 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
- 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
- 2015Unsupervised Learning of Video Representations using LSTMscited by 435
- 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
- 2015Deep multi-scale video prediction beyond mean square errorcited by 300
- 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
- 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
- 2017Unsupervised Representation Learning by Sorting Sequencescited by 221
- 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 204
- 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
- 2018Tracking Emerges by Colorizing Videoscited by 166
- 2016Learning Features by Watching Objects Movecited by 138
- 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
- 2016An Uncertain Future: Forecasting from Static Images using Variational Autoencoderscited by 87
- 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
- 2018Adaptive Temporal Encoding Network for Video Instance-level Human Parsingcited by 36
- 2019Slow Feature Analysis for Human Action Recognitioncited by 16
Built on by
Citing papers per year: 2024: 1.
1 paper.