mlsanityData to 14 Oct 2026

Exploring Long-Sequence Masked Autoencoders

arXiv 2210.07224 ↗First posted 13 Oct 2022cs.CVcited by 22

Not on the map. Why?

Abstract (arXiv)

Masked Autoencoding (MAE) has emerged as an effective approach for pre-training representations across multiple domains. In contrast to discrete tokens in natural languages, the input for image MAE is continuous and subject to additional specifications. We systematically study each input specification during the pre-training stage, and find sequence length is a key axis that further scales MAE. Our study leads to a long-sequence version of MAE with minimal changes to the original recipe, by just decoupling the mask size from the patch size. For object detection and semantic segmentation, our long-sequence MAE shows consistent gains across all the experimental setups without extra computation cost during the transfer. While long-sequence pre-training is discerned most beneficial for detection and segmentation, we also achieve strong results on ImageNet-1K classification by keeping a standard image size and only increasing the sequence length. We hope our findings can provide new insights and avenues for scaling in computer vision.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 43 of its 52 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

43 of 52 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2017Mask R-CNNcited by 7,277
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
31 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  2. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  9. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  10. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  11. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  12. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  13. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  14. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  15. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,133
  16. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  17. 2020Designing Network Design Spacescited by 960
  18. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  19. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  20. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 670
  21. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 596
  22. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  23. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  24. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  25. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  26. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  27. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  28. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 131
  29. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  30. 2022Better plain ViT baselines for ImageNet-1kcited by 98
  31. 2022The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Trainingcited by 33

Built on by

Cited by year

Citing papers per year: 2023: 14; 2024: 4; 2025: 4.

22 papers.

Sort:

By area:

  1. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  2. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  3. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  4. 2023Diffusion Models as Masked Autoencoderscited by 71
  5. 2024When Do We Not Need Larger Vision Models?cited by 67
  6. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 64
  7. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
  8. 2024An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixelscited by 33
  9. 2023PixMIM: Rethinking Pixel Reconstruction in Masked Image Modelingcited by 28
  10. 2023RECLIP: Resource-efficient CLIP by Training with Small Imagescited by 16
  11. 2023Contrastive Tuning: A Little Help to Make Masked Autoencoders Forgetcited by 15
  12. 2023R-MAE: Regions Meet Masked Autoencoderscited by 13
10 more
  1. 2025In Pursuit of Pixel Supervision for Visual Pre-trainingcited by 13
  2. 2024How Well Can Vision Language Models See Image Details?cited by 10
  3. 2023Understanding Self-Supervised Pretraining with Part-Aware Representation Learningcited by 7
  4. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  5. 2025Understanding and Enhancing Mask-Based Pretraining towards Universal Representationscited by 3
  6. 2023Remote Sensing Scene Classification with Masked Image Modeling (MIM)cited by 2
  7. 2023Efficient Masked Autoencoders with Self-Consistencycited by 2
  8. 2025A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capabilitycited by 1
  9. 2024A Study on Context Length and Efficient Transformers for Biomedical Image Analysiscited by 0
  10. 2025Masked Autoencoders for Ultrasound Signals: Robust Representation Learning for Downstream Applicationscited by 0