mlsanityData to 14 Oct 2026

Multiscale Vision Transformers

arXiv 2104.11227 ↗First posted 22 Apr 2021cs.CV cs.AI cs.LGcited by 897

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10x more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https://github.com/facebookresearch/SlowFast

Key facts. Cited by 897 papers in the map (data through 14 Oct 2026). Builds on 99 of its 120 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

99 of 120 references.

References · 99

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2017Mask R-CNNcited by 7,277
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  12. 2016Layer Normalizationcited by 5,669
38 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2014Going Deeper with Convolutionscited by 4,241
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  11. 2015Fast R-CNNcited by 2,933
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  13. 2017Non-local Neural Networkscited by 2,293
  14. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  15. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  16. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  17. 2018SlowFast Networks for Video Recognitioncited by 1,867
  18. 2017Decoupled Weight Decay Regularizationcited by 1,831
  19. 2020Reformer: The Efficient Transformercited by 1,668
  20. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  21. 2021ViViT: A Video Vision Transformercited by 1,620
  22. 2020Rethinking Attention with Performerscited by 1,601
  23. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  24. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  25. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  26. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  27. 2020Point Transformercited by 1,470
  28. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  29. 2017Random Erasing Data Augmentationcited by 1,239
  30. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  31. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  32. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  33. 2016Deep Networks with Stochastic Depthcited by 979
  34. 2020Designing Network Design Spacescited by 960
  35. 2020Object-Centric Learning with Slot Attentioncited by 876
  36. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  37. 2020PCT: Point cloud transformercited by 732
  38. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 72; 2022: 223; 2023: 241; 2024: 171; 2025: 125; 2026: 65.

897 papers.

Sort:

By area:

  1. 2022A ConvNet for the 2020scited by 3,579
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  3. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  4. 2021Video Swin Transformercited by 1,091
  5. 2022Token Merging: Your ViT But Fastercited by 1,021
  6. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  7. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  8. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  9. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  10. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  11. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  12. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
38 more
  1. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  2. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  3. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  4. 2022DeiT III: Revenge of the ViTcited by 419
  5. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 404
  6. 2021Early Convolutions Help Transformers See Bettercited by 396
  7. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  8. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 359
  9. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 350
  10. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  11. 2023ClimaX: A foundation model for weather and climatecited by 337
  12. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  13. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  14. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  15. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 311
  16. 2022Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learningcited by 308
  17. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 290
  18. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  19. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  20. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  21. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  22. 2022CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformerscited by 227
  23. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  24. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  25. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  26. 2023Video Understanding with Large Language Models: A Surveycited by 206
  27. 2021Anticipative Video Transformercited by 200
  28. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  29. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 193
  30. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  31. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 188
  32. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 184
  33. 2022SimVP: Simpler yet Better Video Predictioncited by 184
  34. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  35. 2022Language as Queries for Referring Video Object Segmentationcited by 182
  36. 2022Frozen CLIP Models are Efficient Video Learnerscited by 182
  37. 2022FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Samplingcited by 179
  38. 2023RTMPose: Real-Time Multi-Person Pose Estimation based on MMPosecited by 175

50 most cited of 897 All 897 (JSON)