mlsanityData to 7 Oct 2026

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

arXiv 2103.15358 ↗First posted 29 Mar 2021cs.CV cs.AI cs.LGcited by 207

Not on the map. Why?

Abstract (arXiv)

This paper presents a new Vision Transformer (ViT) architecture Multi-Scale Vision Longformer, which significantly enhances the ViT of citedosovitskiy2020image for encoding high-resolution images using two techniques. The first is the multi-scale model structure, which provides image encodings at multiple scales with manageable computational cost. The second is the attention mechanism of vision Longformer, which is a variant of Longformer citebeltagy2020longformer, originally developed for natural language processing, and achieves a linear complexity w.r.t. the number of input tokens. A comprehensive empirical study shows that the new ViT significantly outperforms several strong baselines, including the existing ViT models and their ResNet counterparts, and the Pyramid Vision Transformer from a concurrent work citewang2021pyramid, on a range of vision tasks, including image classification, object detection, and segmentation. The models and source code are released at urlhttps://github.com/microsoft/vision-longformer.

Key facts. Cited by 207 papers in the map (data through 7 Oct 2026). Builds on 55 of its 61 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

55 of 61 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  12. 2020Longformer: The Long-Document Transformercited by 3,362
38 more
  1. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  16. 2020Pre-Trained Image Processing Transformercited by 891
  17. 2020Efficient Transformers: A Surveycited by 784
  18. 2018Image Transformercited by 770
  19. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  20. 2017Focal Loss for Dense Object Detectioncited by 749
  21. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  22. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  23. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  24. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  25. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  26. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  27. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  28. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  29. 2019Axial Attention in Multidimensional Transformerscited by 350
  30. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  31. 2020Sparse Sinkhorn Attentioncited by 250
  32. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  33. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  34. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  35. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  36. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  37. 2020Learning Joint Spatial-Temporal Transformations for Video Inpaintingcited by 162
  38. 2019Blockwise Self-Attention for Long Document Understandingcited by 158

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 73; 2023: 46; 2024: 30; 2025: 7; 2026: 10.

207 papers.

Sort:

By area:

  1. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  2. 2021Grounded Language-Image Pre-trainingcited by 1,147
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  4. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  5. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  6. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  7. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  8. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  9. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  10. 2022Vision Transformer with Deformable Attentioncited by 293
  11. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  12. 2022Neighborhood Attention Transformercited by 235
38 more
  1. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  2. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  3. 2023SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagerycited by 202
  4. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  5. 2022Focal Modulation Networkscited by 186
  6. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  7. 2022DaViT: Dual Attention Vision Transformerscited by 174
  8. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  9. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  10. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  11. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139
  12. 2021CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptationcited by 129
  13. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  14. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  15. 2021A Survey of Visual Transformerscited by 117
  16. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  17. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  18. 2022Inception Transformercited by 99
  19. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  20. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  21. 2021Multi-Scale High-Resolution Vision Transformer for Semantic Segmentationcited by 92
  22. 2021Shunted Self-Attention via Multi-Scale Token Aggregationcited by 90
  23. 2021P2T: Pyramid Pooling Transformer for Scene Understandingcited by 84
  24. 2021S2-MLP: Spatial-Shift MLP Architecture for Visioncited by 83
  25. 2022Fast Vision Transformers with HiLo Attentioncited by 78
  26. 2022Boosting Crowd Counting via Multifaceted Attentioncited by 73
  27. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  28. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  29. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 64
  30. 2021On Efficient Transformer-Based Image Pre-training for Low-Level Visioncited by 64
  31. 2021On the Connection between Local Attention and Dynamic Depth-wise Convolutioncited by 61
  32. 2022Safe Self-Refinement for Transformer-based Domain Adaptationcited by 61
  33. 2022Patch-level Representation Learning for Self-supervised Vision Transformerscited by 60
  34. 2022Towards Spatio-Temporal Aware Traffic Time Series Forecasting--Full Versioncited by 59
  35. 2022Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecastingcited by 54
  36. 2022ReconFormer: Accelerated MRI Reconstruction Using Recurrent Transformercited by 52
  37. 2023Scale-Aware Modulation Meet Transformercited by 49
  38. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 49

50 most cited of 207 All 207 (JSON)