mlsanityData to 7 Oct 2026

Atlas: Multi-Scale Attention Improves Long Context Image Modeling

arXiv 2503.12355 ↗First posted 16 Mar 2025cs.CV cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Efficiently modeling massive images is a long-standing challenge in machine learning. To this end, we introduce Multi-Scale Attention (MSA). MSA relies on two key ideas, (i) multi-scale representations (ii) bi-directional cross-scale communication. MSA creates O(log N) scales to represent the image across progressively coarser features and leverages cross-attention to propagate information across scales. We then introduce Atlas, a novel neural network architecture based on MSA. We demonstrate that Atlas significantly improves the compute-performance tradeoff of long-context image modeling in a high-resolution variant of ImageNet 100. At 1024px resolution, Atlas-B achieves 91.04% accuracy, comparable to ConvNext-B (91.92%) while being 4.3x faster. Atlas is 2.95x faster and 7.38% better than FasterViT, 2.25x faster and 4.96% better than LongViT. In comparisons against MambaVision-S, we find Atlas-S achieves 5%, 16% and 32% higher accuracy at 1024px, 2048px and 4096px respectively, while obtaining similar runtimes. Code for reproducing our experiments and pretrained models is available at https://github.com/yalalab/atlas.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 16 Mar 2025.

Builds on

26 of 32 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
14 more
  1. 2019Contrastive Multiview Codingcited by 1,297
  2. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  3. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  4. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  5. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  6. 2021Early Convolutions Help Transformers See Bettercited by 395
  7. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  8. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  9. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  10. 2021Visformer: The Vision-friendly Transformercited by 137
  11. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  12. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  13. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  14. 2023When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathologycited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Pillar-0: A New Frontier for Radiology Foundation Modelscited by 24
  2. 2026Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learningcited by 4
  3. 2026Enhancing Fine-Grained Spatial Grounding in 3D CT Report Generation via Discriminative Guidancecited by 3
  4. 2026UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretationcited by 0