mlsanityData to 14 Oct 2026

HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers

arXiv 2608.07616 ↗First posted 7 Aug 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Vision transformers face significant computational overheads in high-resolution dense prediction due to the quadratic complexity of self-attention. Linear attention offers efficiency but sacrifices local context modeling. We propose HSMLA (Hierarchical Softmax Multi-scale Linear Attention), which combines ReLU-based linear attention for global context, selective softmax refinement for critical local features, and multi-scale token representations via depthwise convolutions. HSMLA achieves superior accuracy-efficiency trade-offs: up to 4.2× inference-time speedup across dense prediction tasks, 87.3% Dice with 3.2× speedup on CT organ segmentation, and 94.2% AUC with 4.1× speedup on pathology WSI.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 45 of its 55 references that are in the map. First posted on arXiv 7 Aug 2026.

Builds on

45 of 55 references.

References · 45

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  5. 2022A ConvNet for the 2020scited by 3,579
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  8. 2016Pyramid Scene Parsing Networkcited by 2,479
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,627
33 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  5. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  6. 2024VMamba: Visual State Space Modelcited by 969
  7. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  8. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  9. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  10. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  11. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  12. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  13. 2021Random Feature Attentioncited by 354
  14. 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 320
  15. 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
  16. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  17. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  18. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  19. 2016Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challengecited by 201
  20. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 177
  21. 20192017 Robotic Instrument Segmentation Challengecited by 168
  22. 2018Online normalizer calculation for softmaxcited by 162
  23. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 151
  24. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  25. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  26. 2022ReconFormer: Accelerated MRI Reconstruction Using Recurrent Transformercited by 52
  27. 2023Recursive Generalization Transformer for Image Super-Resolutioncited by 29
  28. 2025FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximationcited by 24
  29. 2025TinyServe: Query-Aware Cache Selection for Efficient LLM Servingcited by 9
  30. 2026MKA: Memory-Keyed Attention for Efficient Long-Context Reasoningcited by 6
  31. 2025PiKV: KV Cache Management System for Mixture of Expertscited by 5
  32. 2026AdaCorrection: Adaptive Offset Cache Correction for Accurate Diffusion Transformerscited by 3
  33. 2026Accelerating Frequency Domain Diffusion Models with Error-Feedback Event-Driven Cachingcited by 1

Built on by

Not cited yet.