# EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction

arXiv: https://arxiv.org/abs/2205.14756 · HTML: https://mlsanity.com/paper/2205.14756/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 69 references that are in the map.
- First posted on arXiv 29 May 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)

## Built on by (most cited first, 30 of 42)

- [AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One](https://mlsanity.com/paper/2312.06709/) (2023)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [ACE-Step: A Step Towards Music Generation Foundation Model](https://mlsanity.com/paper/2506.00045/) (2025)
- [U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers](https://mlsanity.com/paper/2405.02730/) (2024)
- [Pack and Force Your Memory: Long-form and Consistent Video Generation](https://mlsanity.com/paper/2510.01784/) (2025)
- [PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation](https://mlsanity.com/paper/2410.01680/) (2024)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [UNIT: Unifying Image and Text Recognition in One Vision Encoder](https://mlsanity.com/paper/2409.04095/) (2024)
- [Onboard Satellite Image Classification for Earth Observation: A Comparative Study of ViT Models](https://mlsanity.com/paper/2409.03901/) (2024)
- [Exploring the design space of deep-learning-based weather forecasting systems](https://mlsanity.com/paper/2410.07472/) (2024)
- [LOGCAN++: Adaptive Local-global class-aware network for semantic segmentation of remote sensing imagery](https://mlsanity.com/paper/2406.16502/) (2024)
- [LiVOS: Light Video Object Segmentation with Gated Linear Matching](https://mlsanity.com/paper/2411.02818/) (2024)
- [Lite-SAM Is Actually What You Need for Segment Everything](https://mlsanity.com/paper/2407.08965/) (2024)
- [PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer](https://mlsanity.com/paper/2407.11306/) (2024)
- [RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers](https://mlsanity.com/paper/2505.21847/) (2025)
- [Representation Shift: Unifying Token Compression with FlashAttention](https://mlsanity.com/paper/2508.00367/) (2025)
- [Tiny Models are the Computational Saver for Large Models](https://mlsanity.com/paper/2403.17726/) (2024)
- [ToSA: Token Selective Attention for Efficient Vision Transformers](https://mlsanity.com/paper/2406.08816/) (2024)
- [MangoLeafViT: Leveraging Lightweight Vision Transformer with Runtime Augmentation for Efficient Mango Leaf Disease Classification](https://mlsanity.com/paper/2505.23961/) (2025)
- [GIST: Gauge-Invariant Spectral Transformers for Scalable Graph Neural Operators](https://mlsanity.com/paper/2603.16849/) (2026)
- [LoG-CAN: local-global Class-aware Network for semantic segmentation of remote sensing images](https://mlsanity.com/paper/2303.07747/) (2023)
- [SqueezeSAM: User friendly mobile interactive segmentation](https://mlsanity.com/paper/2312.06736/) (2023)
- [A Study on Inference Latency for Vision Transformers on Mobile Devices](https://mlsanity.com/paper/2510.25166/) (2025)
- [DeltaSeg: Tiered Attention and Deep Delta Learning for Multi-Class Structural Defect Segmentation](https://mlsanity.com/paper/2604.18745/) (2026)
- [ATT-CR: Adaptive Triangular Transformer for Cloud Removal](https://mlsanity.com/paper/2606.05999/) (2026)
- [Efficient and High-Quality Depth Estimation via Pixel-Space Diffusion with Linear Attention](https://mlsanity.com/paper/2608.30129/) (2026)
- [Reinforcement Learning as a Parsimonious Alternative to Prediction Cascades: A Case Study on Image Segmentation](https://mlsanity.com/paper/2402.11760/) (2024)
- [CFPFormer: Feature-pyramid like Transformer Decoder for Segmentation and Detection](https://mlsanity.com/paper/2404.15451/) (2024)
- [A Cascaded Dilated Convolution Approach for Mpox Lesion Classification](https://mlsanity.com/paper/2412.10106/) (2024)
- [ORXE: Orchestrating Experts for Dynamically Configurable Efficiency](https://mlsanity.com/paper/2505.04850/) (2025)

## Abstract (arXiv)

High-resolution dense prediction enables many appealing real-world applications, such as computational photography, autonomous driving, etc. However, the vast computational cost makes deploying state-of-the-art high-resolution dense prediction models on hardware devices difficult. This work presents EfficientViT, a new family of high-resolution vision models with novel multi-scale linear attention. Unlike prior high-resolution dense prediction models that rely on heavy softmax attention, hardware-inefficient large-kernel convolution, or complicated topology structure to obtain good performances, our multi-scale linear attention achieves the global receptive field and multi-scale learning (two desirable features for high-resolution dense prediction) with only lightweight and hardware-efficient operations. As such, EfficientViT delivers remarkable performance gains over previous state-of-the-art models with significant speedup on diverse hardware platforms, including mobile CPU, edge GPU, and cloud GPU. Without performance loss on Cityscapes, our EfficientViT provides up to 13.9× and 6.2× GPU latency reduction over SegFormer and SegNeXt, respectively. For super-resolution, EfficientViT delivers up to 6.4x speedup over Restormer while providing 0.11dB gain in PSNR. For Segment Anything, EfficientViT delivers 48.9x higher throughput on A100 GPU while achieving slightly better zero-shot instance segmentation performance on COCO.