# ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attention

arXiv: https://arxiv.org/abs/2405.18425 · HTML: https://mlsanity.com/paper/2405.18425/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 105 of its 116 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 30 of 105)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)

## Built on by (most cited first, 13 of 13)

- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation](https://mlsanity.com/paper/2502.13145/) (2025)
- [InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models](https://mlsanity.com/paper/2512.08829/) (2025)
- [MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modeling](https://mlsanity.com/paper/2503.13440/) (2025)
- [DragNeXt: Rethinking Drag-Based Image Editing](https://mlsanity.com/paper/2506.07611/) (2025)
- [EMOv2: Pushing 5M Vision Model Frontier](https://mlsanity.com/paper/2412.06674/) (2024)
- [SAGA: Selective Adaptive Gating for Efficient and Expressive Linear Attention](https://mlsanity.com/paper/2509.12817/) (2025)
- [HAMSA: Scanning-Free Vision State Space Models via SpectralPulseNet](https://mlsanity.com/paper/2604.14724/) (2026)
- [Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attention](https://mlsanity.com/paper/2506.21137/) (2025)
- [Efficient Inter-Task Attention for Multitask Transformer Models](https://mlsanity.com/paper/2508.04422/) (2025)
- [MirrorLA: Reflecting Feature Map for Vision Linear Attention](https://mlsanity.com/paper/2602.04346/) (2026)
- [Self-Attention And Beyond the Infinite: Towards Linear Transformers with Infinite Self-Attention](https://mlsanity.com/paper/2603.00175/) (2026)
- [GTR: Gated Token Recurrence for Efficient Dense Prediction](https://mlsanity.com/paper/2609.26590/) (2026)

## Abstract (arXiv)

Recently, linear complexity sequence modeling networks have achieved modeling capabilities similar to Vision Transformers on a variety of computer vision tasks, while using fewer FLOPs and less memory. However, their advantage in terms of actual runtime speed is not significant. To address this issue, we introduce Gated Linear Attention (GLA) for vision, leveraging its superior hardware-awareness and efficiency. We propose direction-wise gating to capture 1D global context through bidirectional modeling and a 2D gating locality injection to adaptively inject 2D local details into 1D global context. Our hardware-aware implementation further merges forward and backward scanning into a single kernel, enhancing parallelism and reducing memory cost and latency. The proposed model, ViG, offers a favorable trade-off in accuracy, parameters, and FLOPs on ImageNet and downstream tasks, outperforming popular Transformer and CNN-based models. Notably, ViG-S matches DeiT-B's accuracy while using only 27% of the parameters and 20% of the FLOPs, running 2× faster on 224×224 images. At 1024×1024 resolution, ViG-T uses 5.2× fewer FLOPs, saves 90% GPU memory, runs 4.8× faster, and achieves 20.7% higher top-1 accuracy than DeiT-T. These results position ViG as an efficient and scalable solution for visual representation learning. Code is available at urlhttps://github.com/hustvl/ViG.