mlsanityData to 14 Oct 2026

FLatten Transformer: Vision Transformer using Focused Linear Attention

arXiv 2308.00442 ↗First posted 1 Aug 2023cs.CVcited by 171

Not on the map. Why?

Abstract (arXiv)

The quadratic computation complexity of self-attention has been a persistent challenge when applying Transformer models to vision tasks. Linear attention, on the other hand, offers a much more efficient alternative with its linear complexity by approximating the Softmax operation through carefully designed mapping functions. However, current linear attention approaches either suffer from significant performance degradation or introduce additional computation overhead from the mapping functions. In this paper, we propose a novel Focused Linear Attention module to achieve both high efficiency and expressiveness. Specifically, we first analyze the factors contributing to the performance degradation of linear attention from two perspectives: the focus ability and feature diversity. To overcome these limitations, we introduce a simple yet effective mapping function and an efficient rank restoration module to enhance the expressiveness of self-attention while maintaining low computation complexity. Extensive experiments show that our linear attention module is applicable to a variety of advanced vision Transformers, and achieves consistently improved performances on multiple benchmarks. Code is available at https://github.com/LeapLabTHU/FLatten-Transformer.

Key facts. Cited by 171 papers in the map (data through 14 Oct 2026). Builds on 57 of its 64 references that are in the map. First posted on arXiv 1 Aug 2023.

Builds on

57 of 64 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2017Mask R-CNNcited by 7,277
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  4. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  5. 2020Reformer: The Efficient Transformercited by 1,668
  6. 2020Rethinking Attention with Performerscited by 1,601
  7. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  8. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  9. 2017Random Erasing Data Augmentationcited by 1,239
  10. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  13. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  14. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  15. 2019Panoptic Feature Pyramid Networkscited by 645
  16. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  17. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  18. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  19. 2021Dynamic Neural Networks: A Surveycited by 408
  20. 2021Early Convolutions Help Transformers See Bettercited by 396
  21. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  22. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  23. 2022Vision Transformer with Deformable Attentioncited by 295
  24. 2022Neighborhood Attention Transformercited by 239
  25. 2021Mobile-Former: Bridging MobileNet and Transformercited by 235
  26. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  27. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 165
  28. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  29. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 144
  30. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  31. 2020Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classificationcited by 100
  32. 2021Luna: Linear Unified Nested Attentioncited by 96
  33. 2021On the Integration of Self-Attention and Convolutioncited by 74
  34. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  35. 2019Explicit Sparse Transformer: Concentrated Attention Through Explicit Selectioncited by 62
  36. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  37. 2023Adaptive Rotated Convolution for Rotated Object Detectioncited by 51
  38. 2022Learning to Weight Samples for Dynamic Early-exiting Networkscited by 44

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 60; 2025: 60; 2026: 45.

171 papers.

Sort:

By area:

  1. 2023SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimationcited by 117
  2. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 117
  3. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 102
  4. 2024Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detectioncited by 93
  5. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  6. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  7. 2024ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictionscited by 66
  8. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  9. 2024Bridging the Divide: Reconsidering Softmax and Linear Attentioncited by 48
  10. 2024A Survey on Transformer Compressioncited by 46
  11. 2024Hypformer: Exploring Efficient Transformer Fully in Hyperbolic Spacecited by 43
  12. 2025MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restorationcited by 36
38 more
  1. 2025Training-Free Efficient Video Generation via Dynamic Token Carvingcited by 34
  2. 2025PolaFormer: Polarity-aware Linear Attention for Vision Transformerscited by 32
  3. 2024Breaking the Low-Rank Dilemma of Linear Attentioncited by 31
  4. 2025Visual Instance-aware Prompt Tuningcited by 29
  5. 2025OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domaincited by 26
  6. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediatorscited by 24
  7. 2024P-Mamba: Marrying Perona Malik Diffusion with Mamba for Efficient Pediatric Echocardiographic Left Ventricular Segmentationcited by 21
  8. 2025Spiking Vision Transformer with Saccadic Attentioncited by 21
  9. 2025FreEformer: Frequency Enhanced Transformer for Multivariate Time Series Forecastingcited by 18
  10. 2025ViT3: Unlocking Test-Time Training in Visioncited by 18
  11. 2024SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalizationcited by 17
  12. 2026PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformerscited by 17
  13. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  14. 2025Rectifying Magnitude Neglect in Linear Attentioncited by 16
  15. 2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
  16. 2024Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentationcited by 14
  17. 2024A Lightweight Sparse Focus Transformer for Remote Sensing Image Change Captioningcited by 13
  18. 2024SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobilecited by 12
  19. 2024PyramidMamba: Rethinking Pyramid Feature Fusion with Selective Space State Model for Semantic Segmentation of Remote Sensing Imagerycited by 12
  20. 2024CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applicationscited by 12
  21. 2024Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysiscited by 12
  22. 2024ScatterFormer: Efficient Voxel Transformer with Scattered Linear Attentioncited by 11
  23. 2024A Novel State Space Model with Local Enhancement and State Sharing for Image Fusioncited by 11
  24. 2024Agent Aggregator with Mask Denoise Mechanism for Histopathology Whole Slide Image Analysiscited by 11
  25. 2024Sub-Adjacent Transformer: Improving Time Series Anomaly Detection with Reconstruction Error from Sub-Adjacent Neighborhoodscited by 10
  26. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generationcited by 10
  27. 2025ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Planscited by 10
  28. 2025ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscalingcited by 10
  29. 2025Massive Memorization with Hundreds of Trillions of Parameters for Sequential Transducer Generative Recommenderscited by 10
  30. 2026SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Normcited by 10
  31. 2023GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Valuescited by 9
  32. 2023SBCFormer: Lightweight Network Capable of Full-size ImageNet Classification at 1 FPS on Single Board Computerscited by 9
  33. 2024LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretationcited by 9
  34. 2025DAMamba: Vision State Space Model with Dynamic Adaptive Scancited by 9
  35. 2025Taming Transformer Without Using Learning Rate Warmupcited by 9
  36. 2025FSHNet: Fully Sparse Hybrid Network for 3D Object Detectioncited by 9
  37. 2024ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesiscited by 8
  38. 2025AnomalyMoE: Towards a Language-free Generalist Model for Unified Visual Anomaly Detectioncited by 8

50 most cited of 171 All 171 (JSON)