mlsanityData to 14 Oct 2026

On The Application of Linear Attention in Multimodal Transformers

arXiv 2604.10064 ↗First posted 11 Apr 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Multimodal Transformers serve as the backbone for state-of-the-art vision-language models, yet their quadratic attention complexity remains a critical barrier to scalability. In this work, we investigate the viability of Linear Attention (LA) as a high-efficiency alternative within multimodal frameworks. By integrating LA, we reduce the computational overhead from quadratic to linear relative to sequence length while preserving competitive performance. We evaluate our approach across ViT-S/16, ViT-B/16, and ViT-L/16 architectures trained on the LAION-400M dataset, with validation focused on ImageNet-21K zero-shot accuracy. Our systematic evaluation demonstrates that Linear Attention not only yields significant computational savings but also adheres to the same scaling laws as standard softmax attention. These findings position Linear Attention as a robust, scalable solution for next-generation multimodal Transformers tasked with processing increasingly large and complex datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 38 references that are in the map. First posted on arXiv 11 Apr 2026.

Builds on

36 of 38 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
24 more
  1. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  2. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  3. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  4. 2021ImageNet-21K Pretraining for the Massescited by 613
  5. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  6. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  7. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  9. 2022Multimodal Learning with Transformers: A Surveycited by 400
  10. 2020Multi-modal Transformer for Video Retrievalcited by 395
  11. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  12. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  13. 2022Transformer Quality in Linear Timecited by 191
  14. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
  15. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
  16. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 171
  17. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  18. 2022The Devil in Linear Transformercited by 98
  19. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  20. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  21. 2023TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormercited by 34
  22. 2024Multimodal Transformers for Real-Time Surgical Activity Predictioncited by 11
  23. 2024FAST: Factorizable Attention for Speeding up Transformerscited by 1
  24. 2025Transformer Based Linear Attention with Optimized GPU Kernel Implementationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025EDiT: Efficient Diffusion Transformers with Linear Compressed Attention
  2. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention
  3. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer
  4. 2025The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
  5. 2025Token Sequence Compression for Efficient Multimodal Computing
  6. 2024TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Models
  7. 2025Multimodal Language Models See Better When They Look Shallower
  8. 2025Zoomer: Adaptive Image Focus Optimization for Black-box MLLM