mlsanityData to 7 Oct 2026

Axial Attention in Multidimensional Transformers

arXiv 1912.12180 ↗First posted 20 Dec 2019cs.CVcited by 350

Not on the map. Why?

Abstract (arXiv)

We propose Axial Transformers, a self-attention-based autoregressive model for images and other data organized as high dimensional tensors. Existing autoregressive models either suffer from excessively large computational resource requirements for high dimensional data, or make compromises in terms of distribution expressiveness or ease of implementation in order to decrease resource requirements. Our architecture, by contrast, maintains both full expressiveness over joint distributions over data and ease of implementation with standard deep learning frameworks, while requiring reasonable memory and computation and achieving state-of-the-art results on standard generative modeling benchmarks. Our models are based on axial attention, a simple generalization of self-attention that naturally aligns with the multiple dimensions of the tensors in both the encoding and the decoding settings. Notably the proposed structure of the layers allows for the vast majority of the context to be computed in parallel during decoding without introducing any independence assumptions. This semi-parallel structure goes a long way to making decoding from even a very large Axial Transformer broadly applicable. We demonstrate state-of-the-art results for the Axial Transformer on the ImageNet-32 and ImageNet-64 image benchmarks as well as on the BAIR Robotic Pushing video benchmark. We open source the implementation of Axial Transformers.

Key facts. Cited by 350 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 82; 2022: 73; 2023: 65; 2024: 55; 2025: 47; 2026: 28.

350 papers.

Sort:

By area:

  1. 2022Video Diffusion Modelscited by 2,280
  2. 2021ViViT: A Video Vision Transformercited by 1,610
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  5. 2021Transformers in Vision: A Surveycited by 1,009
  6. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  7. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  8. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  9. 2021A Survey of Transformerscited by 399
  10. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  11. 2021Random Feature Attentioncited by 350
  12. 2021XCiT: Cross-Covariance Image Transformerscited by 335
38 more
  1. 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 319
  2. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  3. 2021Skillful Precipitation Nowcasting using Deep Generative Models of Radarcited by 308
  4. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 300
  5. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  6. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  7. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  8. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  9. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  10. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  11. 2022Medical Image Segmentation Review: The success of U-Netcited by 190
  12. 2022Transformer Quality in Linear Timecited by 189
  13. 2024CBraMod: A Criss-Cross Brain Foundation Model for EEG Decodingcited by 188
  14. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  15. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  16. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  17. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  18. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 150
  19. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 146
  20. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  21. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 135
  22. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  23. 2021High-Fidelity Pluralistic Image Completion with Transformerscited by 134
  24. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  25. 2025Training Agents Inside of Scalable World Modelscited by 122
  26. 2021Generative Adversarial Transformerscited by 120
  27. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 117
  28. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 116
  29. 2023Retrieval meets Long Context Large Language Modelscited by 114
  30. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 112
  31. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  32. 2022ShapeFormer: Transformer-based Shape Completion via Sparse Representationcited by 106
  33. 2021Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognitioncited by 103
  34. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  35. 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101
  36. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  37. 2021Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsitycited by 98
  38. 2021Mixed Transformer U-Net For Medical Image Segmentationcited by 95

50 most cited of 350 All 350 (JSON)