Alias-Free ViT: Fractional Shift Invariance via Linear Attention
Not on the map. Why?
Abstract (arXiv)
Transformers have emerged as a competitive alternative to convnets in vision tasks, yet they lack the architectural inductive bias of convnets, which may hinder their potential performance. Specifically, Vision Transformers (ViTs) are not translation-invariant and are more sensitive to minor image translations than standard convnets. Previous studies have shown, however, that convnets are also not perfectly shift-invariant, due to aliasing in downsampling and nonlinear layers. Consequently, anti-aliasing approaches have been proposed to certify convnets' translation robustness. Building on this line of work, we propose an Alias-Free ViT, which combines two main components. First, it uses alias-free downsampling and nonlinearities. Second, it uses linear cross-covariance attention that is shift-equivariant to both integer and fractional translations, enabling a shift-invariant global representation. Our model maintains competitive performance in image classification and outperforms similar-sized models in terms of robustness to adversarial translations.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 57 of its 72 references that are in the map. First posted on arXiv 26 Oct 2025.
Builds on
57 of 72 references.
References · 57
- 2017Attention Is All You Needcited by 59,932
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2020Rethinking Attention with Performerscited by 1,584
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
- 2021Alias-Free Generative Adversarial Networkscited by 1,181
- 2018Self-Attention with Relative Position Representationscited by 1,112
- 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
- 2021Transformers in Vision: A Surveycited by 1,009
38 more
- 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
- 2021Multiscale Vision Transformerscited by 896
- 2021Going deeper with Image Transformerscited by 759
- 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
- 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
- 2021MetaFormer Is Actually What You Need for Visioncited by 523
- 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
- 2021Conditional Positional Encodings for Vision Transformerscited by 402
- 2019Making Convolutional Networks Shift-Invariant Againcited by 390
- 2021XCiT: Cross-Covariance Image Transformerscited by 335
- 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
- 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
- 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
- 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
- 2023Convolutional Neural Operators for robust and accurate learning of PDEscited by 207
- 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
- 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
- 2015Spectral Representations for Convolutional Neural Networkscited by 79
- 2020Group Equivariant Stand-Alone Self-Attention For Visioncited by 63
- 2021On the Connection between Local Attention and Dynamic Depth-wise Convolutioncited by 61
- 2020Delving Deeper into Anti-aliasing in ConvNetscited by 60
- 2023Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learningcited by 57
- 2020Truly shift-invariant convolutional neural networkscited by 53
- 2024Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularitycited by 33
- 2022SimA: Simple Softmax-free Attention for Vision Transformerscited by 32
- 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
- 2022FrequencyLowCut Pooling -- Plug & Play against Catastrophic Overfittingcited by 28
- 2023E(2)-Equivariant Vision Transformercited by 27
- 2020An Effective Anti-Aliasing Approach for Residual Networkscited by 23
- 2022Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networkscited by 18
- 2021Blending Anti-Aliasing into Vision Transformercited by 16
- 2023Making Vision Transformers Truly Shift-Equivariantcited by 16
- 2023Truly Scale-Equivariant Deep Nets with Fourier Layerscited by 16
- 2019Convolution, attention and structure embeddingcited by 15
- 2024Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasetscited by 14
- 2025Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Spacecited by 14
- 2021Anti-aliasing Deep Image Classifiers using Novel Depth Adaptive Blurring and Activation Functioncited by 11
- 2023Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activationscited by 11
50 most cited of 57 All 57 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.