mlsanityData to 7 Oct 2026

Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models

arXiv 2605.20839 ↗First posted 20 May 2026cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern vision backbones treat pointwise activations (e.g., ReLU, GELU) and exponential softmax as essential sources of nonlinearity, but we demonstrate they are not required within MetaFormer-style vision backbones. We design activation-free polynomial alternatives for three core primitives (MLPs, convolutions, and attention), where Hadamard products replace standard nonlinearities to yield polynomial functions of the input. These modules integrate seamlessly into existing architectures: instantiated within MetaFormer, a modular framework for vision backbones, our PolyNeXt models match or exceed activation-based counterparts across model scales on ImageNet classification, ADE20K semantic segmentation, and out-of-distribution robustness. We also substantially outperform prior polynomial networks at reduced computational cost, showing that polynomial variants of standard modules beat complex custom architectures.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 50 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

40 of 50 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2017Squeeze-and-Excitation Networkscited by 4,763
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2020GLU Variants Improve Transformercited by 1,724
28 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  3. 2019Natural Adversarial Examplescited by 1,401
  4. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  5. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  6. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  7. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  8. 2021Going deeper with Image Transformerscited by 759
  9. 2016Language Modeling with Gated Convolutional Networkscited by 688
  10. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  11. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  12. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  13. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  14. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  15. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  16. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  17. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 197
  18. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 150
  19. 2022Understanding The Robustness in Vision Transformerscited by 144
  20. 2022MetaFormer Baselines for Visioncited by 131
  21. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  22. 2023TransNeXt: Robust Foveal Visual Perception for Vision Transformerscited by 61
  23. 2024Rewrite the Starscited by 55
  24. 2023Regularization of polynomial networks for image recognitioncited by 13
  25. 2024Multilinear Operator Networkscited by 12
  26. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  27. 2025Deep Tree Tensor Networkscited by 4
  28. 2025UniConvNet: Expanding Effective Receptive Field while Maintaining Asymptotically Gaussian Distribution for ConvNets of Any Scalecited by 3

Built on by

Not cited yet.