# Understanding when spatial transformer networks do not support invariance, and what to do about it

arXiv: https://arxiv.org/abs/2004.11678 · HTML: https://mlsanity.com/paper/2004.11678/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 6 of 6)

- [RIC-CNN: Rotation-Invariant Coordinate Convolutional Neural Network](https://mlsanity.com/paper/2211.11812/) (2022)
- [Riesz networks: scale invariant neural networks in a single forward pass](https://mlsanity.com/paper/2305.04665/) (2023)
- [Geometrically Constrained and Token-Based Probabilistic Spatial Transformers](https://mlsanity.com/paper/2509.11218/) (2025)
- [Just a Matter of Scale? Reevaluating Scale Equivariance in Convolutional Neural Networks](https://mlsanity.com/paper/2211.10288/) (2022)
- [Rethinking Rotation-Invariant Recognition of Fine-grained Shapes from the Perspective of Contour Points](https://mlsanity.com/paper/2503.10992/) (2025)
- [Tailored Transformation Invariance for Industrial Anomaly Detection](https://mlsanity.com/paper/2509.17670/) (2025)

## Abstract (arXiv)

Spatial transformer networks (STNs) were designed to enable convolutional neural networks (CNNs) to learn invariance to image transformations. STNs were originally proposed to transform CNN feature maps as well as input images. This enables the use of more complex features when predicting transformation parameters. However, since STNs perform a purely spatial transformation, they do not, in the general case, have the ability to align the feature maps of a transformed image with those of its original. STNs are therefore unable to support invariance when transforming CNN feature maps. We present a simple proof for this and study the practical implications, showing that this inability is coupled with decreased classification accuracy. We therefore investigate alternative STN architectures that make use of complex features. We find that while deeper localization networks are difficult to train, localization networks that share parameters with the classification network remain stable as they grow deeper, which allows for higher classification accuracy on difficult datasets. Finally, we explore the interaction between localization network complexity and iterative image alignment.