# Revisiting Spatial Invariance with Low-Rank Local Connectivity

arXiv: https://arxiv.org/abs/2002.02959 · HTML: https://mlsanity.com/paper/2002.02959/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2020.

## Built on by (most cited first, 29 of 29)

- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](https://mlsanity.com/paper/2103.12731/) (2021)
- [Approximately Equivariant Networks for Imperfectly Symmetric Dynamics](https://mlsanity.com/paper/2201.11969/) (2022)
- [Rethinking Spatial Invariance of Convolutional Networks for Object Counting](https://mlsanity.com/paper/2206.05253/) (2022)
- [TAda! Temporally-Adaptive Convolutions for Video Understanding](https://mlsanity.com/paper/2110.06178/) (2021)
- [Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution](https://mlsanity.com/paper/2310.02299/) (2023)
- [Spatially Covariant Image Registration with Text Prompts](https://mlsanity.com/paper/2311.15607/) (2023)
- [Spatial Mixture-of-Experts](https://mlsanity.com/paper/2211.13491/) (2022)
- [Towards Biologically Plausible Convolutional Networks](https://mlsanity.com/paper/2106.13031/) (2021)
- [Spatially Covariant Lesion Segmentation](https://mlsanity.com/paper/2301.07895/) (2023)
- [RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks](https://mlsanity.com/paper/2304.03198/) (2023)
- [On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory](https://mlsanity.com/paper/2412.11521/) (2024)
- [Resolution Enhancement Processing on Low Quality Images Using Swin Transformer Based on Interval Dense Connection Strategy](https://mlsanity.com/paper/2303.09190/) (2023)
- [Equivariance-aware Architectural Optimization of Neural Networks](https://mlsanity.com/paper/2210.05484/) (2022)
- [Temporally-Adaptive Models for Efficient Video Understanding](https://mlsanity.com/paper/2308.05787/) (2023)
- [A Regularization-Guided Equivariant Approach for Image Restoration](https://mlsanity.com/paper/2505.19799/) (2025)
- [Improving Sample Efficiency of Value Based Models Using Attention and Vision Transformers](https://mlsanity.com/paper/2202.00710/) (2022)
- [R2Det: Exploring Relaxed Rotation Equivariance in 2D object detection](https://mlsanity.com/paper/2408.11760/) (2024)
- [Modularity in Deep Learning: A Survey](https://mlsanity.com/paper/2310.01154/) (2023)
- [SENTINEL: Securing Indoor Localization against Adversarial Attacks with Capsule Neural Networks](https://mlsanity.com/paper/2407.11091/) (2024)
- [A Circular Argument : Does RoPE need to be Equivariant for Vision?](https://mlsanity.com/paper/2511.08368/) (2025)
- [DnSwin: Toward Real-World Denoising via Continuous Wavelet Sliding-Transformer](https://mlsanity.com/paper/2207.13861/) (2022)
- [SIDDA: SInkhorn Dynamic Domain Adaptation for Image Classification with Equivariant Neural Networks](https://mlsanity.com/paper/2501.14048/) (2025)
- [Studying inductive biases in image classification task](https://mlsanity.com/paper/2210.17141/) (2022)
- [STF: Spatio-Temporal Fusion Module for Improving Video Object Detection](https://mlsanity.com/paper/2402.10752/) (2024)
- [Symmetry From Scratch: Group Equivariance as a Supervised Learning Task](https://mlsanity.com/paper/2410.03989/) (2024)
- [Group-Equivariant Poincar'e Convolutional Networks](https://mlsanity.com/paper/2607.00556/) (2026)
- [The Computational Value of Sensory-Aligned Receptive Fields Depends on Neuronal Expressivity](https://mlsanity.com/paper/2609.26940/) (2026)

## Abstract (arXiv)

Convolutional neural networks are among the most successful architectures in deep learning with this success at least partially attributable to the efficacy of spatial invariance as an inductive bias. Locally connected layers, which differ from convolutional layers only in their lack of spatial invariance, usually perform poorly in practice. However, these observations still leave open the possibility that some degree of relaxation of spatial invariance may yield a better inductive bias than either convolution or local connectivity. To test this hypothesis, we design a method to relax the spatial invariance of a network layer in a controlled manner; we create a textitlow-rank locally connected layer, where the filter bank applied at each position is constructed as a linear combination of basis set of filter banks with spatially varying combining weights. By varying the number of basis filter banks, we can control the degree of relaxation of spatial invariance. In experiments with small convolutional networks, we find that relaxing spatial invariance improves classification accuracy over both convolution and locally connected layers across MNIST, CIFAR-10, and CelebA datasets, thus suggesting that spatial invariance may be an overly restrictive prior.