# Scale-invariant scale-channel networks: Deep networks that generalise to previously unseen scales

arXiv: https://arxiv.org/abs/2106.06418 · HTML: https://mlsanity.com/paper/2106.06418/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 121 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 30 of 62)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [One pixel attack for fooling deep neural networks](https://mlsanity.com/paper/1710.08864/) (2017)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data](https://mlsanity.com/paper/1807.02547/) (2018)
- [Steerable CNNs](https://mlsanity.com/paper/1612.08498/) (2016)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://mlsanity.com/paper/1612.04642/) (2016)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks](https://mlsanity.com/paper/1312.6229/) (2013)
- [Learning Steerable Filters for Rotation Equivariant CNNs](https://mlsanity.com/paper/1711.07289/) (2017)
- [Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution](https://mlsanity.com/paper/1904.05049/) (2019)
- [ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation](https://mlsanity.com/paper/1803.06815/) (2018)
- [An Analysis of Scale Invariance in Object Detection - SNIP](https://mlsanity.com/paper/1711.08189/) (2017)
- [Deep Scale-spaces: Equivariance Over Scale](https://mlsanity.com/paper/1905.11697/) (2019)
- [Scale-Equivariant Steerable Networks](https://mlsanity.com/paper/1910.11093/) (2019)

## Built on by (most cited first, 17 of 17)

- [DEVIANT: Depth EquiVarIAnt NeTwork for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.10758/) (2022)
- [Covariance properties under natural image transformations for the generalized Gaussian derivative model for visual receptive fields](https://mlsanity.com/paper/2303.09803/) (2023)
- [Riesz networks: scale invariant neural networks in a single forward pass](https://mlsanity.com/paper/2305.04665/) (2023)
- [Canonical Factors for Hybrid Neural Fields](https://mlsanity.com/paper/2308.15461/) (2023)
- [Scale-Equivariant Deep Learning for 3D Data](https://mlsanity.com/paper/2304.05864/) (2023)
- [Unified theory for joint covariance properties under geometric image transformations for spatio-temporal receptive fields according to the generalized Gaussian derivative model for visual receptive fields](https://mlsanity.com/paper/2311.10543/) (2023)
- [Riesz feature representation: scale equivariant scattering network for classification tasks](https://mlsanity.com/paper/2307.08467/) (2023)
- [Optimizing Multi-Scale Representations to Detect Effect Heterogeneity Using Earth Observation and Computer Vision: Applications to Two Anti-Poverty RCTs](https://mlsanity.com/paper/2411.02134/) (2024)
- [Scale generalisation properties of extended scale-covariant and scale-invariant Gaussian derivative networks on image datasets with spatial scaling variations](https://mlsanity.com/paper/2409.11140/) (2024)
- [Resource-Efficient Invariant Networks: Exponential Gains by Unrolled Optimization](https://mlsanity.com/paper/2203.05006/) (2022)
- [Just a Matter of Scale? Reevaluating Scale Equivariance in Convolutional Neural Networks](https://mlsanity.com/paper/2211.10288/) (2022)
- [Advancing Perception in Artificial Intelligence through Principles of Cognitive Science](https://mlsanity.com/paper/2310.08803/) (2023)
- [SRMAE: Masked Image Modeling for Scale-Invariant Deep Representations](https://mlsanity.com/paper/2308.08884/) (2023)
- [GERD: Geometric event response data generation](https://mlsanity.com/paper/2412.03259/) (2024)
- [Graph Lineages and Skeletal Graph Products](https://mlsanity.com/paper/2508.00197/) (2025)
- [Hybrid Lie semi-group and cascade structures for the generalized Gaussian derivative model for visual receptive fields](https://mlsanity.com/paper/2509.15748/) (2025)
- [Scale-invariant Gaussian derivative residual networks](https://mlsanity.com/paper/2603.02843/) (2026)

## Abstract (arXiv)

The ability to handle large scale variations is crucial for many real world visual tasks. A straightforward approach for handling scale in a deep network is to process an image at several scales simultaneously in a set of scale channels. Scale invariance can then, in principle, be achieved by using weight sharing between the scale channels together with max or average pooling over the outputs from the scale channels. The ability of such scale channel networks to generalise to scales not present in the training set over significant scale ranges has, however, not previously been explored. In this paper, we present a systematic study of this methodology by implementing different types of scale channel networks and evaluating their ability to generalise to previously unseen scales. We develop a formalism for analysing the covariance and invariance properties of scale channel networks, and explore how different design choices, unique to scaling transformations, affect the overall performance of scale channel networks. We first show that two previously proposed scale channel network designs do not generalise well to scales not present in the training set. We explain theoretically and demonstrate experimentally why generalisation fails in these cases. We then propose a new type of foveated scale channel architecture}, where the scale channels process increasingly larger parts of the image with decreasing resolution. This new type of scale channel network is shown to generalise extremely well, provided sufficient image resolution and the absence of boundary effects. Our proposed FovMax and FovAvg networks perform almost identically over a scale range of 8, also when training on single scale training data, and do also give improved performance when learning from datasets with large scale variations in the small sample regime.