# Unified theory for joint covariance properties under geometric image transformations for spatio-temporal receptive fields according to the generalized Gaussian derivative model for visual receptive fields

arXiv: https://arxiv.org/abs/2311.10543 · HTML: https://mlsanity.com/paper/2311.10543/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 64 references that are in the map.
- First posted on arXiv 17 Nov 2023.

## Builds on (most cited first, 21 of 21)

- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Deep Scale-spaces: Equivariance Over Scale](https://mlsanity.com/paper/1905.11697/) (2019)
- [Scale-Equivariant Steerable Networks](https://mlsanity.com/paper/1910.11093/) (2019)
- [B-Spline CNNs on Lie Groups](https://mlsanity.com/paper/1909.12057/) (2019)
- [Geometric Deep Learning and Equivariant Neural Networks](https://mlsanity.com/paper/2105.13926/) (2021)
- [DISCO: accurate Discrete Scale Convolutions](https://mlsanity.com/paper/2106.02733/) (2021)
- [Scale-covariant and scale-invariant Gaussian derivative networks](https://mlsanity.com/paper/2011.14759/) (2020)
- [Scale-invariant scale-channel networks: Deep networks that generalise to previously unseen scales](https://mlsanity.com/paper/2106.06418/) (2021)
- [Normative theory of visual receptive fields](https://mlsanity.com/paper/1701.06333/) (2017)
- [Scale Equivariant U-Net](https://mlsanity.com/paper/2210.04508/) (2022)
- [Scale-Equivariant UNet for Histopathology Image Segmentation](https://mlsanity.com/paper/2304.04595/) (2023)
- [Exploring the ability of CNNs to generalise to previously unseen scales over wide scale ranges](https://mlsanity.com/paper/2004.01536/) (2020)
- [Fully trainable Gaussian derivative convolutional layer](https://mlsanity.com/paper/2207.08424/) (2022)
- [Provably scale-covariant continuous hierarchical networks based on scale-normalized differential expressions coupled in cascade](https://mlsanity.com/paper/1905.13555/) (2019)
- [A time-causal and time-recursive scale-covariant scale-space representation of temporal signals and past time](https://mlsanity.com/paper/2202.09209/) (2022)
- [Covariance properties under natural image transformations for the generalized Gaussian derivative model for visual receptive fields](https://mlsanity.com/paper/2303.09803/) (2023)
- [Riesz networks: scale invariant neural networks in a single forward pass](https://mlsanity.com/paper/2305.04665/) (2023)
- [Time-causal and time-recursive spatio-temporal receptive fields](https://mlsanity.com/paper/1504.02648/) (2015)
- [Scale-Equivariant Deep Learning for 3D Data](https://mlsanity.com/paper/2304.05864/) (2023)
- [Riesz feature representation: scale equivariant scattering network for classification tasks](https://mlsanity.com/paper/2307.08467/) (2023)
- [Temporal scale selection in time-causal scale space](https://mlsanity.com/paper/1701.05088/) (2017)

## Built on by (most cited first, 5 of 5)

- [Scale generalisation properties of extended scale-covariant and scale-invariant Gaussian derivative networks on image datasets with spatial scaling variations](https://mlsanity.com/paper/2409.11140/) (2024)
- [Relationships between the degrees of freedom in the affine Gaussian derivative model for visual receptive fields and 2-D affine image transformations, with application to covariance properties of simple cells in the primary visual cortex](https://mlsanity.com/paper/2411.05673/) (2024)
- [Flow Equivariant Recurrent Neural Networks](https://mlsanity.com/paper/2507.14793/) (2025)
- [Hybrid Lie semi-group and cascade structures for the generalized Gaussian derivative model for visual receptive fields](https://mlsanity.com/paper/2509.15748/) (2025)
- [Scale-invariant Gaussian derivative residual networks](https://mlsanity.com/paper/2603.02843/) (2026)

## Abstract (arXiv)

The influence of natural image transformations on receptive field responses is crucial for modelling visual operations in computer vision and biological vision. In this regard, covariance properties with respect to geometric image transformations in the earliest layers of the visual hierarchy are essential for expressing robust image operations, and for formulating invariant visual operations at higher levels. This paper defines and proves a set of joint covariance properties for spatio-temporal receptive fields in terms of spatio-temporal derivative operators applied to spatio-temporally smoothed image data under compositions of spatial scaling transformations, spatial affine transformations, Galilean transformations and temporal scaling transformations. Specifically, the derived relations show how the parameters of the receptive fields need to be transformed, in order to match the output from spatio-temporal receptive fields under composed spatio-temporal image transformations. For this purpose, we also fundamentally extend the notion of scale-normalized derivatives to affine-normalized derivatives, that are computed based on spatial smoothing with affine Gaussian kernels, and analyze the covariance properties of the resulting affine-normalized derivatives for the affine group as well as for important subgroups thereof. We conclude with a geometric analysis, showing how the derived joint covariance properties make it possible to relate or match spatio-temporal receptive field responses, when observing, possibly moving, local surface patches from different views, under locally linearized perspective or projective transformations, as well as when observing different instances of spatio-temporal events, that may occur either faster or slower between different views of similar spatio-temporal events.