# Blurring the Line Between Structure and Learning to Optimize and Adapt Receptive Fields

arXiv: https://arxiv.org/abs/1904.11487 · HTML: https://mlsanity.com/paper/1904.11487/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2019.

## Built on by (most cited first, 11 of 11)

- [FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes](https://mlsanity.com/paper/2110.08059/) (2021)
- [BlobGAN: Spatially Disentangled Scene Representations](https://mlsanity.com/paper/2205.02837/) (2022)
- [Resolution learning in deep convolutional networks using scale-space theory](https://mlsanity.com/paper/2106.03412/) (2021)
- [Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks](https://mlsanity.com/paper/2105.08714/) (2021)
- [Dilated convolution with learnable spacings](https://mlsanity.com/paper/2112.03740/) (2021)
- [Spectral Leakage and Rethinking the Kernel Size in CNNs](https://mlsanity.com/paper/2101.10143/) (2021)
- [Dilated Convolution with Learnable Spacings: beyond bilinear interpolation](https://mlsanity.com/paper/2306.00817/) (2023)
- [Scale generalisation properties of extended scale-covariant and scale-invariant Gaussian derivative networks on image datasets with spatial scaling variations](https://mlsanity.com/paper/2409.11140/) (2024)
- [Efficient Representation Learning via Adaptive Context Pooling](https://mlsanity.com/paper/2207.01844/) (2022)
- [GMConv: Modulating Effective Receptive Fields for Convolutional Kernels](https://mlsanity.com/paper/2302.04544/) (2023)
- [Deep Continuous Networks](https://mlsanity.com/paper/2402.01557/) (2024)

## Abstract (arXiv)

The visual world is vast and varied, but its variations divide into structured and unstructured factors. We compose free-form filters and structured Gaussian filters, optimized end-to-end, to factorize deep representations and learn both local features and their degree of locality. Our semi-structured composition is strictly more expressive than free-form filtering, and changes in its structured parameters would require changes in free-form architecture. In effect this optimizes over receptive field size and shape, tuning locality to the data and task. Dynamic inference, in which the Gaussian structure varies with the input, adapts receptive field size to compensate for local scale variation. Optimizing receptive field size improves semantic segmentation accuracy on Cityscapes by 1-2 points for strong dilated and skip architectures and by up to 10 points for suboptimal designs. Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.