# INSIDE: Steering Spatial Attention with Non-Imaging Information in CNNs

arXiv: https://arxiv.org/abs/2008.10418 · HTML: https://mlsanity.com/paper/2008.10418/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2020.

## Built on by (most cited first, 6 of 6)

- [Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transform](https://mlsanity.com/paper/2107.05990/) (2021)
- [Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization](https://mlsanity.com/paper/2102.06315/) (2021)
- [Q-space Conditioned Translation Networks for Directional Synthesis of Diffusion Weighted Images from Multi-modal Structural MRI](https://mlsanity.com/paper/2106.13188/) (2021)
- [Rethinking Generalization: The Impact of Annotation Style on Medical Image Segmentation](https://mlsanity.com/paper/2210.17398/) (2022)
- [Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers](https://mlsanity.com/paper/2202.06076/) (2022)
- [Let Me DeCode You: Decoder Conditioning with Tabular Data](https://mlsanity.com/paper/2407.09437/) (2024)

## Abstract (arXiv)

We consider the problem of integrating non-imaging information into segmentation networks to improve performance. Conditioning layers such as FiLM provide the means to selectively amplify or suppress the contribution of different feature maps in a linear fashion. However, spatial dependency is difficult to learn within a convolutional paradigm. In this paper, we propose a mechanism to allow for spatial localisation conditioned on non-imaging information, using a feature-wise attention mechanism comprising a differentiable parametrised function (e.g. Gaussian), prior to applying the feature-wise modulation. We name our method INstance modulation with SpatIal DEpendency (INSIDE). The conditioning information might comprise any factors that relate to spatial or spatio-temporal information such as lesion location, size, and cardiac cycle phase. Our method can be trained end-to-end and does not require additional supervision. We evaluate the method on two datasets: a new CLEVR-Seg dataset where we segment objects based on location, and the ACDC dataset conditioned on cardiac phase and slice location within the volume. Code and the CLEVR-Seg dataset are available at https://github.com/jacenkow/inside.