# Diverse feature visualizations reveal invariances in early layers of deep neural networks

arXiv: https://arxiv.org/abs/1807.10589 · HTML: https://mlsanity.com/paper/1807.10589/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2018.

## Built on by (most cited first, 8 of 8)

- [High Fidelity Visualization of What Your Self-Supervised Representation Knows About](https://mlsanity.com/paper/2112.09164/) (2021)
- [How Well do Feature Visualizations Support Causal Understanding of CNN Activations?](https://mlsanity.com/paper/2106.12447/) (2021)
- [MedBN: Robust Test-Time Adaptation against Malicious Test Samples](https://mlsanity.com/paper/2403.19326/) (2024)
- [Normalizing Flow based Hidden Markov Models for Classification of Speech Phones with Explainability](https://mlsanity.com/paper/2107.00730/) (2021)
- [Most discriminative stimuli for functional cell type clustering](https://mlsanity.com/paper/2401.05342/) (2023)
- [Invariance Measures for Neural Networks](https://mlsanity.com/paper/2310.17404/) (2023)
- [Stretching Beyond the Obvious: A Gradient-Free Framework to Unveil the Hidden Landscape of Visual Invariance](https://mlsanity.com/paper/2506.17040/) (2025)
- [ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers](https://mlsanity.com/paper/2606.21947/) (2026)

## Abstract (arXiv)

Visualizing features in deep neural networks (DNNs) can help understanding their computations. Many previous studies aimed to visualize the selectivity of individual units by finding meaningful images that maximize their activation. However, comparably little attention has been paid to visualizing to what image transformations units in DNNs are invariant. Here we propose a method to discover invariances in the responses of hidden layer units of deep neural networks. Our approach is based on simultaneously searching for a batch of images that strongly activate a unit while at the same time being as distinct from each other as possible. We find that even early convolutional layers in VGG-19 exhibit various forms of response invariance: near-perfect phase invariance in some units and invariance to local diffeomorphic transformations in others. At the same time, we uncover representational differences with ResNet-50 in its corresponding layers. We conclude that invariance transformations are a major computational component learned by DNNs and we provide a systematic method to study them.