# Deephys: Deep Electrophysiology, Debugging Neural Networks under Distribution Shifts

arXiv: https://arxiv.org/abs/2303.11912 · HTML: https://mlsanity.com/paper/2303.11912/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 17 Mar 2023.

## Builds on (most cited first, 24 of 24)

- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Debugging Tests for Model Explanations](https://mlsanity.com/paper/2011.05429/) (2020)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation](https://mlsanity.com/paper/1906.02611/) (2019)
- [Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration](https://mlsanity.com/paper/2107.05446/) (2021)
- [A Framework for Learning Ante-hoc Explainable Models via Concepts](https://mlsanity.com/paper/2108.11761/) (2021)
- [Feature Shift Detection: Localizing Which Features Have Shifted via Conditional Distribution Tests](https://mlsanity.com/paper/2107.06929/) (2021)
- [On Invariance and Selectivity in Representation Learning](https://mlsanity.com/paper/1503.05938/) (2015)
- [Concept-based Explanations for Out-Of-Distribution Detectors](https://mlsanity.com/paper/2203.02586/) (2022)
- [Robustness to Transformations Across Categories: Is Robustness To Transformations Driven by Invariant Neural Representations?](https://mlsanity.com/paper/2007.00112/) (2020)

## Built on by (most cited first, 1 of 1)

- [Modularity Trumps Invariance for Compositional Robustness](https://mlsanity.com/paper/2306.09005/) (2023)

## Abstract (arXiv)

Deep Neural Networks (DNNs) often fail in out-of-distribution scenarios. In this paper, we introduce a tool to visualize and understand such failures. We draw inspiration from concepts from neural electrophysiology, which are based on inspecting the internal functioning of a neural networks by analyzing the feature tuning and invariances of individual units. Deep Electrophysiology, in short Deephys, provides insights of the DNN's failures in out-of-distribution scenarios by comparative visualization of the neural activity in in-distribution and out-of-distribution datasets. Deephys provides seamless analyses of individual neurons, individual images, and a set of set of images from a category, and it is capable of revealing failures due to the presence of spurious features and novel features. We substantiate the validity of the qualitative visualizations of Deephys thorough quantitative analyses using convolutional and transformers architectures, in several datasets and distribution shifts (namely, colored MNIST, CIFAR-10 and ImageNet).