# Identifying Interpretable Visual Features in Artificial and Biological Neural Systems arXiv: https://arxiv.org/abs/2310.11431 ยท HTML: https://mlsanity.com/paper/2310.11431/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 26 of its 80 references that are in the map. - First posted on arXiv 17 Oct 2023. ## Builds on (most cited first, 26 of 26) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013) - [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023) - [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022) - [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018) - [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023) - [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023) - [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023) - [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016) - [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015) - [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016) - [Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding](https://mlsanity.com/paper/2007.10930/) (2020) - [Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs](https://mlsanity.com/paper/2003.00152/) (2020) - [On the importance of single directions for generalization](https://mlsanity.com/paper/1803.06959/) (2018) - [Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolution](https://mlsanity.com/paper/2210.08340/) (2022) - [Understanding Neural Networks via Feature Visualization: A survey](https://mlsanity.com/paper/1904.08939/) (2019) - [Identifiability of latent-variable and structural-equation models: from linear to nonlinear](https://mlsanity.com/paper/2302.02672/) (2023) - [Towards falsifiable interpretability research](https://mlsanity.com/paper/2010.12016/) (2020) - [Unsupervised Model Selection for Variational Disentangled Representation Learning](https://mlsanity.com/paper/1905.12614/) (2019) - [The Sparse Manifold Transform](https://mlsanity.com/paper/1806.08887/) (2018) - [Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons](https://mlsanity.com/paper/2006.14304/) (2020) - [Neural system identification for large populations separating "what" and "where"](https://mlsanity.com/paper/1711.02653/) (2017) - [How Well do Feature Visualizations Support Causal Understanding of CNN Activations?](https://mlsanity.com/paper/2106.12447/) (2021) - [Scale Alone Does not Improve Mechanistic Interpretability in Vision Models](https://mlsanity.com/paper/2307.05471/) (2023) - [On Simplicity and Complexity in the Brave New World of Large-Scale Neuroscience](https://mlsanity.com/paper/1503.08779/) (2015) - [The population doctrine in cognitive neuroscience](https://mlsanity.com/paper/2104.00145/) (2021) ## Built on by (most cited first, 9 of 9) - [Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders](https://mlsanity.com/paper/2411.13117/) (2024) - [From superposition to sparse codes: interpretable representations in neural networks](https://mlsanity.com/paper/2503.01824/) (2025) - [Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds](https://mlsanity.com/paper/2606.25234/) (2026) - [Superposition disentanglement of neural representations reveals hidden alignment](https://mlsanity.com/paper/2510.03186/) (2025) - [Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse Autoencoders](https://mlsanity.com/paper/2508.09363/) (2025) - [A Geometric Unification of Concept Learning with Concept Cones](https://mlsanity.com/paper/2512.07355/) (2025) - [Characterizing Universal Object Representations Across Vision Models](https://mlsanity.com/paper/2605.13675/) (2026) - [Understanding Inhibition Through Maximally Tense Images](https://mlsanity.com/paper/2406.05598/) (2024) - [Similarity of Neural Network Representations in Superposition](https://mlsanity.com/paper/2604.00208/) (2026) ## Abstract (arXiv) Single neurons in neural networks are often interpretable in that they represent individual, intuitively meaningful features. However, many neurons exhibit textitmixed selectivity, i.e., they represent multiple unrelated features. A recent hypothesis proposes that features in deep networks may be represented in textitsuperposition, i.e., on non-orthogonal axes by multiple neurons, since the number of possible interpretable features in natural data is generally larger than the number of neurons in a given network. Accordingly, we should be able to find meaningful directions in activation space that are not aligned with individual neurons. Here, we propose (1) an automated method for quantifying visual interpretability that is validated against a large database of human psychophysics judgments of neuron interpretability, and (2) an approach for finding meaningful directions in network activation space. We leverage these methods to discover directions in convolutional neural networks that are more intuitively meaningful than individual neurons, as we confirm and investigate in a series of analyses. Moreover, we apply the same method to three recent datasets of visual neural responses in the brain and find that our conclusions largely transfer to real neural data, suggesting that superposition might be deployed by the brain. This also provides a link with disentanglement and raises fundamental questions about robust, efficient and factorized representations in both artificial and biological neural systems.