# Multi-Depth Concept Extraction for Post-Hoc Vision Encoder Explanation

arXiv: https://arxiv.org/abs/2411.19700 · HTML: https://mlsanity.com/paper/2411.19700/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 49 references that are in the map.
- First posted on arXiv 29 Nov 2024.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [CRAFT: Concept Recursive Activation FacTorization for Explainability](https://mlsanity.com/paper/2211.10154/) (2022)
- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering](https://mlsanity.com/paper/2103.17070/) (2021)
- [Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks](https://mlsanity.com/paper/1801.03454/) (2018)
- [Hypercolumns for Object Segmentation and Fine-grained Localization](https://mlsanity.com/paper/1411.5752/) (2014)
- [Explain Any Concept: Segment Anything Meets Concept-Based Explanation](https://mlsanity.com/paper/2305.10289/) (2023)
- [From Clustering to Cluster Explanations via Neural Networks](https://mlsanity.com/paper/1906.07633/) (2019)
- [Explainable Face Recognition](https://mlsanity.com/paper/2008.00916/) (2020)
- [Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces](https://mlsanity.com/paper/2212.14855/) (2022)

## Built on by (most cited first, 2 of 2)

- [Explaining deep learning for ECG using time-localized clusters](https://mlsanity.com/paper/2509.15198/) (2025)
- [Post-hoc Self-explanation of CNNs](https://mlsanity.com/paper/2603.28466/) (2026)

## Abstract (arXiv)

Explainable AI methods for vision models aim to identify the parts of the input that are important for the final prediction and subsequently relate these regions to human-understandable concepts. Here, we propose focusing solely on the encoder and relating its intermediate outputs to the input, instead. We introduce Neuro-Activated Vision Explanations (NAVE), a post-hoc, unsupervised, and architecture-agnostic (across CNNs and ViTs) method for extracting and visualizing internal representations from frozen vision model encoders. Specifically, NAVE clusters composite feature activations from multiple encoder depths to produce interpretable segmentation maps with controllable granularity, requiring no fine-tuning or architectural modifications. Through extensive experiments, we quantitatively demonstrate that NAVE's concepts align with input semantics and can be used in downstream tasks. We further demonstrate NAVE as an inspection tool by analyzing how training strategies and architectures affect encoder representations. Overall, our results establish NAVE as an effective tool for post-hoc model inspection and enhancing transparency in vision models. texttthttps://github.com/Ahcene-B/NAVE