# Explaining Deep Neural Networks using Unsupervised Clustering

arXiv: https://arxiv.org/abs/2007.07477 · HTML: https://mlsanity.com/paper/2007.07477/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2020.

## Built on by (most cited first, 9 of 9)

- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Nearest Class-Center Simplification through Intermediate Layers](https://mlsanity.com/paper/2201.08924/) (2022)
- [ECLAD: Extracting Concepts with Local Aggregated Descriptors](https://mlsanity.com/paper/2206.04531/) (2022)
- [A Comprehensive Survey of Image-Based Food Recognition and Volume Estimation Methods for Dietary Assessment](https://mlsanity.com/paper/2106.11776/) (2021)
- [Interpretability for Multimodal Emotion Recognition using Concept Activation Vectors](https://mlsanity.com/paper/2202.01072/) (2022)
- [Multi-Layer Confidence Scoring for Detection of Out-of-Distribution Samples, Adversarial Attacks, and In-Distribution Misclassifications](https://mlsanity.com/paper/2512.19472/) (2025)
- [The Deleuzian Representation Hypothesis](https://mlsanity.com/paper/2512.19734/) (2025)
- [On-board Telemetry Monitoring in Autonomous Satellites: Challenges and Opportunities](https://mlsanity.com/paper/2604.08424/) (2026)
- [A Convolutional Layer Activation Dimensionality Reduction for Out-of-Distribution and Adversarial Attack Detection Methods](https://mlsanity.com/paper/2608.10203/) (2026)

## Abstract (arXiv)

We propose a novel method to explain trained deep neural networks (DNNs), by distilling them into surrogate models using unsupervised clustering. Our method can be applied flexibly to any subset of layers of a DNN architecture and can incorporate low-level and high-level information. On image datasets given pre-trained DNNs, we demonstrate the strength of our method in finding similar training samples, and shedding light on the concepts the DNNs base their decisions on. Via user studies, we show that our model can improve the user trust in model's prediction.