# PathFinder: Discovering Decision Pathways in Deep Neural Networks

arXiv: https://arxiv.org/abs/2210.00319 · HTML: https://mlsanity.com/paper/2210.00319/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 10 references that are in the map.
- First posted on arXiv 1 Oct 2022.

## Builds on (most cited first, 6 of 6)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning](https://mlsanity.com/paper/1803.04765/) (2018)
- [Visual Interpretability for Deep Learning: a Survey](https://mlsanity.com/paper/1802.00614/) (2018)
- [Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning](https://mlsanity.com/paper/1912.08881/) (2019)

## Built on by (most cited first, 1 of 1)

- [Cluster Paths: Navigating Interpretability in Neural Networks](https://mlsanity.com/paper/2510.06541/) (2025)

## Abstract (arXiv)

Explainability is becoming an increasingly important topic for deep neural networks. Though the operation in convolutional layers is easier to understand, processing becomes opaque in fully-connected layers. The basic idea in our work is that each instance, as it flows through the layers, causes a different activation pattern in the hidden layers and in our Paths methodology, we cluster these activation vectors for each hidden layer and then see how the clusters in successive layers connect to one another as activation flows from the input layer to the output. We find that instances of the same class follow a small number of cluster sequences over the layers, which we name ``decision paths." Such paths explain how classification decisions are typically made, and also help us determine outliers that follow unusual paths. We also propose using the Sankey diagram to visualize such pathways. We validate our method with experiments on two feed-forward networks trained on MNIST and CELEB data sets, and one recurrent network trained on PenDigits.