# Explaining Neural Networks by Decoding Layer Activations

arXiv: https://arxiv.org/abs/2005.13630 · HTML: https://mlsanity.com/paper/2005.13630/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2020.

## Built on by (most cited first, 13 of 13)

- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Large Language Model Bias Mitigation from the Perspective of Knowledge Editing](https://mlsanity.com/paper/2405.09341/) (2024)
- [Identifying and Mitigating Social Bias Knowledge in Language Models](https://mlsanity.com/paper/2408.11843/) (2024)
- [The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks](https://mlsanity.com/paper/2405.10928/) (2024)
- [Concept-based Adversarial Attacks: Tricking Humans and Classifiers Alike](https://mlsanity.com/paper/2203.10166/) (2022)
- [Understanding and Leveraging the Learning Phases of Neural Networks](https://mlsanity.com/paper/2312.06887/) (2023)
- [SynthTree: Co-supervised Local Model Synthesis for Explainable Prediction](https://mlsanity.com/paper/2406.10962/) (2024)
- [PatchX: Explaining Deep Models by Intelligible Pattern Patches for Time-series Classification](https://mlsanity.com/paper/2102.05917/) (2021)
- [Domain Transformer: Predicting Samples of Unseen, Future Domains](https://mlsanity.com/paper/2106.06057/) (2021)
- [The learning phases in NN: From Fitting the Majority to Fitting a Few](https://mlsanity.com/paper/2202.08299/) (2022)
- [Cluster Paths: Navigating Interpretability in Neural Networks](https://mlsanity.com/paper/2510.06541/) (2025)

## Abstract (arXiv)

We present a `CLAssifier-DECoder' architecture (ClaDec) which facilitates the comprehension of the output of an arbitrary layer in a neural network (NN). It uses a decoder to transform the non-interpretable representation of the given layer to a representation that is more similar to the domain a human is familiar with. In an image recognition problem, one can recognize what information is represented by a layer by contrasting reconstructed images of ClaDec with those of a conventional auto-encoder(AE) serving as reference. We also extend ClaDec to allow the trade-off between human interpretability and fidelity. We evaluate our approach for image classification using Convolutional NNs. We show that reconstructed visualizations using encodings from a classifier capture more relevant information for classification than conventional AEs. Relevant code is available at urlhttps://github.com/JohnTailor/ClaDec