# Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagation

arXiv: https://arxiv.org/abs/2002.11018 · HTML: https://mlsanity.com/paper/2002.11018/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2020.

## Built on by (most cited first, 11 of 11)

- [Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy](https://mlsanity.com/paper/2106.13200/) (2021)
- [Revealing Hidden Context Bias in Segmentation and Object Detection through Concept-specific Explanations](https://mlsanity.com/paper/2211.11426/) (2022)
- [ECQx: Explainability-Driven Quantization for Low-Bit and Sparse DNNs](https://mlsanity.com/paper/2109.04236/) (2021)
- [Optimizing Explanations by Network Canonization and Hyperparameter Search](https://mlsanity.com/paper/2211.17174/) (2022)
- [Visualizing and Understanding Contrastive Learning](https://mlsanity.com/paper/2206.09753/) (2022)
- [Measurably Stronger Explanation Reliability via Model Canonization](https://mlsanity.com/paper/2202.06621/) (2022)
- [Explainable Online Lane Change Predictions on a Digital Twin with a Layer Normalized LSTM and Layer-wise Relevance Propagation](https://mlsanity.com/paper/2204.01292/) (2022)
- [Deep Learning Inference Frameworks Benchmark](https://mlsanity.com/paper/2210.04323/) (2022)
- [Unsupervised Model Drift Estimation with Batch Normalization Statistics for Dataset Shift Detection and Model Selection](https://mlsanity.com/paper/2107.00191/) (2021)
- [Optimizing Convolutional Neural Network Architecture](https://mlsanity.com/paper/2401.01361/) (2023)
- [When LRP Diverges from Leave-One-Out in Transformers](https://mlsanity.com/paper/2510.18810/) (2025)

## Abstract (arXiv)

The lack of transparency of neural networks stays a major break for their use. The Layerwise Relevance Propagation technique builds heat-maps representing the relevance of each input in the model s decision. The relevance spreads backward from the last to the first layer of the Deep Neural Network. Layer-wise Relevance Propagation does not manage normalization layers, in this work we suggest a method to include normalization layers. Specifically, we build an equivalent network fusing normalization layers and convolutional or fully connected layers. Heatmaps obtained with our method on MNIST and CIFAR 10 datasets are more accurate for convolutional layers. Our study also prevents from using Layerwise Relevance Propagation with networks including a combination of connected layers and normalization layer.