# A Detailed Study of Interpretability of Deep Neural Network based Top Taggers

arXiv: https://arxiv.org/abs/2210.04371 · HTML: https://mlsanity.com/paper/2210.04371/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 91 references that are in the map.
- First posted on arXiv 9 Oct 2022.

## Builds on (most cited first, 30 of 34)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Explainability in Graph Neural Networks: A Taxonomic Survey](https://mlsanity.com/paper/2012.15445/) (2020)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Model-Agnostic Interpretability of Machine Learning](https://mlsanity.com/paper/1606.05386/) (2016)
- [Higher-Order Explanations of Graph Neural Networks via Relevant Walks](https://mlsanity.com/paper/2006.03589/) (2020)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [Lorentz Group Equivariant Neural Network for Particle Physics](https://mlsanity.com/paper/2006.04780/) (2020)
- [Particle Transformer for Jet Tagging](https://mlsanity.com/paper/2202.03772/) (2022)
- [Reducing Overfitting in Deep Networks by Decorrelating Representations](https://mlsanity.com/paper/1511.06068/) (2015)
- [Energy Flow Networks: Deep Sets for Particle Jets](https://mlsanity.com/paper/1810.05165/) (2018)
- [Explainable Artificial Intelligence: a Systematic Review](https://mlsanity.com/paper/2006.00093/) (2020)
- [How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations](https://mlsanity.com/paper/2101.08758/) (2021)
- [Distance-Weighted Graph Neural Networks on FPGAs for Real-Time Particle Reconstruction in High Energy Physics](https://mlsanity.com/paper/2008.03601/) (2020)
- [Variational AutoEncoder For Regression: Application to Brain Aging Analysis](https://mlsanity.com/paper/1904.05948/) (2019)
- [Accelerated Charged Particle Tracking with Graph Neural Networks on FPGAs](https://mlsanity.com/paper/2012.01563/) (2020)
- [QCD-Aware Recursive Neural Networks for Jet Physics](https://mlsanity.com/paper/1702.00748/) (2017)
- [GLSR-VAE: Geodesic Latent Space Regularization for Variational AutoEncoder Architectures](https://mlsanity.com/paper/1707.04588/) (2017)
- [Better Latent Spaces for Better Autoencoders](https://mlsanity.com/paper/2104.08291/) (2021)
- [Bump Hunting in Latent Space](https://mlsanity.com/paper/2103.06595/) (2021)
- [Jet Constituents for Deep Neural Network Based Top Quark Tagging](https://mlsanity.com/paper/1704.02124/) (2017)
- [Explainable AI for ML jet taggers using expert variables and layerwise relevance propagation](https://mlsanity.com/paper/2011.13466/) (2020)
- [Long Short-Term Memory (LSTM) networks with jet constituents for boosted top tagging at the LHC](https://mlsanity.com/paper/1711.09059/) (2017)
- [Explaining machine-learned particle-flow reconstruction](https://mlsanity.com/paper/2111.12840/) (2021)
- [Bridging the Gap Between Explainable AI and Uncertainty Quantification to Enhance Trustability](https://mlsanity.com/paper/2105.11828/) (2021)
- [Snowmass 2021 Computational Frontier CompF03 Topical Group Report: Machine Learning](https://mlsanity.com/paper/2209.07559/) (2022)

## Built on by (most cited first, 7 of 7)

- [Interpretable Machine Learning in Physics: A Review](https://mlsanity.com/paper/2503.23616/) (2025)
- [FAIR AI Models in High Energy Physics](https://mlsanity.com/paper/2212.05081/) (2022)
- [Feature Selection with Distance Correlation](https://mlsanity.com/paper/2212.00046/) (2022)
- [Evidential Deep Learning for Uncertainty Quantification and Out-of-Distribution Detection in Jet Identification using Deep Neural Networks](https://mlsanity.com/paper/2501.05656/) (2025)
- [Interpretability of an Interaction Network for identifying H rightarrow bbarb jets](https://mlsanity.com/paper/2211.12770/) (2022)
- [Dissecting Jet-Tagger Through Mechanistic Interpretability](https://mlsanity.com/paper/2605.09881/) (2026)
- [Local Diagnostics of Learned Event Representations for Anomaly Detection at the HL-LHC](https://mlsanity.com/paper/2610.13437/) (2026)

## Abstract (arXiv)

Recent developments in the methods of explainable AI (XAI) allow researchers to explore the inner workings of deep neural networks (DNNs), revealing crucial information about input-output relationships and realizing how data connects with machine learning models. In this paper we explore interpretability of DNN models designed to identify jets coming from top quark decay in high energy proton-proton collisions at the Large Hadron Collider (LHC). We review a subset of existing top tagger models and explore different quantitative methods to identify which features play the most important roles in identifying the top jets. We also investigate how and why feature importance varies across different XAI metrics, how correlations among features impact their explainability, and how latent space representations encode information as well as correlate with physically meaningful quantities. Our studies uncover some major pitfalls of existing XAI methods and illustrate how they can be overcome to obtain consistent and meaningful interpretation of these models. We additionally illustrate the activity of hidden layers as Neural Activation Pattern (NAP) diagrams and demonstrate how they can be used to understand how DNNs relay information across the layers and how this understanding can help to make such models significantly simpler by allowing effective model reoptimization and hyperparameter tuning. These studies not only facilitate a methodological approach to interpreting models but also unveil new insights about what these models learn. Incorporating these observations into augmented model design, we propose the Particle Flow Interaction Network (PFIN) model and demonstrate how interpretability-inspired model augmentation can improve top tagging performance.