# Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geoscience

arXiv: https://arxiv.org/abs/2202.03407 · HTML: https://mlsanity.com/paper/2202.03407/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey](https://mlsanity.com/paper/2006.11371/) (2020)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [The Shattered Gradients Problem: If resnets are the answer, then what is the question?](https://mlsanity.com/paper/1702.08591/) (2017)
- [Do Feature Attribution Methods Correctly Attribute Features?](https://mlsanity.com/paper/2104.14403/) (2021)
- [Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey](https://mlsanity.com/paper/1911.12116/) (2019)
- [Learning how to explain neural networks: PatternNet and PatternAttribution](https://mlsanity.com/paper/1705.05598/) (2017)
- [Towards Best Practice in Explaining Neural Network Decisions with LRP](https://mlsanity.com/paper/1910.09840/) (2019)
- [Machine Learning for the Geosciences: Challenges and Opportunities](https://mlsanity.com/paper/1711.04708/) (2017)
- [Towards falsifiable interpretability research](https://mlsanity.com/paper/2010.12016/) (2020)
- [A trans-disciplinary review of deep learning research for water resources scientists](https://mlsanity.com/paper/1712.02162/) (2017)
- [Physically Interpretable Neural Networks for the Geosciences: Applications to Earth System Variability](https://mlsanity.com/paper/1912.01752/) (2019)
- [Towards Robust Explanations for Deep Neural Networks](https://mlsanity.com/paper/2012.10425/) (2020)
- [A Comprehensive Review of Deep Learning Applications in Hydrology and Water Resources](https://mlsanity.com/paper/2007.12269/) (2020)
- [Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset](https://mlsanity.com/paper/2103.10005/) (2021)
- [Development and Interpretation of a Neural Network-Based Synthetic Radar Reflectivity Estimator Using GOES-R Satellite Observations](https://mlsanity.com/paper/2004.07906/) (2020)
- [Interpreting the Predictions of Complex ML Models by Layer-wise Relevance Propagation](https://mlsanity.com/paper/1611.08191/) (2016)
- [Controlling Explanatory Heatmap Resolution and Semantics via Decomposition Depth](https://mlsanity.com/paper/1603.06463/) (2016)

## Built on by (most cited first, 28 of 28)

- [Finding the right XAI method -- A Guide for the Evaluation and Ranking of Explainable AI Methods in Climate Science](https://mlsanity.com/paper/2303.00652/) (2023)
- [Assessing Fidelity in XAI post-hoc techniques: A Comparative Study with Ground Truth Explanations Datasets](https://mlsanity.com/paper/2311.01961/) (2023)
- [Explainable Artificial Intelligence for Bayesian Neural Networks: Towards trustworthy predictions of ocean dynamics](https://mlsanity.com/paper/2205.00202/) (2022)
- [Using Explainability to Inform Statistical Downscaling Based on Deep Learning Beyond Standard Validation Approaches](https://mlsanity.com/paper/2302.01771/) (2023)
- [Learning Closed-form Equations for Subgrid-scale Closures from High-fidelity Data: Promises and Challenges](https://mlsanity.com/paper/2306.05014/) (2023)
- [Towards Desiderata-Driven Design of Visual Counterfactual Explainers](https://mlsanity.com/paper/2506.14698/) (2025)
- [Time Series Predictions in Unmonitored Sites: A Survey of Machine Learning Techniques in Water Resources](https://mlsanity.com/paper/2308.09766/) (2023)
- [Lazy Estimation of Variable Importance for Large Neural Networks](https://mlsanity.com/paper/2207.09097/) (2022)
- [EvalAttAI: A Holistic Approach to Evaluating Attribution Maps in Robust and Non-Robust Models](https://mlsanity.com/paper/2303.08866/) (2023)
- [A novel approach to generate datasets with XAI ground truth to evaluate image models](https://mlsanity.com/paper/2302.05624/) (2023)
- [Survey on AI Ethics: A Socio-technical Perspective](https://mlsanity.com/paper/2311.17228/) (2023)
- [Interpretable Machine Learning for Weather and Climate Prediction: A Survey](https://mlsanity.com/paper/2403.18864/) (2024)
- [Tackling the Accuracy-Interpretability Trade-off in a Hierarchy of Machine Learning Models for the Prediction of Extreme Heatwaves](https://mlsanity.com/paper/2410.00984/) (2024)
- [Forecasting large-scale circulation regimes using deformable convolutional neural networks and global spatiotemporal climate data](https://mlsanity.com/paper/2202.04964/) (2022)
- [Physically Explainable Deep Learning for Convective Initiation Nowcasting Using GOES-16 Satellite Observations](https://mlsanity.com/paper/2310.16015/) (2023)
- [Using Deep Learning to Identify Initial Error Sensitivity for Interpretable ENSO Forecasts](https://mlsanity.com/paper/2404.15419/) (2024)
- [Distilling Machine Learning's Added Value: Pareto Fronts in Atmospheric Applications](https://mlsanity.com/paper/2408.02161/) (2024)
- [Prototype-Based Methods in Explainable AI and Emerging Opportunities in the Geosciences](https://mlsanity.com/paper/2410.19856/) (2024)
- [Characterizing climate pathways using feature importance on echo state networks](https://mlsanity.com/paper/2310.08495/) (2023)
- [Solving the enigma: Enhancing faithfulness and comprehensibility in explanations of deep networks](https://mlsanity.com/paper/2405.10008/) (2024)
- [Applications of Explainable artificial intelligence in Earth system science](https://mlsanity.com/paper/2406.11882/) (2024)
- [Uncertainty Quantification of Wind Gust Predictions in the Northeast United States: An Evidential Neural Network and Explainable Artificial Intelligence Approach](https://mlsanity.com/paper/2502.00300/) (2025)
- [Towards an Evaluation Framework for Explainable Artificial Intelligence Systems for Health and Well-being](https://mlsanity.com/paper/2504.08552/) (2025)
- [Exploring SAIG Methods for an Objective Evaluation of XAI](https://mlsanity.com/paper/2602.08715/) (2026)
- [Knowledge-Guided Machine Learning: Illustrating the use of Explainable Boosting Machines to Identify Overshooting Tops in Satellite Imagery](https://mlsanity.com/paper/2507.03183/) (2025)
- [Concept-based explanations of Segmentation and Detection models in Natural Disaster Management](https://mlsanity.com/paper/2603.23020/) (2026)
- [SimCast-S2S: A Computationally Efficient Diffusion Model for Subseasonal Precipitation Forecasting](https://mlsanity.com/paper/2608.26594/) (2026)
- [Understanding Perturbed Parameter Ensemble Sensitivities Using A Contrastive Learning Approach](https://mlsanity.com/paper/2609.30420/) (2026)

## Abstract (arXiv)

Convolutional neural networks (CNNs) have recently attracted great attention in geoscience due to their ability to capture non-linear system behavior and extract predictive spatiotemporal patterns. Given their black-box nature however, and the importance of prediction explainability, methods of explainable artificial intelligence (XAI) are gaining popularity as a means to explain the CNN decision-making strategy. Here, we establish an intercomparison of some of the most popular XAI methods and investigate their fidelity in explaining CNN decisions for geoscientific applications. Our goal is to raise awareness of the theoretical limitations of these methods and gain insight into the relative strengths and weaknesses to help guide best practices. The considered XAI methods are first applied to an idealized attribution benchmark, where the ground truth of explanation of the network is known a priori, to help objectively assess their performance. Secondly, we apply XAI to a climate-related prediction setting, namely to explain a CNN that is trained to predict the number of atmospheric rivers in daily snapshots of climate simulations. Our results highlight several important issues of XAI methods (e.g., gradient shattering, inability to distinguish the sign of attribution, ignorance to zero input) that have previously been overlooked in our field and, if not considered cautiously, may lead to a distorted picture of the CNN decision-making strategy. We envision that our analysis will motivate further investigation into XAI fidelity and will help towards a cautious implementation of XAI in geoscience, which can lead to further exploitation of CNNs and deep learning for prediction problems.