# Using Attribution to Decode Dataset Bias in Neural Network Models for Chemistry

arXiv: https://arxiv.org/abs/1811.11310 · HTML: https://mlsanity.com/paper/1811.11310/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2018.

## Built on by (most cited first, 25 of 25)

- [Impossibility Theorems for Feature Attribution](https://mlsanity.com/paper/2212.11870/) (2022)
- [Node-Level Differentially Private Graph Neural Networks](https://mlsanity.com/paper/2111.15521/) (2021)
- [Releasing Graph Neural Networks with Differential Privacy Guarantees](https://mlsanity.com/paper/2109.08907/) (2021)
- [Interpretable Geometric Deep Learning via Learnable Randomness Injection](https://mlsanity.com/paper/2210.16966/) (2022)
- [Quantitative Evaluation of Explainable Graph Neural Networks for Molecular Property Prediction](https://mlsanity.com/paper/2107.04119/) (2021)
- [How Interpretable Are Interpretable Graph Neural Networks?](https://mlsanity.com/paper/2406.07955/) (2024)
- [Exact Computation of Any-Order Shapley Interactions for Graph Neural Networks](https://mlsanity.com/paper/2501.16944/) (2025)
- [Composite Feature Selection using Deep Ensembles](https://mlsanity.com/paper/2211.00631/) (2022)
- [Counterfactual Explanation with Multi-Agent Reinforcement Learning for Drug Target Prediction](https://mlsanity.com/paper/2103.12983/) (2021)
- [MEGAN: Multi-Explanation Graph Attention Network](https://mlsanity.com/paper/2211.13236/) (2022)
- [Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Features](https://mlsanity.com/paper/2408.17198/) (2024)
- [PAGE: Prototype-Based Model-Level Explanations for Graph Neural Networks](https://mlsanity.com/paper/2210.17159/) (2022)
- [Interpretable Retinal Disease Prediction Using Biology-Informed Heterogeneous Graph Representations](https://mlsanity.com/paper/2502.16697/) (2025)
- [B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data](https://mlsanity.com/paper/2505.22252/) (2025)
- [Co-modeling the Sequential and Graphical Routes for Peptide Representation Learning](https://mlsanity.com/paper/2310.02964/) (2023)
- [Automatic Identification of Chemical Moieties](https://mlsanity.com/paper/2203.16205/) (2022)
- [Applications of Explainable artificial intelligence in Earth system science](https://mlsanity.com/paper/2406.11882/) (2024)
- [Towards Understanding Sensitive and Decisive Patterns in Explainable AI: A Case Study of Model Interpretation in Geometric Deep Learning](https://mlsanity.com/paper/2407.00849/) (2024)
- [Transformers from Diffusion: A Unified Framework for Neural Message Passing](https://mlsanity.com/paper/2409.09111/) (2024)
- [Structure-Aware Compound-Protein Affinity Prediction via Graph Neural Networks with Group Lasso Regularization](https://mlsanity.com/paper/2507.03318/) (2025)
- [GLAMOUR: Graph Learning over Macromolecule Representations](https://mlsanity.com/paper/2103.02565/) (2021)
- [Meaningful machine learning models and machine-learned pharmacophores from fragment screening campaigns](https://mlsanity.com/paper/2204.06348/) (2022)
- [Regression Trees Know Calculus](https://mlsanity.com/paper/2405.13846/) (2024)
- [An Interpretable Framework Applying Protein Words to Predict Protein-Small Molecule Complementary Pairing Rules](https://mlsanity.com/paper/2604.16550/) (2026)
- [WOMBAT: Whitebox Oracle for Molecular Benchmarking and Attribution Testing](https://mlsanity.com/paper/2610.00713/) (2026)

## Abstract (arXiv)

Deep neural networks have achieved state of the art accuracy at classifying molecules with respect to whether they bind to specific protein targets. A key breakthrough would occur if these models could reveal the fragment pharmacophores that are causally involved in binding. Extracting chemical details of binding from the networks could potentially lead to scientific discoveries about the mechanisms of drug actions. But doing so requires shining light into the black box that is the trained neural network model, a task that has proved difficult across many domains. Here we show how the binding mechanism learned by deep neural network models can be interrogated, using a recently described attribution method. We first work with carefully constructed synthetic datasets, in which the 'fragment logic' of binding is fully known. We find that networks that achieve perfect accuracy on held out test datasets still learn spurious correlations due to biases in the datasets, and we are able to exploit this non-robustness to construct adversarial examples that fool the model. The dataset bias makes these models unreliable for accurately revealing information about the mechanisms of protein-ligand binding. In light of our findings, we prescribe a test that checks for dataset bias given a hypothesis. If the test fails, it indicates that either the model must be simplified or regularized and/or that the training dataset requires augmentation.