# I-SAFE: Wasserstein Coherence Metrics for Structural Auditing of Scientific AI Models

arXiv: https://arxiv.org/abs/2605.21731 · HTML: https://mlsanity.com/paper/2605.21731/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 20 May 2026.

## Builds on (most cited first, 24 of 24)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [DeepDTA: Deep Drug-Target Binding Affinity Prediction](https://mlsanity.com/paper/1801.10193/) (2018)
- [Veridical Data Science](https://mlsanity.com/paper/1901.08152/) (2019)
- [DeepConv-DTI: Prediction of drug-target interactions via deep learning with convolution on protein sequences](https://mlsanity.com/paper/1811.02114/) (2018)
- [LLMScan: Causal Scan for LLM Misbehavior Detection](https://mlsanity.com/paper/2410.16638/) (2024)
- [RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation](https://mlsanity.com/paper/2506.15455/) (2025)
- [On Rank Graduation Metrics for High Dimensional Ordinal Data](https://mlsanity.com/paper/2511.23100/) (2025)
- [The Equivalence of Fourier-based and Wasserstein Metrics on Imaging Problems](https://mlsanity.com/paper/2005.06530/) (2020)

## Abstract (arXiv)

Deep learning models are increasingly used in scientific prediction tasks where strong benchmark performance is often interpreted as evidence of scientifically meaningful behavior. This interpretation is fragile, as models may exploit shortcut features, dataset-specific regularities, or distributional biases that are predictive on held-out data but not aligned with domain-relevant structure. To address this limitation, we introduce the textscI-SAFE (Interventional Secure, Accurate, Fair and Explainable) framework, a post-hoc distributional auditing framework for scientific AI models centered on the Wasserstein Coherence Metric (WCM). Given a trained black-box predictor and an external structural prior encoding domain knowledge about task-relevant input structure, textscI-SAFE evaluates raw model outputs under structurally guided perturbations of the input. The proposed audit measures output-distribution coherence through three complementary metrics: a Quantile-Based Metric (QBM) for location-level coherence, the WCM for ordinal coherence, and a translation-invariant WCM variant for shape coherence. We instantiate textscI-SAFE on drug--target interaction (DTI) prediction using the Davis kinase benchmark, KLIFS (Kinase--Ligand Interaction Fingerprints and Structures) binding-pocket annotations, and three sequence-based DTI models: DeepConvDTI, DeepDTA, and TAPB. Although the models operate in a comparable predictive regime, textscI-SAFE reveals substantially different distributional response profiles, a distinction invisible to accuracy-based evaluation. The framework is model-agnostic and applicable to any domain where inputs admit a structured decomposition and an external prior is available.