# Delta-Audit: Explaining What Changes When Models Change

arXiv: https://arxiv.org/abs/2508.19589 · HTML: https://mlsanity.com/paper/2508.19589/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 27 Aug 2025.

## Builds on (most cited first, 18 of 18)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing](https://mlsanity.com/paper/2403.07175/) (2024)
- [Smoothed Geometry for Robust Attribution](https://mlsanity.com/paper/2006.06643/) (2020)
- [An Information-theoretic Approach to Distribution Shifts](https://mlsanity.com/paper/2106.03783/) (2021)
- ["Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts](https://mlsanity.com/paper/2210.10769/) (2022)
- [Explanation Shift: Detecting distribution shifts on tabular data via the explanation space](https://mlsanity.com/paper/2210.12369/) (2022)
- [Explanation Shift: How Did the Distribution Shift Impact the Model?](https://mlsanity.com/paper/2303.08081/) (2023)
- [Reliable Evaluation of Attribution Maps in CNNs: A Perturbation-Based Approach](https://mlsanity.com/paper/2411.14946/) (2024)

## Abstract (arXiv)

Model updates (new hyperparameters, kernels, depths, solvers, or data) change performance, but the reason often remains opaque. We introduce Delta-Attribution (mboxDelta-Attribution), a model-agnostic framework that explains what changed between versions A and B by differencing per-feature attributions: Deltaφ(x)=φB(x)-φA(x). We evaluate Deltaφ with a Delta-Attribution Quality Suite covering magnitude/sparsity (L1, Top-k, entropy), agreement/shift (rank-overlap@10, Jensen--Shannon divergence), behavioural alignment (Delta Conservation Error, DCE; Behaviour--Attribution Coupling, BAC; CODeltaF), and robustness (noise, baseline sensitivity, grouped occlusion). Instantiated via fast occlusion/clamping in standardized space with a class-anchored margin and baseline averaging, we audit 45 settings: five classical families (Logistic Regression, SVC, Random Forests, Gradient Boosting, kNN), three datasets (Breast Cancer, Wine, Digits), and three A/B pairs per family. Findings. Inductive-bias changes yield large, behaviour-aligned deltas (e.g., SVC poly!rightarrowrbf on Breast Cancer: BAC≈0.998, DCE≈6.6; Random Forest feature-rule swap on Digits: BAC≈0.997, DCE≈7.5), while ``cosmetic'' tweaks (SVC textttgamma=scale vs. textttauto, kNN search) show rank-overlap@10=1.0 and DCE≈0. The largest redistribution appears for deeper GB on Breast Cancer (JSD≈0.357). Delta-Attribution offers a lightweight update audit that complements accuracy by distinguishing benign changes from behaviourally meaningful or risky reliance shifts.