# Fooling Partial Dependence via Data Poisoning

arXiv: https://arxiv.org/abs/2105.12837 · HTML: https://mlsanity.com/paper/2105.12837/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 36 of its 59 references that are in the map.
- First posted on arXiv 26 May 2021.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [InterpretML: A Unified Framework for Machine Learning Interpretability](https://mlsanity.com/paper/1909.09223/) (2019)
- [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)
- [Evaluating and Aggregating Feature-based Model Explanations](https://mlsanity.com/paper/2005.00631/) (2020)
- [Feature relevance quantification in explainable AI: A causal problem](https://mlsanity.com/paper/1910.13413/) (2019)
- [Debugging Tests for Model Explanations](https://mlsanity.com/paper/2011.05429/) (2020)
- [Fooling Neural Network Interpretations via Adversarial Model Manipulation](https://mlsanity.com/paper/1902.02041/) (2019)
- ["How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations](https://mlsanity.com/paper/1911.06473/) (2019)
- [Fairwashing: the risk of rationalization](https://mlsanity.com/paper/1901.09749/) (2019)
- [Interpretable Deep Learning under Fire](https://mlsanity.com/paper/1812.00891/) (2018)
- [Poisoning Attacks on Algorithmic Fairness](https://mlsanity.com/paper/2004.07401/) (2020)
- [iNNvestigate neural networks!](https://mlsanity.com/paper/1808.04260/) (2018)
- [Proper Network Interpretability Helps Adversarial Robustness in Classification](https://mlsanity.com/paper/2006.14748/) (2020)
- [dalex: Responsible Machine Learning with Interactive Explainability and Fairness in Python](https://mlsanity.com/paper/2012.14406/) (2020)
- [Evaluating Explanation Methods for Deep Learning in Security](https://mlsanity.com/paper/1906.02108/) (2019)

## Built on by (most cited first, 16 of 16)

- [Adversarial attacks and defenses in explainable artificial intelligence: A survey](https://mlsanity.com/paper/2306.06123/) (2023)
- [Don't trust your eyes: on the (un)reliability of feature visualizations](https://mlsanity.com/paper/2306.04719/) (2023)
- [Manifold Integrated Gradients: Riemannian Geometry for Feature Attribution](https://mlsanity.com/paper/2405.09800/) (2024)
- [DALE: Differential Accumulated Local Effects for efficient and accurate global explanations](https://mlsanity.com/paper/2210.04542/) (2022)
- [RHALE: Robust and Heterogeneity-aware Accumulated Local Effects](https://mlsanity.com/paper/2309.11193/) (2023)
- [Manipulation Risks in Explainable AI: The Implications of the Disagreement Problem](https://mlsanity.com/paper/2306.13885/) (2023)
- [Fool SHAP with Stealthily Biased Sampling](https://mlsanity.com/paper/2205.15419/) (2022)
- [SoK: Unintended Interactions among Machine Learning Defenses and Risks](https://mlsanity.com/paper/2312.04542/) (2023)
- [On the Robustness of Explanations of Deep Neural Network Models: A Survey](https://mlsanity.com/paper/2211.04780/) (2022)
- [Why You Should Not Trust Interpretations in Machine Learning: Adversarial Attacks on Partial Dependence Plots](https://mlsanity.com/paper/2404.18702/) (2024)
- [The Effect of Data Poisoning on Counterfactual Explanations](https://mlsanity.com/paper/2402.08290/) (2024)
- [On the Robustness of Global Feature Effect Explanations](https://mlsanity.com/paper/2406.09069/) (2024)
- [Understanding User Preferences in Explainable Artificial Intelligence: A Survey and a Mapping Function Proposal](https://mlsanity.com/paper/2302.03180/) (2023)
- [XAI-CF -- Examining the Role of Explainable Artificial Intelligence in Cyber Forensics](https://mlsanity.com/paper/2402.02452/) (2024)
- [Do not explain without context: addressing the blind spot of model explanations](https://mlsanity.com/paper/2105.13787/) (2021)
- [Towards Understanding the Influence of Training Samples on Explanations](https://mlsanity.com/paper/2406.03012/) (2024)

## Abstract (arXiv)

Many methods have been developed to understand complex predictive models and high expectations are placed on post-hoc model explainability. It turns out that such explanations are not robust nor trustworthy, and they can be fooled. This paper presents techniques for attacking Partial Dependence (plots, profiles, PDP), which are among the most popular methods of explaining any predictive model trained on tabular data. We showcase that PD can be manipulated in an adversarial manner, which is alarming, especially in financial or medical applications where auditability became a must-have trait supporting black-box machine learning. The fooling is performed via poisoning the data to bend and shift explanations in the desired direction using genetic and gradient algorithms. We believe this to be the first work using a genetic algorithm for manipulating explanations, which is transferable as it generalizes both ways: in a model-agnostic and an explanation-agnostic manner.