mlsanityData to 14 Oct 2026

Fooling Partial Dependence via Data Poisoning

arXiv 2105.12837 ↗First posted 26 May 2021cs.LG stat.MLcited by 16

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Many methods have been developed to understand complex predictive models and high expectations are placed on post-hoc model explainability. It turns out that such explanations are not robust nor trustworthy, and they can be fooled. This paper presents techniques for attacking Partial Dependence (plots, profiles, PDP), which are among the most popular methods of explaining any predictive model trained on tabular data. We showcase that PD can be manipulated in an adversarial manner, which is alarming, especially in financial or medical applications where auditability became a must-have trait supporting black-box machine learning. The fooling is performed via poisoning the data to bend and shift explanations in the desired direction using genetic and gradient algorithms. We believe this to be the first work using a genetic algorithm for manipulating explanations, which is transferable as it generalizes both ways: in a model-agnostic and an explanation-agnostic manner.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 36 of its 59 references that are in the map. First posted on arXiv 26 May 2021.

Builds on

36 of 59 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  6. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
  7. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  8. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  9. 2018Sanity Checks for Saliency Mapscited by 1,156
  10. 2016The Mythos of Model Interpretabilitycited by 721
  11. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  12. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
24 more
  1. 2017Interpretation of Neural Networks is Fragilecited by 444
  2. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 443
  3. 2017The (Un)reliability of saliency methodscited by 333
  4. 2019InterpretML: A Unified Framework for Machine Learning Interpretabilitycited by 246
  5. 2019Explanations can be manipulated and geometry is to blamecited by 216
  6. 2016Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Modelscited by 213
  7. 2020Evaluating and Aggregating Feature-based Model Explanationscited by 188
  8. 2019Feature relevance quantification in explainable AI: A causal problemcited by 167
  9. 2020Debugging Tests for Model Explanationscited by 147
  10. 2019Fooling Neural Network Interpretations via Adversarial Model Manipulationcited by 112
  11. 2019"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanationscited by 111
  12. 2019Fairwashing: the risk of rationalizationcited by 81
  13. 2018Interpretable Deep Learning under Firecited by 74
  14. 2020Poisoning Attacks on Algorithmic Fairnesscited by 59
  15. 2018iNNvestigate neural networks!cited by 58
  16. 2020Proper Network Interpretability Helps Adversarial Robustness in Classificationcited by 41
  17. 2020dalex: Responsible Machine Learning with Interactive Explainability and Fairness in Pythoncited by 40
  18. 2019Evaluating Explanation Methods for Deep Learning in Securitycited by 38
  19. 2020Smoothed Geometry for Robust Attributioncited by 38
  20. 2018DALEX: explainers for complex predictive modelscited by 28
  21. 2020A simple defense against adversarial attacks on heatmap explanationscited by 23
  22. 2019Faking Fairness via Stealthily Biased Samplingcited by 14
  23. 2020On Saliency Maps and Adversarial Robustnesscited by 11
  24. 2019Aggregating explanation methods for stable and robust explainabilitycited by 9

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 6; 2024: 6.

16 papers.

Sort:

By area:

  1. 2023Adversarial attacks and defenses in explainable artificial intelligence: A surveycited by 48
  2. 2023Don't trust your eyes: on the (un)reliability of feature visualizationscited by 32
  3. 2024Manifold Integrated Gradients: Riemannian Geometry for Feature Attributioncited by 21
  4. 2022DALE: Differential Accumulated Local Effects for efficient and accurate global explanationscited by 11
  5. 2023RHALE: Robust and Heterogeneity-aware Accumulated Local Effectscited by 8
  6. 2023Manipulation Risks in Explainable AI: The Implications of the Disagreement Problemcited by 7
  7. 2022Fool SHAP with Stealthily Biased Samplingcited by 6
  8. 2023SoK: Unintended Interactions among Machine Learning Defenses and Riskscited by 6
  9. 2022On the Robustness of Explanations of Deep Neural Network Models: A Surveycited by 5
  10. 2024Why You Should Not Trust Interpretations in Machine Learning: Adversarial Attacks on Partial Dependence Plotscited by 5
  11. 2024The Effect of Data Poisoning on Counterfactual Explanationscited by 3
  12. 2024On the Robustness of Global Feature Effect Explanationscited by 2
4 more
  1. 2023Understanding User Preferences in Explainable Artificial Intelligence: A Survey and a Mapping Function Proposalcited by 1
  2. 2024XAI-CF -- Examining the Role of Explainable Artificial Intelligence in Cyber Forensicscited by 1
  3. 2021Do not explain without context: addressing the blind spot of model explanationscited by 0
  4. 2024Towards Understanding the Influence of Training Samples on Explanationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Crushing the Evidence: A Dual-Penalty Evasion Framework for Fooling White-Box Explainable AI Auditors
  2. 2024Have You Poisoned My Data? Defending Neural Networks against Data Poisoning
  3. 2026Stealthy World Model Manipulation via Data Poisoning
  4. 2023Mendata: A Framework to Purify Manipulated Training Data
  5. 2026A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents
  6. 2023Exploring the Limits of Model-Targeted Indiscriminate Data Poisoning Attacks
  7. 2024Empirical Perturbation Analysis of Linear System Solvers from a Data Poisoning Perspective
  8. 2022Autoregressive Perturbations for Data Poisoning