mlsanityData to 7 Oct 2026

Why You Should Not Trust Interpretations in Machine Learning: Adversarial Attacks on Partial Dependence Plots

arXiv 2404.18702 ↗First posted 29 Apr 2024cs.LG cs.CR stat.APcited by 5

Not on the map. Why?

Abstract (arXiv)

The adoption of artificial intelligence (AI) across industries has led to the widespread use of complex black-box models and interpretation tools for decision making. This paper proposes an adversarial framework to uncover the vulnerability of permutation-based interpretation methods for machine learning tasks, with a particular focus on partial dependence (PD) plots. This adversarial framework modifies the original black box model to manipulate its predictions for instances in the extrapolation domain. As a result, it produces deceptive PD plots that can conceal discriminatory behaviors while preserving most of the original model's predictions. This framework can produce multiple fooled PD plots via a single model. By using real-world datasets including an auto insurance claims dataset and COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) dataset, our results show that it is possible to intentionally hide the discriminatory behavior of a predictor and make the black-box model appear neutral through interpretation tools like PD plots while retaining almost all the predictions of the original black-box model. Managerial insights for regulators and practitioners are provided based on the findings.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 26 of its 69 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

26 of 69 references.

References · 26

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2017Interpretation of Neural Networks is Fragilecited by 443
  6. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 440
  7. 2019Explanations can be manipulated and geometry is to blamecited by 215
  8. 2016Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Modelscited by 212
  9. 2021Counterfactual Explanations Can Be Manipulatedcited by 113
  10. 2019Fooling Neural Network Interpretations via Adversarial Model Manipulationcited by 112
  11. 2019Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importancecited by 85
  12. 2020General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Modelscited by 74
14 more
  1. 2022Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contextscited by 53
  2. 2023Adversarial attacks and defenses in explainable artificial intelligence: A surveycited by 47
  3. 2020Transparency, Auditability and eXplainability of Machine Learning Models in Credit Scoringcited by 26
  4. 2021Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Processcited by 24
  5. 2022Unifying local and global model explanations by functional decomposition of low dimensional structurescited by 18
  6. 2019Boosting insights in insurance tariff plans with tree-based machine learning methodscited by 17
  7. 2022REPID: Regional Effect Plots with implicit Interaction Detectioncited by 17
  8. 2021Fooling Partial Dependence via Data Poisoningcited by 16
  9. 2019VINE: Visualizing Statistical Interactions in Black Box Modelscited by 13
  10. 2023Artificial Intelligence Index Report 2023cited by 13
  11. 2015Insurance Premium Prediction via Gradient Tree-Boosted Tweedie Compound Poisson Modelscited by 11
  12. 2022Model Transparency and Interpretability : Survey and Application to the Insurance Industrycited by 5
  13. 2020Towards Explainability of Machine Learning Models in Insurance Pricingcited by 3
  14. 2022Detection of Interacting Variables for Generalized Linear Models via Neural Networkscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2024Explainable AI needs formalizationcited by 4
  2. 2025Unveil Sources of Uncertainty: Feature Contribution to Conformal Prediction Intervalscited by 4
  3. 2025FINCH: Locally Visualizing Higher-Order Feature Interactions in Black Box Modelscited by 2
  4. 2024From Point to probabilistic gradient boosting for claim frequency and severity predictioncited by 1
  5. 2025Explainable AI the Latest Advancements and New Trendscited by 1