# Interpretable and Explainable Surrogate Modeling for Simulations: A State-of-the-Art Survey and Perspectives on Explainable AI for Decision-Making

arXiv: https://arxiv.org/abs/2604.14240 · HTML: https://mlsanity.com/paper/2604.14240/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 435 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Builds on (most cited first, 30 of 35)

- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey](https://mlsanity.com/paper/2006.11371/) (2020)
- [Local Rule-Based Explanations of Black Box Decision Systems](https://mlsanity.com/paper/1805.10820/) (2018)
- [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01602/) (2022)
- [Argumentative XAI: A Survey](https://mlsanity.com/paper/2105.11266/) (2021)
- [A Simple and Effective Model-Based Variable Importance Measure](https://mlsanity.com/paper/1805.04755/) (2018)
- [Distance-Restricted Explanations: Theoretical Underpinnings & Efficient Implementation](https://mlsanity.com/paper/2405.08297/) (2024)
- [Do Not Trust Additive Explanations](https://mlsanity.com/paper/1903.11420/) (2019)
- [Explainable Machine Learning with Prior Knowledge: An Overview](https://mlsanity.com/paper/2105.10172/) (2021)
- [Towards a Rigorous Evaluation of Explainability for Multivariate Time Series](https://mlsanity.com/paper/2104.04075/) (2021)
- [Which LIME should I trust? Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2503.24365/) (2025)
- [Understanding surrogate explanations: the interplay between complexity, fidelity and coverage](https://mlsanity.com/paper/2107.04309/) (2021)
- [Unexplainability and Incomprehensibility of Artificial Intelligence](https://mlsanity.com/paper/1907.03869/) (2019)
- [Variance based sensitivity analysis for Monte Carlo and importance sampling reliability assessment with Gaussian processes](https://mlsanity.com/paper/2011.15001/) (2020)
- [Deep Argumentative Explanations](https://mlsanity.com/paper/2012.05766/) (2020)
- [Unveil Sources of Uncertainty: Feature Contribution to Conformal Prediction Intervals](https://mlsanity.com/paper/2505.13118/) (2025)
- [Pitfalls of Explainable ML: An Industry Perspective](https://mlsanity.com/paper/2106.07758/) (2021)
- [Framework for Certification of AI-Based Systems](https://mlsanity.com/paper/2302.11049/) (2023)
- [Interpretable and Explainable Machine Learning Methods for Predictive Process Monitoring: A Systematic Literature Review](https://mlsanity.com/paper/2312.17584/) (2023)
- [Multi-objective Bayesian Optimization With Mixed-categorical Design Variables for Expensive-to-evaluate Aeronautical Applications](https://mlsanity.com/paper/2504.09930/) (2025)
- [Fairness-Aware Grouping for Continuous Sensitive Variables: Application for Debiasing Face Analysis with respect to Skin Tone](https://mlsanity.com/paper/2507.11247/) (2025)
- [Exposing the Illusion of Fairness: Auditing Vulnerabilities to Distributional Manipulation Attacks](https://mlsanity.com/paper/2507.20708/) (2025)
- [iLOCO: Distribution-Free Inference for Feature Interactions](https://mlsanity.com/paper/2502.06661/) (2025)
- [Bayesian Optimization of a Lightweight and Accurate Neural Network for Aerodynamic Performance Prediction](https://mlsanity.com/paper/2503.19479/) (2025)
- [Challenges in interpretability of additive models](https://mlsanity.com/paper/2504.10169/) (2025)
- [Beyond Shapley Values: Cooperative Games for the Interpretation of Machine Learning Models](https://mlsanity.com/paper/2506.13900/) (2025)

## Built on by (most cited first, 1 of 1)

- [Hierarchical Bayesian optimization of an aircraft-based multi-agent system-of-systems](https://mlsanity.com/paper/2609.22130/) (2026)

## Abstract (arXiv)

The simulation of complex systems increasingly relies on sophisticated but fundamentally opaque computational black-box simulators. Surrogate models play a central role in reducing the computational cost of complex systems simulations across a wide range of scientific and engineering domains. Notwithstanding, they inevitably inherit and often exacerbate this black-box nature, obscuring how input variables drive physical responses. Conversely, Explainable Artificial Intelligence (XAI) offers powerful tools to unpack these models. Yet, XAI methods struggle with engineering-specific constraints, such as highly correlated inputs, dynamical systems, and rigorous reliability requirements. Consequently, surrogate modeling and XAI have largely evolved as distinct fields of research, despite their strong complementarity. To reconnect these approaches, this state-of-the-art survey provides a structured perspective that maps existing XAI techniques onto the various stages of surrogate modeling workflows for design and exploration. To ground this synthesis, we draw upon illustrative applications across both equation-based simulations and agent-based modeling. We survey a broad spectrum of techniques, highlighting their strengths for revealing interactions and supporting human comprehension. Finally, we identify pressing open challenges, including the explainability of dynamical systems and the handling of mixed-variable systems, and propose a research agenda to make explainability a core, embedded element of simulation-driven workflows from model construction through decision-making. By transforming opaque emulators into explainable tools, this agenda empowers practitioners to move beyond accelerating simulations to extracting actionable insights from complex system behaviors.