mlsanityData to 14 Oct 2026

Formally Explaining Neural Networks within Reactive Systems

arXiv 2308.00143 ↗First posted 31 Jul 2023cs.AI cs.LG cs.LOcited by 24

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) are increasingly being used as controllers in reactive systems. However, DNNs are highly opaque, which renders it difficult to explain and justify their actions. To mitigate this issue, there has been a surge of interest in explainable AI (XAI) techniques, capable of pinpointing the input features that caused the DNN to act as it did. Existing XAI techniques typically face two limitations: (i) they are heuristic, and do not provide formal guarantees that the explanations are correct; and (ii) they often apply to ``one-shot'' systems, where the DNN is invoked independently of past invocations, as opposed to reactive systems. Here, we begin bridging this gap, and propose a formal DNN-verification-based XAI technique for reasoning about multi-step, reactive systems. We suggest methods for efficiently calculating succinct explanations, by exploiting the system's transition constraints in order to curtail the search space explored by the underlying verifier. We evaluate our approach on two popular benchmarks from the domain of automated navigation; and observe that our methods allow the efficient computation of minimal and minimum explanations, significantly outperforming the state of the art. We also demonstrate that our methods produce formal explanations that are more reliable than competing, non-verification-based XAI techniques.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 63 of its 113 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

63 of 113 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 744
  9. 2017One pixel attack for fooling deep neural networkscited by 582
  10. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  11. 2017Deep Reinforcement Learning: An Overviewcited by 333
  12. 2016Safety Verification of Deep Neural Networkscited by 263
38 more
  1. 2018Metrics for Explainable AI: Challenges and Prospectscited by 260
  2. 2021Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Robustness Verificationcited by 228
  3. 2019Algorithms for Verifying Deep Neural Networkscited by 221
  4. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  5. 2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 167
  6. 2018Abduction-Based Explanations for Machine Learning Modelscited by 160
  7. 2020Explainable Reinforcement Learning: A Surveycited by 141
  8. 2018A Symbolic Approach to Explaining Bayesian Network Classifierscited by 131
  9. 2020Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehiclescited by 110
  10. 2017Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigationcited by 98
  11. 2020Explainability in Deep Reinforcement Learningcited by 97
  12. 2021PRIMA: General and Precise Neural Network Certification via Scalable Convex Hull Approximationscited by 77
  13. 2020Model Interpretability through the Lens of Computational Complexitycited by 75
  14. 2021Introduction to Neural Network Verificationcited by 63
  15. 2020Explaining Naive Bayes and Other Linear Classifiers with Polynomial Time and Delaycited by 57
  16. 2019Quantitative Verification of Neural Networks And its Security Applicationscited by 56
  17. 2021On Guaranteed Optimal Robust Explanations for NLP Modelscited by 54
  18. 2020On Explaining Decision Treescited by 49
  19. 2020An SMT-Based Approach for Verifying Binarized Neural Networkscited by 47
  20. 2019Optimization and Abstraction: A Synergistic Approach for Analyzing Neural Network Robustnesscited by 40
  21. 2021SAT-Based Rigorous Explanations for Decision Listscited by 40
  22. 2019Can I Trust the Explainer? Verifying Post-hoc Explanatory Methodscited by 38
  23. 2022Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysiscited by 38
  24. 2021On Efficiently Explaining Graph-Based Classifierscited by 37
  25. 2020Parallelization Techniques for Verifying Neural Networkscited by 36
  26. 2021Neural Network Robustness as a Verification Property: A Principled Case Studycited by 35
  27. 2020On Tractable Representations of Binary Neural Networkscited by 34
  28. 2020Abstraction based Output Range Analysis for Neural Networkscited by 34
  29. 2021Towards Scalable Verification of Deep Reinforcement Learningcited by 33
  30. 2022Verifying Learning-Based Robotic Navigation Systemscited by 33
  31. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  32. 2019On Validating, Repairing and Refining Heuristic ML Explanationscited by 32
  33. 2022VeriX: Towards Verified Explainability of Deep Neural Networkscited by 30
  34. 2020Verifying Recurrent Neural Networks using Invariant Inferencecited by 26
  35. 2022An Abstraction-Refinement Approach to Verifying Convolutional Neural Networkscited by 24
  36. 2022Feature Necessity & Relevancy in ML Classifier Explanationscited by 23
  37. 2019Weighted Automata Extraction from Recurrent Neural Networks via Regression on State Spacescited by 21
  38. 2017DeepSafe: A Data-driven Approach for Checking Adversarial Robustness in Neural Networkscited by 20

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 7; 2026: 7.

24 papers.

Sort:

By area:

  1. 2024Marabou 2.0: A Versatile Formal Analyzer of Neural Networkscited by 81
  2. 2024Local vs. Global Interpretability: A Computational Complexity Perspectivecited by 23
  3. 2024Formally Verifying Deep Reinforcement Learning Controllers with Lyapunov Barrier Certificatescited by 16
  4. 2024Verification-Guided Shielding for Deep Reinforcement Learningcited by 13
  5. 2025Explaining, Fast and Slow: Abstraction and Refinement of Provable Explanationscited by 13
  6. 2024Shield Synthesis for LTL Modulo Theoriescited by 12
  7. 2025Explain Yourself, Briefly! Self-Explaining Neural Networks with Concise Sufficient Reasonscited by 12
  8. 2024Hard to Explain: On the Computational Hardness of In-Distribution Model Interpretationcited by 11
  9. 2026Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guaranteescited by 11
  10. 2025SHAP Meets Tensor Networks: Provably Tractable Explanations with Parallelismcited by 8
  11. 2024Analyzing Adversarial Inputs in Deep Reinforcement Learningcited by 7
  12. 2025Self-Explaining Neural Networks for Business Process Monitoringcited by 5
12 more
  1. 2024Towards Formal Verification of Deep Neural Networks for Object Detectioncited by 4
  2. 2024Safe and Reliable Training of Learning-Based Aerospace Controllerscited by 4
  3. 2025PICID: Proof-Driven Clause Learning in Neural Network Verificationcited by 3
  4. 2025Additive Models Explained: A Computational Complexity Approachcited by 3
  5. 2026FAME: Formal Abstract Minimal Explanation for Neural Networkscited by 3
  6. 2026Unifying Formal Explanations: A Complexity-Theoretic Perspectivecited by 2
  7. 2026Incremental Neural Network Verification via Learned Conflictscited by 2
  8. 2024Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learningcited by 1
  9. 2026Provably Explaining Neural Additive Modelscited by 1
  10. 2026Concept-Based Abductive and Contrastive Explanations for Behaviors of Vision Modelscited by 1
  11. 2025On Improving Deep Active Learning with Formal Verificationcited by 0
  12. 2026Cycle-Consistent Neural Explanation of Formal Verification Certificatescited by 0