mlsanityData to 7 Oct 2026

Confronting the Reproducibility Crisis: A Case Study of Challenges in Cybersecurity AI

arXiv 2405.18753 ↗First posted 29 May 2024cs.LG cs.AI cs.CRcited by 2

Not on the map. Why?

Abstract (arXiv)

In the rapidly evolving field of cybersecurity, ensuring the reproducibility of AI-driven research is critical to maintaining the reliability and integrity of security systems. This paper addresses the reproducibility crisis within the domain of adversarial robustness -- a key area in AI-based cybersecurity that focuses on defending deep neural networks against malicious perturbations. Through a detailed case study, we attempt to validate results from prior work on certified robustness using the VeriGauge toolkit, revealing significant challenges due to software and hardware incompatibilities, version conflicts, and obsolescence. Our findings underscore the urgent need for standardized methodologies, containerization, and comprehensive documentation to ensure the reproducibility of AI models deployed in critical cybersecurity applications. By tackling these reproducibility challenges, we aim to contribute to the broader discourse on securing AI systems against advanced persistent threats, enhancing network and IoT security, and protecting critical infrastructure. This work advocates for a concerted effort within the research community to prioritize reproducibility, thereby strengthening the foundation upon which future cybersecurity advancements are built.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 14 of its 63 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

14 of 63 references.

References · 14

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2016End to End Learning for Self-Driving Carscited by 1,000
  3. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 459
  4. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
  5. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 318
  6. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  7. 2020SoK: Certified Robustness for Deep Neural Networkscited by 100
  8. 2014An introduction to Docker for reproducible research, with examples from the R environmentcited by 25
  9. 2022Towards Training Reproducible Deep Learning Modelscited by 23
  10. 2017eXpose: A Character-Level Convolutional Neural Network with Embeddings For Detecting Malicious URLs, File Paths and Registry Keyscited by 18
  11. 2023Reproducibility in Machine Learning-Driven Researchcited by 15
  12. 2023Reproducibility of Machine Learning: Terminology, Recommendations and Open Issuescited by 8
2 more
  1. 2023A Simple Framework to Enhance the Adversarial Robustness of Deep Learning-based Intrusion Detection Systemcited by 6
  2. 2023Towards Inferential Reproducibility of Machine Learning Researchcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Adversarial Defense in Cybersecurity: A Systematic Review of GANs for Threat Detection and Mitigationcited by 1
  2. 2026Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Paperscited by 1