# Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Papers

arXiv: https://arxiv.org/abs/2603.06862 · HTML: https://mlsanity.com/paper/2603.06862/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Builds on (most cited first, 20 of 20)

- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process](https://mlsanity.com/paper/2503.08569/) (2025)
- [Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants](https://mlsanity.com/paper/2208.09727/) (2022)
- [Getting pwn'd by AI: Penetration Testing with Large Language Models](https://mlsanity.com/paper/2308.00121/) (2023)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review](https://mlsanity.com/paper/2412.01708/) (2024)
- [How to avoid machine learning pitfalls: a guide for academic researchers](https://mlsanity.com/paper/2108.02497/) (2021)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [Large Language Models in Wireless Application Design: In-Context Learning-enhanced Automatic Network Intrusion Detection](https://mlsanity.com/paper/2405.11002/) (2024)
- [Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection](https://mlsanity.com/paper/2408.16400/) (2024)
- [MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation](https://mlsanity.com/paper/2508.14146/) (2025)
- [ReproducedPapers.org: Openly teaching and structuring machine learning reproducibility](https://mlsanity.com/paper/2012.01172/) (2020)
- [LLM-Measure: Generating Valid, Consistent, and Reproducible Text-Based Measures for Social Science Research](https://mlsanity.com/paper/2409.12722/) (2024)
- [Understanding and Improving Artifact Sharing in Software Engineering Research](https://mlsanity.com/paper/2008.01046/) (2020)
- [Confronting the Reproducibility Crisis: A Case Study of Challenges in Cybersecurity AI](https://mlsanity.com/paper/2405.18753/) (2024)
- [Reproducible and Adaptable Log Data Generation for Sound Cybersecurity Experiments](https://mlsanity.com/paper/2111.07847/) (2021)
- [Collaboration is not Evil: A Systematic Look at Security Research for Industrial Use](https://mlsanity.com/paper/2112.11417/) (2021)
- [Research Artifacts in Software Engineering Publications: Status and Trends](https://mlsanity.com/paper/2404.06852/) (2024)

## Built on by (most cited first, 1 of 1)

- [Beyond Reproducibility: Towards Security-Aware Evaluation of Research Artifacts](https://mlsanity.com/paper/2605.06508/) (2026)

## Abstract (arXiv)

Artifact Evaluation (AE) is essential for ensuring the transparency and reliability of research, closing the gap between exploratory work and real-world deployment is particularly important in cybersecurity, particularly in IoT and CPSs, where large-scale, heterogeneous, and privacy-sensitive data meet safety-critical actuation. Yet, manual reproducibility checks are time-consuming and do not scale with growing submission volumes. In this work, we demonstrate that Large Language Models (LLMs) can provide powerful support for AE tasks: (i) text-based reproducibility rating, (ii) autonomous sandboxed execution environment preparation, and (iii) assessment of methodological pitfalls. Our reproducibility-assessment toolkit yields an accuracy of over 72% and autonomously sets up execution environments for 28% of runnable cybersecurity artifacts. Our automated pitfall assessment detects seven prevalent pitfalls with high accuracy (F1 > 92%). Hence, the toolkit significantly reduces reviewer effort and, when integrated into established AE processes, could incentivize authors to submit higher-quality and more reproducible artifacts. IoT, CPS, and cybersecurity conferences and workshops may integrate the toolkit into their peer-review processes to support reviewers' decisions on awarding artifact badges, improving the overall sustainability of the process.