# Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer Review

arXiv: https://arxiv.org/abs/2309.05457 · HTML: https://mlsanity.com/paper/2309.05457/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 11 of 11)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Unlimiformer: Long-Range Transformers with Unlimited Length Input](https://mlsanity.com/paper/2305.01625/) (2023)
- [Automatic Text Scoring Using Neural Networks](https://mlsanity.com/paper/1606.04289/) (2016)
- [Design and Analysis of the NIPS 2016 Review Process](https://mlsanity.com/paper/1708.09794/) (2017)
- [An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process](https://mlsanity.com/paper/2010.05137/) (2020)
- [Automated scholarly paper review: Concepts, technologies, and challenges](https://mlsanity.com/paper/2111.07533/) (2021)
- [Mathematical Language Processing: Automatic Grading and Feedback for Open Response Mathematical Questions](https://mlsanity.com/paper/1501.04346/) (2015)
- [Does double-blind peer-review reduce bias? Evidence from a top computer science conference](https://mlsanity.com/paper/2101.02701/) (2021)
- [Automatic Academic Paper Rating Based on Modularized Hierarchical Convolutional Neural Network](https://mlsanity.com/paper/1805.03977/) (2018)
- [Deep Paper Gestalt](https://mlsanity.com/paper/1812.08775/) (2018)

## Built on by (most cited first, 3 of 3)

- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024)
- [From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review](https://mlsanity.com/paper/2506.11343/) (2025)
- [From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems](https://mlsanity.com/paper/2503.01424/) (2025)

## Abstract (arXiv)

Peer review is the method employed by the scientific community for evaluating research advancements. In the field of cybersecurity, the practice of double-blind peer review is the de-facto standard. This paper touches on the holy grail of peer reviewing and aims to shed light on the performance of AI in reviewing for academic security conferences. Specifically, we investigate the predictability of reviewing outcomes by comparing the results obtained from human reviewers and machine-learning models. To facilitate our study, we construct a comprehensive dataset by collecting thousands of papers from renowned computer science conferences and the arXiv preprint website. Based on the collected data, we evaluate the prediction capabilities of ChatGPT and a two-stage classification approach based on the Doc2Vec model with various classifiers. Our experimental evaluation of review outcome prediction using the Doc2Vec-based approach performs significantly better than the ChatGPT and achieves an accuracy of over 90%. While analyzing the experimental results, we identify the potential advantages and limitations of the tested ML models. We explore areas within the paper-reviewing process that can benefit from automated support approaches, while also recognizing the irreplaceable role of human intellect in certain aspects that cannot be matched by state-of-the-art AI techniques.