Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer Review
Abstract
Peer review is the method employed by the scientific community for evaluating
research advancements. This paper touches on the holy grail of peer
reviewing and aims to shed light on the performance
of AI in reviewing for academic security conferences. Specifically, we
investigate the predictability of reviewing outcomes by comparing the
results obtained from human reviewers and machine-learning models. To
facilitate our study, we construct a comprehensive dataset by collecting
thousands of papers from renowned computer science conferences and
the arXiv preprint website. We evaluate the prediction capabilities of
three representative Large Language Models (i. e., ChatGPT, Qwen, and
DeepSeek), as well as a two-stage classification approach based on the
Doc2Vec model with various classifiers. Our experimental evaluation of
review outcome prediction using the Doc2Vec-based approach achieves an
accuracy of over 90%, whereas LLM-based methods obtain accuracies of
approximately 60%. While analyzing the experimental results, we identify
the potential advantages and limitations of the tested ML models. We
explore areas within the paper-reviewing process that can benefit from
automated support approaches, while also recognizing the irreplaceable
role of human intellect in certain aspects that cannot be matched by
state-of-the-art AI techniques.
research advancements. This paper touches on the holy grail of peer
reviewing and aims to shed light on the performance
of AI in reviewing for academic security conferences. Specifically, we
investigate the predictability of reviewing outcomes by comparing the
results obtained from human reviewers and machine-learning models. To
facilitate our study, we construct a comprehensive dataset by collecting
thousands of papers from renowned computer science conferences and
the arXiv preprint website. We evaluate the prediction capabilities of
three representative Large Language Models (i. e., ChatGPT, Qwen, and
DeepSeek), as well as a two-stage classification approach based on the
Doc2Vec model with various classifiers. Our experimental evaluation of
review outcome prediction using the Doc2Vec-based approach achieves an
accuracy of over 90%, whereas LLM-based methods obtain accuracies of
approximately 60%. While analyzing the experimental results, we identify
the potential advantages and limitations of the tested ML models. We
explore areas within the paper-reviewing process that can benefit from
automated support approaches, while also recognizing the irreplaceable
role of human intellect in certain aspects that cannot be matched by
state-of-the-art AI techniques.
