mlsanityData to 7 Oct 2026

When AI reviews science: Can we trust the referee?

arXiv 2604.23593 ↗First posted 26 Apr 2026cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

The volume of scientific submissions continues to climb, outpacing the capacity of qualified human referees and stretching editorial timelines. At the same time, modern large language models (LLMs) offer impressive capabilities in summarization, fact checking, and literature triage, making the integration of AI into peer review increasingly attractive -- and, in practice, unavoidable. Yet early deployments and informal adoption have exposed acute failure modes. Recent incidents have revealed that hidden prompt injections embedded in manuscripts can steer LLM-generated reviews toward unjustifiably positive judgments. Complementary studies have also demonstrated brittleness to adversarial phrasing, authority and length biases, and hallucinated claims. These episodes raise a central question for scholarly communication: when AI reviews science, can we trust the AI referee? This paper provides a security- and reliability-centered analysis of AI peer review. We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level. We instantiate this taxonomy with four treatment-control probes on a stratified set of ICLR 2025 submissions, using two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores. Together, this taxonomy and experimental audit provide an evidence-based baseline for assessing and tracking the reliability of AI peer review and highlight concrete failure points to guide targeted, testable mitigations.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 98 of its 141 references that are in the map. First posted on arXiv 26 Apr 2026.

Builds on

98 of 141 references.

References · 98

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  9. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  12. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
38 more
  1. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  3. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  4. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  5. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  6. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  7. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  8. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  9. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  10. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  11. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  12. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  13. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  14. 2019On Evaluating Adversarial Robustnesscited by 442
  15. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  16. 2020Backdoor Learning: A Surveycited by 412
  17. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 411
  18. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  19. 2017Certified Defenses for Data Poisoning Attackscited by 288
  20. 2020Data Poisoning Attacks Against Federated Learning Systemscited by 266
  21. 2017On Detecting Adversarial Perturbationscited by 232
  22. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 195
  23. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  24. 2017Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimizationcited by 188
  25. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  26. 2020Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defensescited by 171
  27. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 160
  28. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 152
  29. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  30. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  31. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  32. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 121
  33. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 119
  34. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  35. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  36. 2020Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attackscited by 109
  37. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 107
  38. 2024On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trialcited by 105

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishingcited by 1
  2. 2026From peer review nuances to best practicescited by 0