mlsanityData to 14 Oct 2026

A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review

arXiv 2609.39027 ↗First posted 30 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimization over scientific improvement. We formulate Rhetorical Robustness as the joint requirement of stability across content-preserving rewrites and discrimination across papers. We introduce RobustReview, a controlled full-manuscript benchmark with 1,260 manuscript versions, and evaluate 30 reviewer configurations. The benchmark reveals false robustness, where low rewrite sensitivity coincides with score collapse across papers, and shows that human alignment and rhetorical robustness rank reviewers differently. Moreover, the evaluated content-focused prompting protocol does not consistently improve robustness across backbones. Motivated by these findings, we introduce SciCore, a dual-branch reviewer that averages a full-manuscript judgment with a judgment based on an extracted, structured science core. This design combines manuscript-level assessment with a content-normalized view intended to reduce rhetorical sensitivity. In our primary GPT-5.5 comparison, SciCore achieves a leading joint stability-discrimination profile among the benchmarked reviewers while maintaining competitive human alignment. These results identify rhetorical robustness as a distinct evaluation target and demonstrate the potential of science-core review to improve it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

33 of 46 references.

References · 33

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  6. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  7. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  8. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  9. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 136
  10. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 100
  11. 2024OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviewscited by 65
  12. 2021Can We Automate Scientific Reviewing?cited by 63
21 more
  1. 2020ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesiscited by 54
  2. 2021Lossy Compression for Lossless Predictioncited by 52
  3. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  4. 2025When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Reviewcited by 19
  5. 2025Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attackscited by 16
  6. 2026Stop Automating Peer Review Without Rigorous Evaluationcited by 14
  7. 2025Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Paperscited by 12
  8. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 11
  9. 2025Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviewscited by 11
  10. 2026Paraphrasing Adversarial Attack on LLM-as-a-Reviewercited by 8
  11. 2026Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetrycited by 8
  12. 2025Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attackscited by 6
  13. 2024On the Rigour of Scientific Writing: Criteria, Analysis, and Insightscited by 5
  14. 2021What Makes a Scientific Paper be Accepted for Publication?cited by 4
  15. 2026Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judgescited by 4
  16. 2026From Passive Generation to Investigation: A Proactive Scientific Peer Review Agentcited by 4
  17. 2026Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Communitycited by 3
  18. 2026PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Qualitycited by 3
  19. 2026Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Reviewcited by 3
  20. 2026How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Reviewcited by 3
  21. 2026The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishingcited by 1

Built on by

Not cited yet.