mlsanityData to 7 Oct 2026

Automatic Analysis of Substantiation in Scientific Peer Reviews

arXiv 2311.11967 ↗First posted 20 Nov 2023cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

With the increasing amount of problematic peer reviews in top AI conferences, the community is urgently in need of automatic quality control measures. In this paper, we restrict our attention to substantiation -- one popular quality aspect indicating whether the claims in a review are sufficiently supported by evidence -- and provide a solution automatizing this evaluation process. To achieve this goal, we first formulate the problem as claim-evidence pair extraction in scientific peer reviews, and collect SubstanReview, the first annotated dataset for this task. SubstanReview consists of 550 reviews from NLP conferences annotated by domain experts. On the basis of this dataset, we train an argument mining system to automatically analyze the level of substantiation in peer reviews. We also perform data analysis on the SubstanReview dataset to obtain meaningful insights on peer reviewing quality in NLP conferences over recent years.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 19 of its 41 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

19 of 41 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  7. 2023LIMA: Less Is More for Alignmentcited by 1,080
  8. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  9. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  10. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 135
  11. 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
  12. 2019Argument Mining for Understanding Peer Reviewscited by 41
7 more
  1. 2019Corpus Wide Argument Mining -- a Working Solutioncited by 36
  2. 2021DISAPERE: A Dataset for Discourse Structure in Peer Review Discussionscited by 35
  3. 2020An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Processcited by 23
  4. 2020Argument Mining Driven Analysis of Peer-Reviewscited by 16
  5. 2022Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyondcited by 12
  6. 2021Some Ethical Issues in the Review Process of Machine Learning Conferencescited by 4
  7. 2022Journal Impact Factor and Peer Review Thoroughness and Helpfulness: A Supervised Machine Learning Studycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 8.

19 papers.

Sort:

By area:

  1. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  2. 2025Large language models for automated scholarly paper review: A surveycited by 59
  3. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  4. 2024What Can Natural Language Processing Do for Peer Review?cited by 24
  5. 2025The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authorscited by 16
  6. 2025How Far Are AI Scientists from Changing the World?cited by 15
  7. 2025LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviewscited by 13
  8. 2025FLAWS: A Benchmark for Error Identification and Localization in Scientific Paperscited by 12
  9. 2025CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?cited by 11
  10. 2025Large Language Models Penetration in Scholarly Writing and Peer Reviewcited by 6
  11. 2026Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Futurecited by 6
  12. 2025ReviewScore: Misinformed Peer Review Detection with Large Language Modelscited by 1
7 more
  1. 2026What happens when reviewers receive AI feedback in their reviews?cited by 1
  2. 2026AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verificationcited by 1
  3. 2026Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliancecited by 0
  4. 2026Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviewscited by 0
  5. 2026Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AIcited by 0
  6. 2026Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?cited by 0
  7. 2026How Closely Do LLM Reviews Align with Human Peer Review?cited by 0