mlsanityData to 14 Oct 2026

NLPeer: A Unified Resource for the Computational Study of Peer Review

arXiv 2211.06651 ↗First posted 12 Nov 2022cs.CLcited by 60

Not on the map. Why?

Abstract (arXiv)

Peer review constitutes a core component of scholarly publishing; yet it demands substantial expertise and training, and is susceptible to errors and biases. Various applications of NLP for peer reviewing assistance aim to support reviewers in this complex process, but the lack of clearly licensed datasets and multi-domain corpora prevent the systematic study of NLP for peer review. To remedy this, we introduce NLPeer -- the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues. In addition to the new datasets of paper drafts, camera-ready versions and peer reviews from the NLP community, we establish a unified data representation and augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information. We complement our resource with implementations and analysis of three reviewing assistance tasks, including a novel guided skimming task. Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond. The data and code are publicly available.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Builds on 21 of its 41 references that are in the map. First posted on arXiv 12 Nov 2022.

Builds on

21 of 41 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  4. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  5. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  6. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 136
  7. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  8. 2021Can We Automate Scientific Reviewing?cited by 63
  9. 2021MReD: A Meta-Review Dataset for Structure-Controllable Text Generationcited by 42
  10. 2019Argument Mining for Understanding Peer Reviewscited by 41
  11. 2022Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Reviewcited by 37
  12. 2021DISAPERE: A Dataset for Discourse Structure in Peer Review Discussionscited by 36
9 more
  1. 2016What to do about non-standard (or non-canonical) language in NLPcited by 35
  2. 2020What Can We Do to Improve Peer Review in NLP?cited by 32
  3. 2021Efficient Hierarchical Domain Adaptation for Pretrained Language Modelscited by 32
  4. 2018Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratingscited by 28
  5. 2022Scim: Intelligent Skimming Support for Scientific Paperscited by 27
  6. 2021Unsupervised Domain Adaptation with Adaptercited by 14
  7. 2022Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyondcited by 12
  8. 2022Which structure of academic articles do referees pay more attention to?: perspective of peer review and full-text of academic articlescited by 5
  9. 2021Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspectivecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 23; 2026: 26.

60 papers.

Sort:

By area:

  1. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 164
  2. 2024OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviewscited by 65
  3. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  4. 2025Large language models for automated scholarly paper review: A surveycited by 61
  5. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
  6. 2024Automated Focused Feedback Generation for Scientific Writing Assistancecited by 33
  7. 2024Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysiscited by 33
  8. 2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviewscited by 32
  9. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 26
  10. 2024What Can Natural Language Processing Do for Peer Review?cited by 25
  11. 2025Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussionscited by 23
  12. 2025PeerQA: A Scientific Question Answering Dataset from Peer Reviewscited by 22
38 more
  1. 2023Automatic Analysis of Substantiation in Scientific Peer Reviewscited by 19
  2. 2025IRIS: Interactive Research Ideation System for Accelerating Scientific Discoverycited by 19
  3. 2025ReviewRL: Towards Automated Scientific Review with RLcited by 19
  4. 2023Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttalscited by 18
  5. 2025The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authorscited by 16
  6. 2023Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluationcited by 14
  7. 2025FLAWS: A Benchmark for Error Identification and Localization in Scientific Paperscited by 14
  8. 2025LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviewscited by 13
  9. 2024Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revisioncited by 11
  10. 2025Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedbackcited by 11
  11. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 11
  12. 2025Insights from the ICLR Peer Review and Rebuttal Processcited by 11
  13. 2025MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automationcited by 10
  14. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  15. 2025Automatic Evaluation Metrics for Artificially Generated Scientific Researchcited by 8
  16. 2026NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessmentcited by 7
  17. 2024Acceleron: A Tool to Accelerate Research Ideationcited by 6
  18. 2025Large Language Models Penetration in Scholarly Writing and Peer Reviewcited by 6
  19. 2026RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generationcited by 6
  20. 2025Identifying Aspects in Peer Reviewscited by 4
  21. 2025Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AIcited by 4
  22. 2026Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Reviewcited by 4
  23. 2026Defend: Automated Rebuttals for Peer Review with Minimal Author Guidancecited by 4
  24. 2025Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialoguecited by 3
  25. 2026Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviewscited by 3
  26. 2026Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Reviewcited by 3
  27. 2025A Hierarchical Framework for Measuring Scientific Paper Innovation via Large Language Modelscited by 1
  28. 2026Expos'ia: Teaching and Assessment of Academic Writing Skills for Research Project Proposals and Peer Feedbackcited by 1
  29. 2026Are We Truly Innovating? A Qualitative and Quantitative Study of Originality in AI Research Paperscited by 1
  30. 2026Commitment Checklist: Auditing Author Commitments in Peer Reviewcited by 1
  31. 2026EarlySciRev: A Dataset of Early-Stage Scientific Revisions Extracted from LaTeX Writing Tracescited by 1
  32. 2026Review Arcade: On the Human Alignment and Gameability of LLM Reviewscited by 1
  33. 2026Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Reviewcited by 1
  34. 2026LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challengescited by 1
  35. 2026AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verificationcited by 1
  36. 2025ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Linkscited by 0
  37. 2026Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Timecited by 0
  38. 2026Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliancecited by 0

50 most cited of 60 All 60 (JSON)