# NLPeer: A Unified Resource for the Computational Study of Peer Review

arXiv: https://arxiv.org/abs/2211.06651 · HTML: https://mlsanity.com/paper/2211.06651/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 12 Nov 2022.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [DEMix Layers: Disentangling Domains for Modular Language Modeling](https://mlsanity.com/paper/2108.05036/) (2021)
- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [MReD: A Meta-Review Dataset for Structure-Controllable Text Generation](https://mlsanity.com/paper/2110.07474/) (2021)
- [Argument Mining for Understanding Peer Reviews](https://mlsanity.com/paper/1903.10104/) (2019)
- [Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review](https://mlsanity.com/paper/2204.10805/) (2022)
- [DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions](https://mlsanity.com/paper/2110.08520/) (2021)
- [What to do about non-standard (or non-canonical) language in NLP](https://mlsanity.com/paper/1608.07836/) (2016)
- [What Can We Do to Improve Peer Review in NLP?](https://mlsanity.com/paper/2010.03863/) (2020)
- [Efficient Hierarchical Domain Adaptation for Pretrained Language Models](https://mlsanity.com/paper/2112.08786/) (2021)
- [Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings](https://mlsanity.com/paper/1806.05085/) (2018)
- [Scim: Intelligent Skimming Support for Scientific Papers](https://mlsanity.com/paper/2205.04561/) (2022)
- [Unsupervised Domain Adaptation with Adapter](https://mlsanity.com/paper/2111.00667/) (2021)
- [Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond](https://mlsanity.com/paper/2201.11443/) (2022)
- [Which structure of academic articles do referees pay more attention to?: perspective of peer review and full-text of academic articles](https://mlsanity.com/paper/2209.01841/) (2022)
- [Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective](https://mlsanity.com/paper/2109.01190/) (2021)

## Built on by (most cited first, 30 of 60)

- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://mlsanity.com/paper/2406.16253/) (2024)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [Automated Focused Feedback Generation for Scientific Writing Assistance](https://mlsanity.com/paper/2405.20477/) (2024)
- [Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis](https://mlsanity.com/paper/2407.12857/) (2024)
- [ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews](https://mlsanity.com/paper/2503.08506/) (2025)
- [When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research](https://mlsanity.com/paper/2505.11855/) (2025)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions](https://mlsanity.com/paper/2505.07920/) (2025)
- [PeerQA: A Scientific Question Answering Dataset from Peer Reviews](https://mlsanity.com/paper/2502.13668/) (2025)
- [Automatic Analysis of Substantiation in Scientific Peer Reviews](https://mlsanity.com/paper/2311.11967/) (2023)
- [IRIS: Interactive Research Ideation System for Accelerating Scientific Discovery](https://mlsanity.com/paper/2504.16728/) (2025)
- [ReviewRL: Towards Automated Scientific Review with RL](https://mlsanity.com/paper/2508.10308/) (2025)
- [Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals](https://mlsanity.com/paper/2311.03998/) (2023)
- [The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors](https://mlsanity.com/paper/2509.04484/) (2025)
- [Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation](https://mlsanity.com/paper/2305.14647/) (2023)
- [FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers](https://mlsanity.com/paper/2511.21843/) (2025)
- [LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews](https://mlsanity.com/paper/2504.11042/) (2025)
- [Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision](https://mlsanity.com/paper/2406.00197/) (2024)
- [Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedback](https://mlsanity.com/paper/2507.16007/) (2025)
- [Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework](https://mlsanity.com/paper/2508.21422/) (2025)
- [Insights from the ICLR Peer Review and Rebuttal Process](https://mlsanity.com/paper/2511.15462/) (2025)
- [MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation](https://mlsanity.com/paper/2508.14146/) (2025)
- [From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems](https://mlsanity.com/paper/2503.01424/) (2025)
- [Automatic Evaluation Metrics for Artificially Generated Scientific Research](https://mlsanity.com/paper/2503.05712/) (2025)
- [NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment](https://mlsanity.com/paper/2604.11543/) (2026)
- [Acceleron: A Tool to Accelerate Research Ideation](https://mlsanity.com/paper/2403.04382/) (2024)
- [Large Language Models Penetration in Scholarly Writing and Peer Review](https://mlsanity.com/paper/2502.11193/) (2025)

## Abstract (arXiv)

Peer review constitutes a core component of scholarly publishing; yet it demands substantial expertise and training, and is susceptible to errors and biases. Various applications of NLP for peer reviewing assistance aim to support reviewers in this complex process, but the lack of clearly licensed datasets and multi-domain corpora prevent the systematic study of NLP for peer review. To remedy this, we introduce NLPeer -- the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues. In addition to the new datasets of paper drafts, camera-ready versions and peer reviews from the NLP community, we establish a unified data representation and augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information. We complement our resource with implementations and analysis of three reviewing assistance tasks, including a novel guided skimming task. Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond. The data and code are publicly available.