# What Can We Do to Improve Peer Review in NLP?

arXiv: https://arxiv.org/abs/2010.03863 · HTML: https://mlsanity.com/paper/2010.03863/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2020.

## Built on by (most cited first, 30 of 32)

- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [NLPeer: A Unified Resource for the Computational Study of Peer Review](https://mlsanity.com/paper/2211.06651/) (2022)
- [Changing the World by Changing the Data](https://mlsanity.com/paper/2105.13947/) (2021)
- [The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail](https://mlsanity.com/paper/2110.08300/) (2021)
- [Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards](https://mlsanity.com/paper/2505.04966/) (2025)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach](https://mlsanity.com/paper/2211.06398/) (2022)
- [You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism](https://mlsanity.com/paper/2110.14802/) (2021)
- [The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who](https://mlsanity.com/paper/2304.14238/) (2023)
- [The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors](https://mlsanity.com/paper/2509.04484/) (2025)
- [LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews](https://mlsanity.com/paper/2504.11042/) (2025)
- [Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond](https://mlsanity.com/paper/2201.11443/) (2022)
- [Experimental Standards for Deep Learning in Natural Language Processing Research](https://mlsanity.com/paper/2204.06251/) (2022)
- [SciCoQA: Quality Assurance for Scientific Paper--Code Alignment](https://mlsanity.com/paper/2601.12910/) (2026)
- [Large Language Models Penetration in Scholarly Writing and Peer Review](https://mlsanity.com/paper/2502.11193/) (2025)
- [Determining the Credibility of Science Communication](https://mlsanity.com/paper/2105.14473/) (2021)
- [What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review](https://mlsanity.com/paper/2205.01005/) (2022)
- [On Tables with Numbers, with Numbers](https://mlsanity.com/paper/2408.06062/) (2024)
- [When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews](https://mlsanity.com/paper/2310.18685/) (2023)
- [Is Peer-Reviewing Worth the Effort?](https://mlsanity.com/paper/2412.14351/) (2024)
- [Generating Summaries for Scientific Paper Review](https://mlsanity.com/paper/2109.14059/) (2021)
- [PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf](https://mlsanity.com/paper/2606.08857/) (2026)
- [Text similarity analysis for evaluation of descriptive answers](https://mlsanity.com/paper/2105.02935/) (2021)
- [What do writing features tell us about AI papers?](https://mlsanity.com/paper/2107.06310/) (2021)
- [NLP Meets the World: Toward Improving Conversations With the Public About Natural Language Processing Research](https://mlsanity.com/paper/2507.10559/) (2025)
- [Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time](https://mlsanity.com/paper/2601.15172/) (2026)
- [When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews](https://mlsanity.com/paper/2605.10171/) (2026)

## Abstract (arXiv)

Peer review is our best tool for judging the quality of conference submissions, but it is becoming increasingly spurious. We argue that a part of the problem is that the reviewers and area chairs face a poorly defined task forcing apples-to-oranges comparisons. There are several potential ways forward, but the key difficulty is creating the incentives and mechanisms for their consistent implementation in the NLP community.