mlsanityData to 7 Oct 2026

What Can We Do to Improve Peer Review in NLP?

arXiv 2010.03863 ↗First posted 8 Oct 2020cs.CL cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

Peer review is our best tool for judging the quality of conference submissions, but it is becoming increasingly spurious. We argue that a part of the problem is that the reviewers and area chairs face a poorly defined task forcing apples-to-oranges comparisons. There are several potential ways forward, but the key difficulty is creating the incentives and mechanisms for their consistent implementation in the NLP community.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 6; 2023: 2; 2024: 4; 2025: 5; 2026: 6.

32 papers.

Sort:

By area:

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  3. 2021Can We Automate Scientific Reviewing?cited by 63
  4. 2024OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviewscited by 63
  5. 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
  6. 2021Changing the World by Changing the Datacited by 51
  7. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  8. 2025Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewardscited by 34
  9. 2024What Can Natural Language Processing Do for Peer Review?cited by 24
  10. 2022Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approachcited by 23
  11. 2023The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Whocited by 19
  12. 2021You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanismcited by 18
20 more
  1. 2025The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authorscited by 16
  2. 2025LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviewscited by 13
  3. 2022Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyondcited by 12
  4. 2022Experimental Standards for Deep Learning in Natural Language Processing Researchcited by 9
  5. 2026SciCoQA: Quality Assurance for Scientific Paper--Code Alignmentcited by 9
  6. 2025Large Language Models Penetration in Scholarly Writing and Peer Reviewcited by 6
  7. 2021Determining the Credibility of Science Communicationcited by 4
  8. 2022What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Reviewcited by 4
  9. 2024On Tables with Numbers, with Numberscited by 4
  10. 2023When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviewscited by 2
  11. 2024Is Peer-Reviewing Worth the Effort?cited by 2
  12. 2021Generating Summaries for Scientific Paper Reviewcited by 1
  13. 2021Text similarity analysis for evaluation of descriptive answerscited by 0
  14. 2021What do writing features tell us about AI papers?cited by 0
  15. 2025NLP Meets the World: Toward Improving Conversations With the Public About Natural Language Processing Researchcited by 0
  16. 2026Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Timecited by 0
  17. 2026When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviewscited by 0
  18. 2026PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleafcited by 0
  19. 2026Position: Want Better ML Reviews? Stop Asking Nicely and Start Incentivizing with a Credit Systemcited by 0
  20. 2026Peerify: Benchmarking Peer-Review Claim Verificationcited by 0