# What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

arXiv: https://arxiv.org/abs/2205.01005 · HTML: https://mlsanity.com/paper/2205.01005/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 36 references that are in the map.
- First posted on arXiv 2 May 2022.

## Builds on (most cited first, 8 of 8)

- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment](https://mlsanity.com/paper/2109.09774/) (2021)
- [What Can We Do to Improve Peer Review in NLP?](https://mlsanity.com/paper/2010.03863/) (2020)
- [PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review](https://mlsanity.com/paper/1806.06237/) (2018)
- [Simple and Effective Paraphrastic Similarity from Parallel Translations](https://mlsanity.com/paper/1909.13872/) (2019)
- [A Study of Position Bias in Digital Library Recommender Systems](https://mlsanity.com/paper/1802.06565/) (2018)
- [Combating Collusion Rings is Hard but Possible](https://mlsanity.com/paper/2112.08444/) (2021)

## Built on by (most cited first, 4 of 4)

- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [Assisting Human Decisions in Document Matching](https://mlsanity.com/paper/2302.08450/) (2023)
- [Position: Want Better ML Reviews? Stop Asking Nicely and Start Incentivizing with a Credit System](https://mlsanity.com/paper/2608.14571/) (2026)

## Abstract (arXiv)

Both scientific progress and individual researcher careers depend on the quality of peer review, which in turn depends on paper-reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a matter where different stakeholders (area chairs, reviewers, authors) have accumulated experience worth taking into account. We present the results of the first survey of the NLP community, identifying common issues and perspectives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for improving future NLP conferences, and desiderata for interpretable peer review assignments.