# Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective

arXiv: https://arxiv.org/abs/2109.01190 · HTML: https://mlsanity.com/paper/2109.01190/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 37 references that are in the map.
- First posted on arXiv 2 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Argument Mining for Understanding Peer Reviews](https://mlsanity.com/paper/1903.10104/) (2019)
- [Does My Rebuttal Matter? Insights from a Major NLP Conference](https://mlsanity.com/paper/1903.11367/) (2019)
- [Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings](https://mlsanity.com/paper/1806.05085/) (2018)
- [Examining Citations of Natural Language Processing Literature](https://mlsanity.com/paper/2005.00912/) (2020)
- [Finding Convincing Arguments Using Scalable Bayesian Preference Learning](https://mlsanity.com/paper/1806.02418/) (2018)
- [Structure-Tags Improve Text Classification for Scholarly Document Quality Prediction](https://mlsanity.com/paper/2005.00129/) (2020)

## Built on by (most cited first, 3 of 3)

- [NLPeer: A Unified Resource for the Computational Study of Peer Review](https://mlsanity.com/paper/2211.06651/) (2022)
- [Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond](https://mlsanity.com/paper/2201.11443/) (2022)
- [STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond](https://mlsanity.com/paper/2409.05367/) (2024)

## Abstract (arXiv)

Peer review is the primary means of quality control in academia; as an outcome of a peer review process, program and area chairs make acceptance decisions for each paper based on the review reports and scores they received. Quality of scientific work is multi-faceted; coupled with the subjectivity of reviewing, this makes final decision making difficult and time-consuming. To support this final step of peer review, we formalize it as a paper ranking problem. We introduce a novel, multi-faceted generic evaluation framework for ranking submissions based on peer reviews that takes into account effectiveness, efficiency and fairness. We propose a preference learning perspective on the task that considers both review texts and scores to alleviate the inevitable bias and noise in reviews. Our experiments on peer review data from the ACL 2018 conference demonstrate the superiority of our preference-learning-based approach over baselines and prior work, while highlighting the importance of using both review texts and scores to rank submissions.