# When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews

arXiv: https://arxiv.org/abs/2310.18685 · HTML: https://mlsanity.com/paper/2310.18685/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 46 references that are in the map.
- First posted on arXiv 28 Oct 2023.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Argument Mining for Understanding Peer Reviews](https://mlsanity.com/paper/1903.10104/) (2019)
- [Natural Language Inference over Interaction Space](https://mlsanity.com/paper/1709.04348/) (2017)
- [Design and Analysis of the NIPS 2016 Review Process](https://mlsanity.com/paper/1708.09794/) (2017)
- [What Can We Do to Improve Peer Review in NLP?](https://mlsanity.com/paper/2010.03863/) (2020)
- [Aspect-based Sentiment Analysis of Scientific Reviews](https://mlsanity.com/paper/2006.03257/) (2020)
- [On Testing for Biases in Peer Review](https://mlsanity.com/paper/1912.13188/) (2019)
- [Multi-Instance Multi-Label Learning Networks for Aspect-Category Sentiment Analysis](https://mlsanity.com/paper/2010.02656/) (2020)
- [Contradiction Detection for Rumorous Claims](https://mlsanity.com/paper/1611.02588/) (2016)
- [Macro-Average: Rare Types Are Important Too](https://mlsanity.com/paper/2104.05700/) (2021)

## Built on by (most cited first, 2 of 2)

- [Can Large Language Models Unlock Novel Scientific Research Ideas?](https://mlsanity.com/paper/2409.06185/) (2024)
- [AI-Assisted Authoring for Transparent, Data-Driven Documents](https://mlsanity.com/paper/2601.06027/) (2025)

## Abstract (arXiv)

To this date, the efficacy of the scientific publishing enterprise fundamentally rests on the strength of the peer review process. The journal editor or the conference chair primarily relies on the expert reviewers' assessment, identify points of agreement and disagreement and try to reach a consensus to make a fair and informed decision on whether to accept or reject a paper. However, with the escalating number of submissions requiring review, especially in top-tier Artificial Intelligence (AI) conferences, the editor/chair, among many other works, invests a significant, sometimes stressful effort to mitigate reviewer disagreements. Here in this work, we introduce a novel task of automatically identifying contradictions among reviewers on a given article. To this end, we introduce ContraSciView, a comprehensive review-pair contradiction dataset on around 8.5k papers (with around 28k review pairs containing nearly 50k review pair comments) from the open review-based ICLR and NeurIPS conferences. We further propose a baseline model that detects contradictory statements from the review pairs. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically. We make our dataset and code public for further investigations.