# Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking

arXiv: https://arxiv.org/abs/2602.06078 · HTML: https://mlsanity.com/paper/2602.06078/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Builds on (most cited first, 21 of 21)

- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process](https://mlsanity.com/paper/2503.08569/) (2025)
- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study](https://mlsanity.com/paper/2307.05492/) (2023)
- [Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments](https://mlsanity.com/paper/2006.16437/) (2020)
- [Design and Analysis of the NIPS 2016 Review Process](https://mlsanity.com/paper/1708.09794/) (2017)
- [PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review](https://mlsanity.com/paper/1806.06237/) (2018)
- [Paper Matching with Local Fairness Constraints](https://mlsanity.com/paper/1905.11924/) (2019)
- [You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism](https://mlsanity.com/paper/2110.14802/) (2021)
- [Matching Papers and Reviewers at Large Conferences](https://mlsanity.com/paper/2202.12273/) (2022)
- [Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluation](https://mlsanity.com/paper/2505.23824/) (2025)
- [Counterfactual Evaluation of Peer-Review Assignment Policies](https://mlsanity.com/paper/2305.17339/) (2023)
- [From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review](https://mlsanity.com/paper/2506.11343/) (2025)
- [Calibrating "Cheap Signals" in Peer Review without a Prior](https://mlsanity.com/paper/2312.07269/) (2023)
- [The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review](https://mlsanity.com/paper/2408.13430/) (2024)
- [LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval](https://mlsanity.com/paper/2510.26995/) (2025)
- [A One-Size-Fits-All Approach to Improving Randomness in Paper Assignment](https://mlsanity.com/paper/2310.05995/) (2023)
- [Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models](https://mlsanity.com/paper/2512.22145/) (2025)
- [Arbitrariness of peer review: A Bayesian analysis of the NIPS experiment](https://mlsanity.com/paper/1507.06411/) (2015)
- [NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimation](https://mlsanity.com/paper/2509.25179/) (2025)

## Abstract (arXiv)

This paper argues that large ML conferences should allocate marginal review capacity primarily to papers near the acceptance boundary, rather than spreading extra reviews via random or affinity-driven heuristics. We propose using LLM-based comparative ranking (via pairwise comparisons and a Bradley--Terry model) to identify a borderline band before human reviewing and to allocate marginal reviewer capacity at assignment time. Concretely, given a venue-specific minimum review target (e.g., 3 or 4), we use this signal to decide which papers receive one additional review (e.g., a 4th or 5th), without conditioning on any human reviews and without using LLM outputs for accept/reject. We provide a simple expected-impact calculation in terms of (i) the overlap between the predicted and true borderline sets (ρ) and (ii) the incremental value of an extra review near the boundary (Delta), and we provide retrospective proxies to estimate these quantities.