# Position: Want Better ML Reviews? Stop Asking Nicely and Start Incentivizing with a Credit System

arXiv: https://arxiv.org/abs/2608.14571 · HTML: https://mlsanity.com/paper/2608.14571/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment](https://mlsanity.com/paper/2109.09774/) (2021)
- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment](https://mlsanity.com/paper/2306.03262/) (2023)
- [Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards](https://mlsanity.com/paper/2505.04966/) (2025)
- [What Can We Do to Improve Peer Review in NLP?](https://mlsanity.com/paper/2010.03863/) (2020)
- [Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments](https://mlsanity.com/paper/2311.09497/) (2023)
- [You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism](https://mlsanity.com/paper/2110.14802/) (2021)
- [Tempest: Autonomous Multi-Turn Jailbreaking of Large Language Models with Tree Search](https://mlsanity.com/paper/2503.10619/) (2025)
- [A System-Level Analysis of Conference Peer Review](https://mlsanity.com/paper/2303.09020/) (2023)
- [The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review](https://mlsanity.com/paper/2408.13430/) (2024)
- [On the Detection of Reviewer-Author Collusion Rings From Paper Bidding](https://mlsanity.com/paper/2402.07860/) (2024)
- [What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review](https://mlsanity.com/paper/2205.01005/) (2022)
- [ML Researchers Support Openness in Peer Review But Are Concerned About Resubmission Bias](https://mlsanity.com/paper/2511.23439/) (2025)

## Abstract (arXiv)

With soaring submission counts, stricter reciprocal review policies, widespread adoption of platforms like OpenReview, and without the offsetting pressure of publication fees, the machine learning (ML) community has one of the largest scholarly presences among all scientific fields. And yet, textbfalmost textiteveryone has textitmany unpleasant things to share about their review experience. Worse, there is little public space to seriously discuss, let alone debate, what makes a review system effective or how it might be improved.quad In this position paper, we expand our discussion from two core problems: textitHow can we reasonably limit submission volume? and textitHow can we incentivize good and discourage bad reviewing? We first assess the strengths and shortcomings of existing attempts to address such problems. Specifically, we present four takes on some popular conference mechanisms and propose two alternative designs for improvement.quad Our general position is that meaningful improvement in ML peer review won't come from polite best-practice suggestions tucked into Calls for Papers or Reviewer Guidelines: it requires enforceable yet fine-grained procedural safeguards paired with textbfa currency-like credit system (e.g., our proposed textitOpenReview Points). ML practitioners can ``earn'' such points by contributing good review practices, and ``spend'' them across one or multiple major conferences to redeem different kinds of ``perks,'' such as complimentary registration or the right to request additional review resources.