# Design and Analysis of the NIPS 2016 Review Process

arXiv: https://arxiv.org/abs/1708.09794 · HTML: https://mlsanity.com/paper/1708.09794/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2017.

## Built on by (most cited first, 30 of 33)

- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attacks](https://mlsanity.com/paper/2506.11113/) (2025)
- [Making Paper Reviewing Robust to Bid Manipulation Attacks](https://mlsanity.com/paper/2102.06020/) (2021)
- [A Gold Standard Dataset for the Reviewer Assignment Problem](https://mlsanity.com/paper/2303.16750/) (2023)
- [A Dataset on Malicious Paper Bidding in Peer Review](https://mlsanity.com/paper/2207.02303/) (2022)
- [The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review](https://mlsanity.com/paper/2408.13430/) (2024)
- [On the Detection of Reviewer-Author Collusion Rings From Paper Bidding](https://mlsanity.com/paper/2402.07860/) (2024)
- [Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design](https://mlsanity.com/paper/2108.06371/) (2021)
- [Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertise](https://mlsanity.com/paper/2201.10631/) (2022)
- [How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?](https://mlsanity.com/paper/2211.12966/) (2022)
- [Some Ethical Issues in the Review Process of Machine Learning Conferences](https://mlsanity.com/paper/2106.00810/) (2021)
- [Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer Assignment](https://mlsanity.com/paper/2207.11315/) (2022)
- [How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review](https://mlsanity.com/paper/2510.02143/) (2025)
- [Assisting Human Decisions in Document Matching](https://mlsanity.com/paper/2302.08450/) (2023)
- [Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer Review](https://mlsanity.com/paper/2309.05457/) (2023)
- [Peer Selection with Noisy Assessments](https://mlsanity.com/paper/2107.10121/) (2021)
- [Auctions and Peer Prediction for Academic Peer Review](https://mlsanity.com/paper/2109.00923/) (2021)
- [When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews](https://mlsanity.com/paper/2310.18685/) (2023)
- [FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification](https://mlsanity.com/paper/2604.04074/) (2026)
- [A Unified Statistical Learning Model for Rankings and Scores with Application to Grant Panel Review](https://mlsanity.com/paper/2201.02539/) (2022)
- [Commitment Checklist: Auditing Author Commitments in Peer Review](https://mlsanity.com/paper/2603.00003/) (2026)
- [LitPivot: Developing Well-Situated Research Ideas Through Dynamic Contextualization and Critique within the Literature Landscape](https://mlsanity.com/paper/2604.02600/) (2026)
- [greCAPTCHA: Assessing Understanding as Evidence of Research Authorship Under Generative AI](https://mlsanity.com/paper/2609.20481/) (2026)
- [Modeling and Correcting Bias in Sequential Evaluation](https://mlsanity.com/paper/2205.01607/) (2022)
- [Allocation Schemes in Analytic Evaluation: Applicant-Centric Holistic or Attribute-Centric Segmented?](https://mlsanity.com/paper/2209.08665/) (2022)
- [No Agreement Without Loss: Learning and Social Choice in Peer Review](https://mlsanity.com/paper/2211.02144/) (2022)
- [Participation Interfaces for Human-Centered AI](https://mlsanity.com/paper/2211.08419/) (2022)
- [Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review](https://mlsanity.com/paper/2501.13014/) (2025)
- [Which Coauthor Should I Nominate in My 99 ICLR Submissions? A Mathematical Analysis of the ICLR 2026 Reciprocal Reviewer Nomination Policy](https://mlsanity.com/paper/2512.17950/) (2025)

## Abstract (arXiv)

Neural Information Processing Systems (NIPS) is a top-tier annual conference in machine learning. The 2016 edition of the conference comprised more than 2,400 paper submissions, 3,000 reviewers, and 8,000 attendees. This represents a growth of nearly 40% in terms of submissions, 96% in terms of reviewers, and over 100% in terms of attendees as compared to the previous year. The massive scale as well as rapid growth of the conference calls for a thorough quality assessment of the peer-review process and novel means of improvement. In this paper, we analyze several aspects of the data collected during the review process, including an experiment investigating the efficacy of collecting ordinal rankings from reviewers. Our goal is to check the soundness of the review process, and provide insights that may be useful in the design of the review process of subsequent conferences.