mlsanityData to 7 Oct 2026

Design and Analysis of the NIPS 2016 Review Process

arXiv 1708.09794 ↗First posted 31 Aug 2017cs.DL cs.LG cs.SIcited by 33

Not on the map. Why?

Abstract (arXiv)

Neural Information Processing Systems (NIPS) is a top-tier annual conference in machine learning. The 2016 edition of the conference comprised more than 2,400 paper submissions, 3,000 reviewers, and 8,000 attendees. This represents a growth of nearly 40% in terms of submissions, 96% in terms of reviewers, and over 100% in terms of attendees as compared to the previous year. The massive scale as well as rapid growth of the conference calls for a thorough quality assessment of the peer-review process and novel means of improvement. In this paper, we analyze several aspects of the data collected during the review process, including an experiment investigating the efficacy of collecting ordinal rankings from reviewers. Our goal is to check the soundness of the review process, and provide insights that may be useful in the design of the review process of subsequent conferences.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 9; 2023: 4; 2024: 4; 2025: 4; 2026: 7.

33 papers.

Sort:

  1. 2024OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviewscited by 63
  2. 2024What Can Natural Language Processing Do for Peer Review?cited by 24
  3. 2025Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attackscited by 14
  4. 2021Making Paper Reviewing Robust to Bid Manipulation Attackscited by 12
  5. 2023A Gold Standard Dataset for the Reviewer Assignment Problemcited by 11
  6. 2022A Dataset on Malicious Paper Bidding in Peer Reviewcited by 9
  7. 2024On the Detection of Reviewer-Author Collusion Rings From Paper Biddingcited by 7
  8. 2024The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Reviewcited by 7
  9. 2021Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Designcited by 6
  10. 2022Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertisecited by 6
  11. 2022How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?cited by 5
  12. 2021Some Ethical Issues in the Review Process of Machine Learning Conferencescited by 4
21 more
  1. 2022Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer Assignmentcited by 4
  2. 2023Assisting Human Decisions in Document Matchingcited by 3
  3. 2023Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer Reviewcited by 3
  4. 2025How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Reviewcited by 3
  5. 2021Peer Selection with Noisy Assessmentscited by 2
  6. 2021Auctions and Peer Prediction for Academic Peer Reviewcited by 2
  7. 2023When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviewscited by 2
  8. 2026FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verificationcited by 2
  9. 2022A Unified Statistical Learning Model for Rankings and Scores with Application to Grant Panel Reviewcited by 1
  10. 2026greCAPTCHA: Assessing Understanding as Evidence of Research Authorship Under Generative AIcited by 1
  11. 2022Modeling and Correcting Bias in Sequential Evaluationcited by 0
  12. 2022Allocation Schemes in Analytic Evaluation: Applicant-Centric Holistic or Attribute-Centric Segmented?cited by 0
  13. 2022No Agreement Without Loss: Learning and Social Choice in Peer Reviewcited by 0
  14. 2022Participation Interfaces for Human-Centered AIcited by 0
  15. 2025Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Reviewcited by 0
  16. 2025Which Coauthor Should I Nominate in My 99 ICLR Submissions? A Mathematical Analysis of the ICLR 2026 Reciprocal Reviewer Nomination Policycited by 0
  17. 2026Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Rankingcited by 0
  18. 2026Commitment Checklist: Auditing Author Commitments in Peer Reviewcited by 0
  19. 2026The IJCNN 2025 Review Processcited by 0
  20. 2026LitPivot: Developing Well-Situated Research Ideas Through Dynamic Contextualization and Critique within the Literature Landscapecited by 0
  21. 2026When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviewscited by 0