# Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach arXiv: https://arxiv.org/abs/2211.06398 ยท HTML: https://mlsanity.com/paper/2211.06398/ ## Key facts - Cited by 23 papers in the map (data through 14 Oct 2026). - Builds on 15 of its 25 references that are in the map. - First posted on arXiv 7 Nov 2022. ## Builds on (most cited first, 15 of 15) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016) - [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014) - [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020) - [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018) - [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021) - [ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis](https://mlsanity.com/paper/2010.06119/) (2020) - [Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment](https://mlsanity.com/paper/2109.09774/) (2021) - [Argument Mining for Understanding Peer Reviews](https://mlsanity.com/paper/1903.10104/) (2019) - [Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments](https://mlsanity.com/paper/2006.16437/) (2020) - [What Can We Do to Improve Peer Review in NLP?](https://mlsanity.com/paper/2010.03863/) (2020) - [Does My Rebuttal Matter? Insights from a Major NLP Conference](https://mlsanity.com/paper/1903.11367/) (2019) - [An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process](https://mlsanity.com/paper/2010.05137/) (2020) - [On Testing for Biases in Peer Review](https://mlsanity.com/paper/1912.13188/) (2019) - [Uncovering Latent Biases in Text: Method and Application to Peer Review](https://mlsanity.com/paper/2010.15300/) (2020) ## Built on by (most cited first, 23 of 23) - [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023) - [CycleResearcher: Improving Automated Research via Automated Review](https://mlsanity.com/paper/2411.00816/) (2024) - [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023) - [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024) - [Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis](https://mlsanity.com/paper/2407.12857/) (2024) - [Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions](https://mlsanity.com/paper/2505.07920/) (2025) - [MT-Mol:Multi Agent System with Tool-based Reasoning for Molecular Optimization](https://mlsanity.com/paper/2505.20820/) (2025) - [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025) - [From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review](https://mlsanity.com/paper/2506.11343/) (2025) - [RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance](https://mlsanity.com/paper/2406.10294/) (2024) - [Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future](https://mlsanity.com/paper/2604.27924/) (2026) - [Automatic Paper Reviewing with Heterogeneous Graph Reasoning over LLM-Simulated Reviewer-Author Debates](https://mlsanity.com/paper/2511.08317/) (2025) - [Estimating the Causal Effect of Early ArXiving on Paper Acceptance](https://mlsanity.com/paper/2306.13891/) (2023) - [Defend: Automated Rebuttals for Peer Review with Minimal Author Guidance](https://mlsanity.com/paper/2603.27360/) (2026) - [Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews](https://mlsanity.com/paper/2604.19502/) (2026) - [SciEvo: A 2 Million, 30-Year Cross-disciplinary Dataset for Temporal Scientometric Analysis](https://mlsanity.com/paper/2410.09510/) (2024) - [Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process](https://mlsanity.com/paper/2502.00874/) (2025) - [Disparities in Peer Review Tone and the Role of Reviewer Anonymity](https://mlsanity.com/paper/2507.14741/) (2025) - [When AI reviews science: Can we trust the referee?](https://mlsanity.com/paper/2604.23593/) (2026) - [A global AI community requires language-diverse publishing](https://mlsanity.com/paper/2408.14772/) (2024) - [Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI](https://mlsanity.com/paper/2604.19578/) (2026) - [EGTR-Review: Efficient Evidence-Grounded Scientific Peer Review Generation via Multi-Agent Teacher Distillation](https://mlsanity.com/paper/2606.06025/) (2026) - [Modeling LLM Agent Reviewer Dynamics in Elo-Ranked Review System](https://mlsanity.com/paper/2601.08829/) (2026) ## Abstract (arXiv) Double-blind peer review mechanism has become the skeleton of academic research across multiple disciplines including computer science, yet several studies have questioned the quality of peer reviews and raised concerns on potential biases in the process. In this paper, we conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs). We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date by aggregating data from OpenReview, Google Scholar, arXiv, and CSRanking, and extracting high-level features using language models. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige. We observe that the level of disparity differs and textual features are essential in reducing biases in the predictive modeling. We distill several insights from our analysis on study the peer review process with the help of large LMs. Our database also provides avenues for studying new natural language processing (NLP) methods that facilitate the understanding of the peer review mechanism. We study a concrete example towards automatic machine review systems and provide baseline models for the review generation and scoring tasks such that the database can be used as a benchmark.