mlsanityData to 14 Oct 2026

Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews

arXiv 2509.13400 ↗First posted 16 Sep 2025cs.CY cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

The adoption of large language models (LLMs) is transforming the peer review process, from assisting reviewers in writing detailed evaluations to generating entire reviews automatically. While these capabilities offer new opportunities, they also raise concerns about fairness and reliability. In this paper, we investigate bias in LLM-generated peer reviews through controlled interventions on author metadata, including affiliation, gender, seniority, and publication history. Our analysis consistently shows a strong affiliation bias favoring authors from highly ranked institutions. We also identify directional preferences associated with seniority and prior publication record, which can influence acceptance decisions for borderline papers. Gender effects are smaller but present in several models. Notably, implicit biases become more pronounced when examining token-level soft ratings, suggesting that alignment may mask but not fully eliminate underlying preferences

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 13 of its 32 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

13 of 32 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
  3. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 199
  4. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 100
  5. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 98
  6. 2024The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Ratescited by 69
  7. 2025Large language models for automated scholarly paper review: A surveycited by 61
  8. 2024Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Reviewcited by 53
  9. 2023Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experimentcited by 34
  10. 2025Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviewscited by 24
  11. 2025From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Reviewcited by 11
  12. 2024Certifying Counterfactual Bias in LLMscited by 10
1 more
  1. 2024A Theory of Response Sampling in LLMs: Part Descriptive and Part Prescriptivecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026Paraphrasing Adversarial Attack on LLM-as-a-Reviewercited by 8
  2. 2025Gen-Review: A Large-scale Dataset of AI-Generated (and Human-written) Peer Reviewscited by 4
  3. 2026PeerPrism: Peer Evaluation Expertise vs Review-writing AIcited by 4
  4. 2026Inspectable AI for Science: A Research Object Approach to Generative AI Governancecited by 2
  5. 2026No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisionscited by 2
  6. 2026The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishingcited by 1
  7. 2026Are we still able to recognize pearls? Machine-driven peer review and the risk to creativity: An explainable RAG-XAI detection framework with markers extractioncited by 0
  8. 2026Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detectioncited by 0
  9. 2026Need or Reputation? Region, National Income, and Academic Status in LLM Gatekeeping Decisionscited by 0
  10. 2026Who Pays for Open Review? Visible Author Reputation and Its Effect on Ratingscited by 0
  11. 2026A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Reviewcited by 0