# Modeling LLM Agent Reviewer Dynamics in Elo-Ranked Review System

arXiv: https://arxiv.org/abs/2601.08829 · HTML: https://mlsanity.com/paper/2601.08829/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 10 of 10)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024)
- [Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach](https://mlsanity.com/paper/2211.06398/) (2022)
- [Counterfactual Evaluation of Peer-Review Assignment Policies](https://mlsanity.com/paper/2305.17339/) (2023)
- [What makes a successful rebuttal in computer science conferences? : A perspective on social interaction](https://mlsanity.com/paper/2307.03371/) (2023)
- [A System-Level Analysis of Conference Peer Review](https://mlsanity.com/paper/2303.09020/) (2023)
- [Does double-blind peer-review reduce bias? Evidence from a top computer science conference](https://mlsanity.com/paper/2101.02701/) (2021)
- [Peer Prediction for Peer Review: Designing a Marketplace for Ideas](https://mlsanity.com/paper/2303.16855/) (2023)
- [Reviewer assignment problem: A scoping review](https://mlsanity.com/paper/2305.07887/) (2023)
- [Distinguishing articles in questionable and non-questionable journals using quantitative indicators associated with quality](https://mlsanity.com/paper/2405.06308/) (2024)

## Abstract (arXiv)

In this work, we explore the Large Language Model (LLM) agent reviewer dynamics in an Elo-ranked review system using real-world conference paper submissions. Multiple LLM agent reviewers with different personas are engage in multi round review interactions moderated by an Area Chair. We compare a baseline setting with conditions that incorporate Elo ratings and reviewer memory. Our simulation results showcase several interesting findings, including how incorporating Elo improves Area Chair decision accuracy, as well as reviewers' adaptive review strategy that exploits our Elo system without improving review effort. Our code is available at https://github.com/hsiangwei0903/EloReview.