# RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review

arXiv: https://arxiv.org/abs/2608.00005 · HTML: https://mlsanity.com/paper/2608.00005/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 76 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 30 of 32)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [CycleResearcher: Improving Automated Research via Automated Review](https://mlsanity.com/paper/2411.00816/) (2024)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment](https://mlsanity.com/paper/2510.07743/) (2025)
- [Checklists Are Better Than Reward Models For Aligning Language Models](https://mlsanity.com/paper/2507.18624/) (2025)
- [DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process](https://mlsanity.com/paper/2503.08569/) (2025)
- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning](https://mlsanity.com/paper/2508.16949/) (2025)
- [Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training](https://mlsanity.com/paper/2602.01511/) (2026)
- [Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training](https://mlsanity.com/paper/2509.21500/) (2025)
- [ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews](https://mlsanity.com/paper/2503.08506/) (2025)
- [TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review](https://mlsanity.com/paper/2506.07642/) (2025)
- [Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling](https://mlsanity.com/paper/2510.17314/) (2025)
- [Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics](https://mlsanity.com/paper/2602.10885/) (2026)
- [ReviewRL: Towards Automated Scientific Review with RL](https://mlsanity.com/paper/2508.10308/) (2025)
- [When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Review](https://mlsanity.com/paper/2509.09912/) (2025)
- [Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation](https://mlsanity.com/paper/2602.03619/) (2026)
- [The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors](https://mlsanity.com/paper/2509.04484/) (2025)
- [Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Papers](https://mlsanity.com/paper/2509.19326/) (2025)
- [Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework](https://mlsanity.com/paper/2508.21422/) (2025)
- [DeepReviewer 2.0: A Traceable Agentic System for Auditable Scientific Peer Review](https://mlsanity.com/paper/2604.09590/) (2026)
- [Automatic Paper Reviewing with Heterogeneous Graph Reasoning over LLM-Simulated Reviewer-Author Debates](https://mlsanity.com/paper/2511.08317/) (2025)
- [Reviewriter: AI-Generated Instructions For Peer Review Writing](https://mlsanity.com/paper/2506.04423/) (2025)
- [ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents](https://mlsanity.com/paper/2604.14261/) (2026)

## Built on by (most cited first, 2 of 2)

- [SearchJev: A Fast and Calibrated System-1 Model for Search Agents](https://mlsanity.com/paper/2610.05107/) (2026)
- [SwarmRSI: Recursive Self-Improvement through Evolving Scientific Agent Swarm](https://mlsanity.com/paper/2610.20431/) (2026)

## Abstract (arXiv)

Peer review at major venues is under unprecedented submission pressure, motivating the use of large language models (LLMs) as review assistants. Existing LLM-based reviewers, however, face two structural limitations. First, they map manuscripts directly to reviews, leaving the underlying rubric implicit and entangling its derivation with the judgement. Second, the prevailing paradigms each capture only half of a good review: training-free agents gather broad evidence but produce undirected critiques, while training-based reviewers inherit human discriminative judgement together with its noise and uneven coverage. We introduce RubricReviewer, a fully rubric-driven framework that addresses both limitations. It makes rubric generation an explicit intermediate step, so that both review generation and the final assessment are conditioned on paper-adaptive rubrics. It further combines a training-free agent (Scout) that gathers external evidence with a human-aligned trained model (Aligner) that consumes this evidence, fusing the strengths of both supervision sources. Experiments on real-world submissions show that RubricReviewer produces reviews that are markedly more comprehensive and more discriminative than prior systems, and exhibits the strongest robustness against adversarial prompt-injection attacks. Ablation studies further confirm the necessity of each component.