# REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning

arXiv: https://arxiv.org/abs/2505.11718 · HTML: https://mlsanity.com/paper/2505.11718/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 16 May 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024)
- [Deep Reinforcement Learning for Multi-objective Optimization](https://mlsanity.com/paper/1906.02386/) (2019)
- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis](https://mlsanity.com/paper/2010.06119/) (2020)
- [Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges](https://mlsanity.com/paper/2412.03220/) (2024)
- [ReviewEval: An Evaluation Framework for AI-Generated Reviews](https://mlsanity.com/paper/2502.11736/) (2025)
- [What Have We Learned from OpenReview?](https://mlsanity.com/paper/2103.05885/) (2021)

## Built on by (most cited first, 11 of 11)

- [OmniScientist: Toward a Co-evolving Ecosystem of Human and AI Scientists](https://mlsanity.com/paper/2511.16931/) (2025)
- [Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Papers](https://mlsanity.com/paper/2509.19326/) (2025)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future](https://mlsanity.com/paper/2604.27924/) (2026)
- [ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review](https://mlsanity.com/paper/2601.22638/) (2026)
- [REM-CTX: Automated Peer Review via Reinforcement Learning with Auxiliary Context](https://mlsanity.com/paper/2604.00248/) (2026)
- [CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers](https://mlsanity.com/paper/2605.07905/) (2026)
- [LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges](https://mlsanity.com/paper/2606.25057/) (2026)
- [InternReviewer & InternAdvocate: Objective Reward and Evaluation for Agentic Reinforcement Learning in Peer Review and Rebuttal](https://mlsanity.com/paper/2608.28612/) (2026)
- [PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress](https://mlsanity.com/paper/2609.16995/) (2026)

## Abstract (arXiv)

AI-based peer review systems tend to produce shallow and overpraising suggestions compared to human feedback. Here, we evaluate how well a reasoning LLM trained with multi-objective reinforcement learning (REMOR) can overcome these limitations. We start by designing a multi-aspect reward function that aligns with human evaluation of reviews. The aspects are related to the review itself (e.g., criticisms, novelty) and the relationship between the review and the manuscript (i.e., relevance). First, we perform supervised fine-tuning of DeepSeek-R1-Distill-Qwen-7B using LoRA on PeerRT, a new dataset of high-quality top AI conference reviews enriched with reasoning traces. We then apply Group Relative Policy Optimization (GRPO) to train two models: REMOR-H (with the human-aligned reward) and REMOR-U (with a uniform reward). Interestingly, the human-aligned reward penalizes aspects typically associated with strong reviews, leading REMOR-U to produce qualitatively more substantive feedback. Our results show that REMOR-U and REMOR-H achieve more than twice the average rewards of human reviews, non-reasoning state-of-the-art agentic multi-modal AI review systems, and general commercial LLM baselines. We found that while the best AI and human reviews are comparable in quality, REMOR avoids the long tail of low-quality human reviews. We discuss how reasoning is key to achieving these improvements and release the Human-aligned Peer Review Reward (HPRR) function, the Peer Review Reasoning-enriched Traces (PeerRT) dataset, and the REMOR models, which we believe can help spur progress in the area.