mlsanityData to 14 Oct 2026

DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process

arXiv 2503.08569 ↗First posted 11 Mar 2025cs.CL cs.LGcited by 100

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large Language Models (LLMs) are increasingly utilized in scientific research assessment, particularly in automated paper review. However, existing LLM-based review systems face significant challenges, including limited domain expertise, hallucinated reasoning, and a lack of structured evaluation. To address these limitations, we introduce DeepReview, a multi-stage framework designed to emulate expert reviewers by incorporating structured analysis, literature retrieval, and evidence-based argumentation. Using DeepReview-13K, a curated dataset with structured annotations, we train DeepReviewer-14B, which outperforms CycleReviewer-70B with fewer tokens. In its best mode, DeepReviewer-14B achieves win rates of 88.21% and 80.20% against GPT-o1 and DeepSeek-R1 in evaluations. Our work sets a new benchmark for LLM-based paper review, with all resources publicly available. The code, model, dataset and demo have be released in http://ai-researcher.net.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 72 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

50 of 72 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning9 Sep

Other references · 49

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
37 more
  1. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  2. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  4. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  5. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  6. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  7. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  8. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  9. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 267
  10. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 164
  11. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  12. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 136
  13. 2023Emergent autonomous scientific research capabilities of large language modelscited by 132
  14. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  15. 2024MARG: Multi-Agent Review Generation for Scientific Paperscited by 104
  16. 2023Large Language Models for Automated Open-domain Scientific Hypotheses Discoverycited by 100
  17. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 93
  18. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  19. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 72
  20. 2024The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Ratescited by 69
  21. 2021Can We Automate Scientific Reviewing?cited by 63
  22. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  23. 2025Large language models for automated scholarly paper review: A surveycited by 61
  24. 2024Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideascited by 57
  25. 2020ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesiscited by 54
  26. 2024Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Reviewcited by 53
  27. 2024Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactionscited by 46
  28. 2024AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviewscited by 44
  29. 2024Automated Statistical Model Discovery with Language Modelscited by 41
  30. 2024Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysiscited by 33
  31. 2023Summarizing Multiple Documents with Conversational Structure for Meta-Review Generationcited by 27
  32. 2022CiteBench: A benchmark for Scientific Citation Text Generationcited by 22
  33. 2024ControlLM: Crafting Diverse Personalities for Language Modelscited by 15
  34. 2023Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluationcited by 14
  35. 2023Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networkscited by 12
  36. 2025Potential and Perils of Large Language Models as Judges of Unstructured Textual Datacited by 11
  37. 2023A Critical Examination of the Ethics of AI-Mediated Peer Reviewcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 35; 2026: 65.

100 papers.

Sort:

By area:

  1. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 128
  2. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 127
  3. 2025DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressivelycited by 63
  4. 2026Towards Execution-Grounded Automated AI Researchcited by 28
  5. 2025Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussionscited by 23
  6. 2026AutoFigure: Generating and Refining Publication-Ready Scientific Illustrationscited by 23
  7. 2025AI Scientists Fail Without Strong Implementation Capabilitycited by 21
  8. 2025When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Reviewcited by 19
  9. 2026AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilotcited by 19
  10. 2025OmniScientist: Toward a Co-evolving Ecosystem of Human and AI Scientistscited by 18
  11. 2025The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authorscited by 16
  12. 2025How Far Are AI Scientists from Changing the World?cited by 15
38 more
  1. 2025Prompt Injection Attacks on LLM Generated Reviews of Scientific Publicationscited by 13
  2. 2025Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluationcited by 12
  3. 2025Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedbackcited by 12
  4. 2025Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Paperscited by 12
  5. 2025Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Papercited by 12
  6. 2025From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Reviewcited by 11
  7. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 11
  8. 2025Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviewscited by 11
  9. 2026AI Can Learn Scientific Tastecited by 11
  10. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
  11. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  12. 2025LLM-REVal: Can We Trust LLM Reviewers Yet?cited by 9
  13. 2025GenoMAS: A Multi-Agent Framework for Scientific Discovery via Code-Driven Gene Expression Analysiscited by 8
  14. 2025TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agentscited by 8
  15. 2026Learning to Predict Future-Aligned Research Proposals with Language Modelscited by 8
  16. 2026MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibilitycited by 8
  17. 2025Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Modelscited by 7
  18. 2025XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaborationcited by 7
  19. 2026DeepReviewer 2.0: A Traceable Agentic System for Auditable Scientific Peer Reviewcited by 7
  20. 2026NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessmentcited by 7
  21. 2025When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injectioncited by 6
  22. 2026RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mindcited by 6
  23. 2026RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generationcited by 6
  24. 2026Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Futurecited by 6
  25. 2025"Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewerscited by 5
  26. 2025Automatic Paper Reviewing with Heterogeneous Graph Reasoning over LLM-Simulated Reviewer-Author Debatescited by 5
  27. 2026HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Executioncited by 5
  28. 2025NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimationcited by 4
  29. 2026ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Reviewcited by 4
  30. 2026Defend: Automated Rebuttals for Peer Review with Minimal Author Guidancecited by 4
  31. 2026GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responsescited by 4
  32. 2026SafeReview: Defending LLM-based Review Systems Against Adversarial Hidden Promptscited by 4
  33. 2026LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewerscited by 4
  34. 2025ResearStudio: A Human-Intervenable Framework for Building Controllable Deep-Research Agentscited by 3
  35. 2025ARCHE: A Novel Task to Evaluate LLMs on Latent Reasoning Chain Extractioncited by 3
  36. 2025Multi-Agent LLMs for Generating Research Limitationscited by 3
  37. 2026PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Reviewcited by 3
  38. 2026ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agentscited by 3

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation
  2. 2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews
  3. 2025Can Large Language Models Be Trusted Paper Reviewers? A Feasibility Study
  4. 2026LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform
  5. 2026PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers
  6. 2026When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation
  7. 2024Abstract2Appendix: Academic Reviews Enhance LLM Long-Context Capabilities
  8. 2024LitLLM: A Toolkit for Scientific Literature Review