# Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers

arXiv: https://arxiv.org/abs/2507.02694 · HTML: https://mlsanity.com/paper/2507.02694/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2025.

## Built on by (most cited first, 20 of 20)

- [MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems](https://mlsanity.com/paper/2510.17281/) (2025)
- [AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot](https://mlsanity.com/paper/2604.13940/) (2026)
- [FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers](https://mlsanity.com/paper/2511.21843/) (2025)
- [OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment](https://mlsanity.com/paper/2601.01576/) (2026)
- [NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment](https://mlsanity.com/paper/2604.11543/) (2026)
- [RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation](https://mlsanity.com/paper/2603.09723/) (2026)
- [To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysis](https://mlsanity.com/paper/2512.05925/) (2025)
- [Multi-Agent LLMs for Generating Research Limitations](https://mlsanity.com/paper/2601.11578/) (2025)
- [SciMDR: Advancing Scientific Multimodal Document Reasoning](https://mlsanity.com/paper/2603.12249/) (2026)
- [From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation](https://mlsanity.com/paper/2603.17588/) (2026)
- [What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Review](https://mlsanity.com/paper/2604.19998/) (2026)
- [EchoReview: Learning Peer Review from the Echoes of Scientific Citations](https://mlsanity.com/paper/2602.00733/) (2026)
- [AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality](https://mlsanity.com/paper/2608.03581/) (2026)
- [Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGI](https://mlsanity.com/paper/2511.18517/) (2025)
- [Scientific Knowledge Discovery in the Age of Large Language Models](https://mlsanity.com/paper/2607.26670/) (2026)
- [Tree-of-Concerns: Hierarchical Multi-Agent Debate for Unstated-Limitation Extraction in Scientific Critique](https://mlsanity.com/paper/2608.20777/) (2026)
- [ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation](https://mlsanity.com/paper/2609.09076/) (2026)
- [What Limits Us? Analyzing Self-Reported Limitations in NLP Research](https://mlsanity.com/paper/2609.15191/) (2026)
- [BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction](https://mlsanity.com/paper/2609.24921/) (2026)
- [IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation](https://mlsanity.com/paper/2610.04074/) (2026)

## Abstract (arXiv)

Peer review is fundamental to scientific research, but the growing volume of publications has intensified the challenges of this expertise-intensive process. While LLMs show promise in various scientific tasks, their potential to assist with peer review, particularly in identifying paper limitations, remains understudied. We first present a comprehensive taxonomy of limitation types in scientific research, with a focus on AI. Guided by this taxonomy, for studying limitations, we present LimitGen, the first comprehensive benchmark for evaluating LLMs' capability to support early-stage feedback and complement human peer review. Our benchmark consists of two subsets: LimitGen-Syn, a synthetic dataset carefully created through controlled perturbations of high-quality papers, and LimitGen-Human, a collection of real human-written limitations. To improve the ability of LLM systems to identify limitations, we augment them with literature retrieval, which is essential for grounding identifying limitations in prior scientific findings. Our approach enhances the capabilities of LLM systems to generate limitations in research papers, enabling them to provide more concrete and constructive feedback.