# PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs

arXiv: https://arxiv.org/abs/2604.21304 · HTML: https://mlsanity.com/paper/2604.21304/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 27 of its 54 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)
- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 27 of 27)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning](https://mlsanity.com/paper/2205.00445/) (2022)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models](https://mlsanity.com/paper/2403.00231/) (2024)
- [M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding](https://mlsanity.com/paper/2411.04952/) (2024)
- [Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.01441/) (2025)
- [MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understanding](https://mlsanity.com/paper/2503.13964/) (2025)
- [Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents](https://mlsanity.com/paper/2601.03515/) (2026)
- [Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer](https://mlsanity.com/paper/2402.15173/) (2024)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models](https://mlsanity.com/paper/2406.11633/) (2024)
- [Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL](https://mlsanity.com/paper/2602.13035/) (2026)
- [Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions](https://mlsanity.com/paper/2505.07920/) (2025)
- [PeerQA: A Scientific Question Answering Dataset from Peer Reviews](https://mlsanity.com/paper/2502.13668/) (2025)
- [PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature](https://mlsanity.com/paper/2510.10909/) (2025)
- [FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed](https://mlsanity.com/paper/2506.09034/) (2025)
- [RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training](https://mlsanity.com/paper/2510.00911/) (2025)
- [MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains](https://mlsanity.com/paper/2603.00873/) (2026)
- [CG-RAG: Research Question Answering by Citation Graph Retrieval-Augmented LLMs](https://mlsanity.com/paper/2501.15067/) (2025)
- [DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections](https://mlsanity.com/paper/2508.15851/) (2025)
- [MMCR: Benchmarking Cross-Source Reasoning in Scientific Papers](https://mlsanity.com/paper/2503.16856/) (2025)
- [RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation](https://mlsanity.com/paper/2506.05070/) (2025)

## Built on by (most cited first, 4 of 4)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning](https://mlsanity.com/paper/2607.02509/) (2026)
- [Harnessing Generalist Agents for Contextualized Time Series](https://mlsanity.com/paper/2606.05404/) (2026)

## Abstract (arXiv)

Understanding scientific papers requires more than answering isolated questions or summarizing content. It involves an integrated reasoning process that grounds textual and visual information, interprets experimental evidence, synthesizes information across sources, and critically evaluates scientific claims. However, existing benchmarks typically assess these abilities in isolation, making it difficult to evaluate scientific paper understanding as a unified set of interacting cognitive abilities. In this work, we introduce PaperMind, a benchmark designed to evaluate integrated and agent-oriented scientific reasoning over research papers. PaperMind is constructed from real scientific papers across seven domains, including agriculture, biology, chemistry, computer science, medicine, physics, and economics. It comprises four complementary task families that collectively operationalize distinct cognitive facets of scientific paper reasoning, including multimodal grounding, experimental interpretation, cross-source evidence reasoning, and critical assessment. By analyzing model behavior across multiple tasks, PaperMind enables a diagnostic evaluation of integrated scientific reasoning behaviors that are difficult to assess through isolated task evaluations. Extensive experiments on both opensource and closed-source multimodal LLMs reveal consistent performance gaps across tasks, highlighting persistent challenges in integrated scientific reasoning and critique. Our benchmark and dataset are available at https:// github.com/Yanjun-Zhao/PaperMind.