# PaperScout: An Autonomous Agent for Academic Paper Search with Process-Aware Sequence-Level Policy Optimization

arXiv: https://arxiv.org/abs/2601.10029 · HTML: https://mlsanity.com/paper/2601.10029/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 15 Jan 2026.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2504.20073/) (2025)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://mlsanity.com/paper/2504.05118/) (2025)
- [Agent Lightning: Train ANY AI Agents with Reinforcement Learning](https://mlsanity.com/paper/2508.03680/) (2025)
- [A Deep Look into Neural Ranking Models for Information Retrieval](https://mlsanity.com/paper/1903.06902/) (2019)
- [PaSa: An LLM Agent for Comprehensive Academic Paper Search](https://mlsanity.com/paper/2501.10120/) (2025)
- [Time Series Forecasting via Reasoning: A Slow-Thinking Approach with Reinforcement Fine-Tuned LLMs](https://mlsanity.com/paper/2506.10630/) (2025)
- [Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning](https://mlsanity.com/paper/2511.14460/) (2025)
- [TableMind: An Autonomous Programmatic Agent for Tool-Augmented Table Reasoning](https://mlsanity.com/paper/2509.06278/) (2025)
- [PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature](https://mlsanity.com/paper/2510.10909/) (2025)
- [Scientific Paper Retrieval with LLM-Guided Semantic-Based Ranking](https://mlsanity.com/paper/2505.21815/) (2025)
- [Token-level Proximal Policy Optimization for Query Generation](https://mlsanity.com/paper/2411.00722/) (2024)
- [SPAR: Scholar Paper Retrieval with LLM-based Agents for Enhanced Academic Search](https://mlsanity.com/paper/2507.15245/) (2025)
- [Chain of Retrieval: Multi-Aspect Iterative Search Expansion and Post-Order Search Aggregation for Full Paper Retrieval](https://mlsanity.com/paper/2507.10057/) (2025)
- [Towards Context-aware Reasoning-enhanced Generative Searching in E-commerce](https://mlsanity.com/paper/2510.16925/) (2025)

## Built on by (most cited first, 5 of 5)

- [Cast-R1: Learning Tool-Augmented Sequential Decision Policies for Time Series Forecasting](https://mlsanity.com/paper/2602.13802/) (2026)
- [CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities](https://mlsanity.com/paper/2604.18401/) (2026)
- [ScholarSum: Student-Teacher Abstractive Summarization via Knowledge Graph Reasoning and Reflective Refinement](https://mlsanity.com/paper/2606.18850/) (2026)
- [ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments](https://mlsanity.com/paper/2606.20235/) (2026)
- [ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research](https://mlsanity.com/paper/2610.02202/) (2026)

## Abstract (arXiv)

Academic paper search is a fundamental task in scientific research, yet most existing approaches organize retrieval around predefined workflows or structured interaction protocols that struggle with complex, conditional queries. To address this limitation, we propose PaperScout, an autonomous agent that reformulates paper search as a sequential decision-making process. Unlike static workflows, PaperScout dynamically decides whether, when, and how to invoke search and expand tools based on accumulated retrieval context. However, training such agents presents a fundamental challenge: standard reinforcement learning methods, typically designed for single-turn tasks, suffer from a granularity mismatch when applied to multi-turn agentic tasks, where token-level optimization diverges from the granularity of sequence-level interactions, leading to noisy credit assignment and unstable training dynamics. We introduce Proximal Sequence Policy Optimization (PSPO), a process-aware, sequence-level policy optimization method that aligns optimization with agent--environment interaction. Comprehensive experiments on both synthetic and real-world benchmarks demonstrate that PaperScout significantly outperforms strong structured retrieval and RL baselines in both recall and relevance, validating the effectiveness of our adaptive agentic framework and optimization strategy.