# ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

arXiv: https://arxiv.org/abs/2610.02202 · HTML: https://mlsanity.com/paper/2610.02202/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 71 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 30 of 30)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts](https://mlsanity.com/paper/2205.01833/) (2022)
- [The Semantic Scholar Open Data Platform](https://mlsanity.com/paper/2301.10140/) (2023)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs](https://mlsanity.com/paper/2411.14199/) (2024)
- [Deep Research: A Survey of Autonomous Research Agents](https://mlsanity.com/paper/2508.12752/) (2025)
- [Tongyi DeepResearch Technical Report](https://mlsanity.com/paper/2510.24701/) (2025)
- [Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction](https://mlsanity.com/paper/2605.05242/) (2026)
- [DIVER: A Multi-Stage Approach for Reasoning-intensive Information Retrieval](https://mlsanity.com/paper/2508.07995/) (2025)
- [RAR-b: Reasoning as Retrieval Benchmark](https://mlsanity.com/paper/2404.06347/) (2024)
- [SciPIP: An LLM-based Scientific Paper Idea Proposer](https://mlsanity.com/paper/2410.23166/) (2024)
- [Is Grep All You Need? How Agent Harnesses Reshape Agentic Search](https://mlsanity.com/paper/2605.15184/) (2026)
- [CSFCube -- A Test Collection of Computer Science Research Articles for Faceted Query by Example](https://mlsanity.com/paper/2103.12906/) (2021)
- [ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval](https://mlsanity.com/paper/2510.08252/) (2025)
- [Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini](https://mlsanity.com/paper/2605.27295/) (2026)
- [GIANTS: Generative Insight Anticipation from Scientific Literature](https://mlsanity.com/paper/2604.09793/) (2026)
- [OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries](https://mlsanity.com/paper/2605.06235/) (2026)
- [Harnessing Large Language Models for Scientific Novelty Detection](https://mlsanity.com/paper/2505.24615/) (2025)
- [DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search](https://mlsanity.com/paper/2607.27178/) (2026)
- [Chain of Retrieval: Multi-Aspect Iterative Search Expansion and Post-Order Search Aggregation for Full Paper Retrieval](https://mlsanity.com/paper/2507.10057/) (2025)
- [PaperScout: An Autonomous Agent for Academic Paper Search with Process-Aware Sequence-Level Policy Optimization](https://mlsanity.com/paper/2601.10029/) (2026)
- [HTML papers on arXiv -- why it is important, and how we made it happen](https://mlsanity.com/paper/2402.08954/) (2024)
- [SciNet: Evaluating AI Agents in Relation-Aware Scientific Literature Retrieval](https://mlsanity.com/paper/2601.03260/) (2025)

## Abstract (arXiv)

What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.