# ProCIS: A Benchmark for Proactive Retrieval in Conversations

arXiv: https://arxiv.org/abs/2405.06460 · HTML: https://mlsanity.com/paper/2405.06460/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2024.

## Built on by (most cited first, 11 of 11)

- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues](https://mlsanity.com/paper/2602.01313/) (2026)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents](https://mlsanity.com/paper/2510.19771/) (2025)
- [mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs](https://mlsanity.com/paper/2505.11180/) (2025)
- [ProactBench: Beyond What The User Asked For](https://mlsanity.com/paper/2605.09228/) (2026)
- [Knowing Isn't Understanding: Re-grounding Generative Proactivity with Epistemic and Behavioral Insight](https://mlsanity.com/paper/2602.15259/) (2026)
- [JIR-Arena: The First Benchmark Dataset for Just-in-time Information Recommendation](https://mlsanity.com/paper/2505.13550/) (2025)
- [Image-Seeking Intent Prediction for Cross-Device Product Search](https://mlsanity.com/paper/2511.14764/) (2025)
- [ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation](https://mlsanity.com/paper/2609.11101/) (2026)
- [CompCQR: Compositional Query Generation for Training-Free Conversational Search](https://mlsanity.com/paper/2609.14646/) (2026)

## Abstract (arXiv)

The field of conversational information seeking, which is rapidly gaining interest in both academia and industry, is changing how we interact with search engines through natural language interactions. Existing datasets and methods are mostly evaluating reactive conversational information seeking systems that solely provide response to every query from the user. We identify a gap in building and evaluating proactive conversational information seeking systems that can monitor a multi-party human conversation and proactively engage in the conversation at an opportune moment by retrieving useful resources and suggestions. In this paper, we introduce a large-scale dataset for proactive document retrieval that consists of over 2.8 million conversations. We conduct crowdsourcing experiments to obtain high-quality and relatively complete relevance judgments through depth-k pooling. We also collect annotations related to the parts of the conversation that are related to each document, enabling us to evaluate proactive retrieval systems. We introduce normalized proactive discounted cumulative gain (npDCG) for evaluating these systems, and further provide benchmark results for a wide range of models, including a novel model we developed for this task. We believe that the developed dataset, called ProCIS, paves the path towards developing proactive conversational information seeking systems.