mlsanityData to 7 Oct 2026

Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents

arXiv 2402.17896 ↗First posted 27 Feb 2024cs.CL cs.AIcited by 22

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Existing question answering (QA) datasets are no longer challenging to most powerful Large Language Models (LLMs). Traditional QA benchmarks like TriviaQA, NaturalQuestions, ELI5 and HotpotQA mainly study ``known unknowns'' with clear indications of both what information is missing, and how to find it to answer the question. Hence, good performance on these benchmarks provides a false sense of security. A yet unmet need of the NLP community is a bank of non-factoid, multi-perspective questions involving a great deal of unclear information needs, i.e. ``unknown uknowns''. We claim we can find such questions in search engine logs, which is surprising because most question-intent queries are indeed factoid. We present Researchy Questions, a dataset of search engine queries tediously filtered to be non-factoid, ``decompositional'' and multi-perspective. We show that users spend a lot of ``effort'' on these questions in terms of signals like clicks and session length, and that they are also challenging for GPT-4. We also show that ``slow thinking'' answering techniques, like decomposition into sub-questions shows benefit over answering directly. We release ~ 100k Researchy Questions, along with the Clueweb22 URLs that were clicked.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 59 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

36 of 59 references.

References · 36

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
24 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  5. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  6. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  7. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  8. 2024Self-Rewarding Language Modelscited by 654
  9. 2019ELI5: Long Form Question Answeringcited by 567
  10. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  11. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  12. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  13. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  14. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  15. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  16. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  17. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  18. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  19. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  20. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  21. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54
  22. 2020AQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarizationcited by 42
  23. 2023End-to-End Beam Retrieval for Multi-Hop Question Answeringcited by 35
  24. 2022Query Refinement Prompts for Closed-Book Long-Form Question Answeringcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 9; 2026: 8.

22 papers.

Sort:

By area:

  1. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  2. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 58
  3. 2025WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Researchcited by 44
  4. 2024Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Trackcited by 31
  5. 2024Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Frameworkcited by 28
  6. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  7. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judgescited by 14
  8. 2024ResearchArena: Benchmarking Large Language Models' Ability to Collect and Organize Information as Research Agentscited by 11
  9. 2025An Adversary-Resistant Multi-Agent LLM System via Credibility Scoringcited by 10
  10. 2024Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coveragecited by 8
  11. 2025APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Trainingcited by 6
  12. 2025Query Decomposition for RAG: Balancing Exploration-Exploitationcited by 5
10 more
  1. 2025A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesiscited by 2
  2. 2026Incorporating Q&A Nuggets into Retrieval-Augmented Generationcited by 2
  3. 2026AgentDisCo: Towards Disentanglement and Collaboration in Open-ended Deep Research Agentscited by 2
  4. 2026DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluationcited by 1
  5. 2026Benchmarking LLM Serving Systems for Agentic AI Workloads with XPerfcited by 1
  6. 2025Document Quality Scoring for Web Crawlingcited by 0
  7. 2026Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Oncecited by 0
  8. 2026An Interactive Paradigm for Deep Researchcited by 0
  9. 2026Relational Priors as Convergence Pressure in LLM-Based Multi-Agent Systemscited by 0
  10. 2026Not Worth Another Token: Marginal Value Estimation for Efficient Deep Research Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Large Language Model Agents for Question Answering
  2. 2026LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
  3. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models
  4. 2025An Analysis of Decoding Methods for LLM-based Agents for Faithful Multi-Hop Question Answering
  5. 2025Synthesizing Agentic Data for Web Agents with Progressive Difficulty Enhancement Mechanisms
  6. 2026A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models
  7. 2024AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs
  8. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text