# Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations

arXiv: https://arxiv.org/abs/2408.15232 · HTML: https://mlsanity.com/paper/2408.15232/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 27 Aug 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 31)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration](https://mlsanity.com/paper/2310.02170/) (2023)
- [Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration](https://mlsanity.com/paper/2307.05300/) (2023)
- [Open Question Answering over Tables and Text](https://mlsanity.com/paper/2010.10439/) (2020)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://mlsanity.com/paper/2402.14207/) (2024)
- [A Critical Evaluation of Evaluations for Long-form Question Answering](https://mlsanity.com/paper/2305.18201/) (2023)
- [HybriDialogue: An Information-Seeking Dialogue Dataset Grounded on Tabular and Textual Data](https://mlsanity.com/paper/2204.13243/) (2022)
- [TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervision](https://mlsanity.com/paper/2403.00165/) (2024)
- [Beyond Summarization: Designing AI Support for Real-World Expository Writing Tasks](https://mlsanity.com/paper/2304.02623/) (2023)
- [User Intent Prediction in Information-seeking Conversations](https://mlsanity.com/paper/1901.03489/) (2019)
- [Interactive Machine Comprehension with Information Seeking Agents](https://mlsanity.com/paper/1908.10449/) (2019)
- [Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts](https://mlsanity.com/paper/2212.06002/) (2022)
- [KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions](https://mlsanity.com/paper/2403.03866/) (2024)
- [Open-Domain Question-Answering for COVID-19 and Other Emergent Domains](https://mlsanity.com/paper/2110.06962/) (2021)

## Built on by (most cited first, 30 of 43)

- [MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents](https://mlsanity.com/paper/2506.15841/) (2025)
- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges](https://mlsanity.com/paper/2508.05668/) (2025)
- [Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforce](https://mlsanity.com/paper/2506.06576/) (2025)
- [LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey](https://mlsanity.com/paper/2505.00753/) (2025)
- [Synergizing RAG and Reasoning: A Systematic Review](https://mlsanity.com/paper/2504.15909/) (2025)
- [RAPID: Efficient Retrieval-Augmented Long Text Generation with Writing Planning and Information Discovery](https://mlsanity.com/paper/2503.00751/) (2025)
- [OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking](https://mlsanity.com/paper/2501.09751/) (2025)
- [Beyond Outlining: Heterogeneous Recursive Planning for Adaptive Long-form Writing with Language Models](https://mlsanity.com/paper/2503.08275/) (2025)
- [TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications](https://mlsanity.com/paper/2510.18586/) (2025)
- [SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation](https://mlsanity.com/paper/2506.12689/) (2025)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notes](https://mlsanity.com/paper/2509.00877/) (2025)
- [XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaboration](https://mlsanity.com/paper/2505.11336/) (2025)
- [DREAM: Deep Research Evaluation with Agentic Metrics](https://mlsanity.com/paper/2602.18940/) (2026)
- [DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytelling](https://mlsanity.com/paper/2604.06474/) (2026)
- [WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generation](https://mlsanity.com/paper/2503.19065/) (2025)
- [Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings](https://mlsanity.com/paper/2506.00178/) (2025)
- [Interaction as Intelligence: Deep Research With Human-AI Partnership](https://mlsanity.com/paper/2507.15759/) (2025)
- [Perspectra: Choosing Your Experts Enhances Critical Thinking in Multi-Agent Research Ideation](https://mlsanity.com/paper/2509.20553/) (2025)
- [Grounding Agent Memory in Contextual Intent](https://mlsanity.com/paper/2601.10702/) (2026)
- [Deep-Research Agents Can Be Poisoned via User-Generated Content](https://mlsanity.com/paper/2605.24245/) (2026)
- [Test-time Corpus Feedback: From Retrieval to RAG](https://mlsanity.com/paper/2508.15437/) (2025)
- [HetaRAG: Hybrid Deep Retrieval-Augmented Generation across Heterogeneous Data Stores](https://mlsanity.com/paper/2509.21336/) (2025)
- [Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges](https://mlsanity.com/paper/2607.26212/) (2026)
- [LLM×MapReduce-V2: Entropy-Driven Convolutional Test-Time Scaling for Generating Long-Form Articles from Extremely Long Resources](https://mlsanity.com/paper/2504.05732/) (2025)
- [PROPER Agents: Proactivity Driven Personalized Agents for Advancing Knowledge Gap Navigation](https://mlsanity.com/paper/2601.09926/) (2026)
- [Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users](https://mlsanity.com/paper/2603.16120/) (2026)
- [Self-Evolving Deep Research via Joint Generation and Evaluation](https://mlsanity.com/paper/2606.04507/) (2026)
- [DeFine: A Decomposed and Fine-Grained Annotated Dataset for Long-form Article Generation](https://mlsanity.com/paper/2503.07170/) (2025)

## Abstract (arXiv)

While language model (LM)-powered chatbots and generative search engines excel at answering concrete queries, discovering information in the terrain of unknown unknowns remains challenging for users. To emulate the common educational scenario where children/students learn by listening to and participating in conversations of their parents/teachers, we create Collaborative STORM (Co-STORM). Unlike QA systems that require users to ask all the questions, Co-STORM lets users observe and occasionally steer the discourse among several LM agents. The agents ask questions on the user's behalf, allowing the user to discover unknown unknowns serendipitously. To facilitate user interaction, Co-STORM assists users in tracking the discourse by organizing the uncovered information into a dynamic mind map, ultimately generating a comprehensive report as takeaways. For automatic evaluation, we construct the WildSeek dataset by collecting real information-seeking records with user goals. Co-STORM outperforms baseline methods on both discourse trace and report quality. In a further human evaluation, 70% of participants prefer Co-STORM over a search engine, and 78% favor it over a RAG chatbot.