# Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models

arXiv: https://arxiv.org/abs/2310.14696 · HTML: https://mlsanity.com/paper/2310.14696/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection](https://mlsanity.com/paper/1911.04118/) (2019)
- [Joint Passage Ranking for Diverse Multi-Answer Retrieval](https://mlsanity.com/paper/2104.08445/) (2021)
- [Asking Clarification Questions to Handle Ambiguity in Open-Domain QA](https://mlsanity.com/paper/2305.13808/) (2023)
- [Answering Ambiguous Questions through Generative Evidence Fusion and Round-Trip Prediction](https://mlsanity.com/paper/2011.13137/) (2020)
- [Query Refinement Prompts for Closed-Book Long-Form Question Answering](https://mlsanity.com/paper/2210.17525/) (2022)
- [Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework](https://mlsanity.com/paper/2110.08544/) (2021)

## Built on by (most cited first, 30 of 43)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Deep Research Agents: A Systematic Examination And Roadmap](https://mlsanity.com/paper/2506.18096/) (2025)
- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [Searching for Best Practices in Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.01219/) (2024)
- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely](https://mlsanity.com/paper/2409.14924/) (2024)
- [Towards Lifelong Learning of Large Language Models: A Survey](https://mlsanity.com/paper/2406.06391/) (2024)
- [Demystifying Chains, Trees, and Graphs of Thoughts](https://mlsanity.com/paper/2401.14295/) (2024)
- [Test-Time Learning for Large Language Models](https://mlsanity.com/paper/2505.20633/) (2025)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [AmbigDocs: Reasoning across Documents on Different Entities under the Same Name](https://mlsanity.com/paper/2404.12447/) (2024)
- [Enhancing Retrieval and Managing Retrieval: A Four-Module Synergy for Improved Quality and Efficiency in RAG Systems](https://mlsanity.com/paper/2407.10670/) (2024)
- [IC-Cache: Efficient Large Language Model Serving via In-context Caching](https://mlsanity.com/paper/2501.12689/) (2025)
- [APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation](https://mlsanity.com/paper/2407.21048/) (2024)
- [Federated In-Context LLM Agent Learning](https://mlsanity.com/paper/2412.08054/) (2024)
- [A Survey of Scaling in Large Language Model Reasoning](https://mlsanity.com/paper/2504.02181/) (2025)
- [Self-Routing RAG: Binding Selective Retrieval with Knowledge Verbalization](https://mlsanity.com/paper/2504.01018/) (2025)
- [Acknowledging Focus Ambiguity in Visual Questions](https://mlsanity.com/paper/2501.02201/) (2025)
- [DiscoverLLM: From Executing Intents to Discovering Them](https://mlsanity.com/paper/2602.03429/) (2026)
- [A Learnable Agent Collaboration Network Framework for Personalized Multimodal AI Search Engine](https://mlsanity.com/paper/2409.00636/) (2024)
- [Learning Steerable Clarification Policies with Collaborative Self-play](https://mlsanity.com/paper/2512.04068/) (2025)
- [Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities](https://mlsanity.com/paper/2505.23838/) (2025)
- [REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing](https://mlsanity.com/paper/2506.16444/) (2025)
- [Think-then-Act: A Dual-Angle Evaluated Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.13050/) (2024)

## Abstract (arXiv)

Questions in open-domain question answering are often ambiguous, allowing multiple interpretations. One approach to handling them is to identify all possible interpretations of the ambiguous question (AQ) and to generate a long-form answer addressing them all, as suggested by Stelmakh et al., (2022). While it provides a comprehensive response without bothering the user for clarification, considering multiple dimensions of ambiguity and gathering corresponding knowledge remains a challenge. To cope with the challenge, we propose a novel framework, Tree of Clarifications (ToC): It recursively constructs a tree of disambiguations for the AQ -- via few-shot prompting leveraging external knowledge -- and uses it to generate a long-form answer. ToC outperforms existing baselines on ASQA in a few-shot setup across the metrics, while surpassing fully-supervised baselines trained on the whole training set in terms of Disambig-F1 and Disambig-ROUGE. Code is available at https://github.com/gankim/tree-of-clarifications.