# Building Open-Retrieval Conversational Question Answering Systems by Generating Synthetic Data and Decontextualizing User Questions

arXiv: https://arxiv.org/abs/2507.04884 · HTML: https://mlsanity.com/paper/2507.04884/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 48 of its 63 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Synthetic reasoning data and bootstrapping](https://mlsanity.com/synth/synthetic-reasoning-data-and-bootstrapping/)

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [Open-Domain Question Answering Goes Conversational via Question Rewriting](https://mlsanity.com/paper/2010.04898/) (2020)
- [Open-Retrieval Conversational Question Answering](https://mlsanity.com/paper/2005.11364/) (2020)
- [Question Rewriting for Conversational Question Answering](https://mlsanity.com/paper/2004.14652/) (2020)
- [doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset](https://mlsanity.com/paper/2011.06623/) (2020)
- [Few-Shot Generative Conversational Query Rewriting](https://mlsanity.com/paper/2006.05009/) (2020)
- [Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search](https://mlsanity.com/paper/2303.06573/) (2023)
- [Few-Shot Conversational Dense Retrieval](https://mlsanity.com/paper/2105.04166/) (2021)
- [Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting](https://mlsanity.com/paper/2310.09716/) (2023)
- [CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning](https://mlsanity.com/paper/2112.08558/) (2021)
- [BERT with History Answer Embedding for Conversational Question Answering](https://mlsanity.com/paper/1905.05412/) (2019)
- [ConvGQR: Generative Query Reformulation for Conversational Search](https://mlsanity.com/paper/2305.15645/) (2023)
- [Dialog Inpainting: Turning Documents into Dialogs](https://mlsanity.com/paper/2205.09073/) (2022)
- [MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents](https://mlsanity.com/paper/2109.12595/) (2021)
- [Query Resolution for Conversational Search with Limited Supervision](https://mlsanity.com/paper/2005.11723/) (2020)
- [Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Models](https://mlsanity.com/paper/2004.01909/) (2020)
- [DoQA -- Accessing Domain-Specific FAQs via Conversational QA](https://mlsanity.com/paper/2005.01328/) (2020)
- [FlowQA: Grasping Flow in History for Conversational Machine Comprehension](https://mlsanity.com/paper/1810.06683/) (2018)

## Abstract (arXiv)

We consider open-retrieval conversational question answering (OR-CONVQA), an extension of question answering where system responses need to be (i) aware of dialog history and (ii) grounded in documents (or document fragments) retrieved per question. Domain-specific OR-CONVQA training datasets are crucial for real-world applications, but hard to obtain. We propose a pipeline that capitalizes on the abundance of plain text documents in organizations (e.g., product documentation) to automatically produce realistic OR-CONVQA dialogs with annotations. Similarly to real-world humanannotated OR-CONVQA datasets, we generate in-dialog question-answer pairs, self-contained (decontextualized, e.g., no referring expressions) versions of user questions, and propositions (sentences expressing prominent information from the documents) the system responses are grounded in. We show how the synthetic dialogs can be used to train efficient question rewriters that decontextualize user questions, allowing existing dialog-unaware retrievers to be utilized. The retrieved information and the decontextualized question are then passed on to an LLM that generates the system's response.