mlsanityData to 14 Oct 2026

Building Open-Retrieval Conversational Question Answering Systems by Generating Synthetic Data and Decontextualizing User Questions

arXiv 2507.04884 ↗First posted 7 Jul 2025cs.CLcited by 0

On the map: Synthetic data and self-improvement for LLMs → Synthetic reasoning data and bootstrapping

Abstract (arXiv)

We consider open-retrieval conversational question answering (OR-CONVQA), an extension of question answering where system responses need to be (i) aware of dialog history and (ii) grounded in documents (or document fragments) retrieved per question. Domain-specific OR-CONVQA training datasets are crucial for real-world applications, but hard to obtain. We propose a pipeline that capitalizes on the abundance of plain text documents in organizations (e.g., product documentation) to automatically produce realistic OR-CONVQA dialogs with annotations. Similarly to real-world humanannotated OR-CONVQA datasets, we generate in-dialog question-answer pairs, self-contained (decontextualized, e.g., no referring expressions) versions of user questions, and propositions (sentences expressing prominent information from the documents) the system responses are grounded in. We show how the synthetic dialogs can be used to train efficient question rewriters that decontextualize user questions, allowing existing dialog-unaware retrievers to be utilized. The retrieved information and the decontextualized question are then passed on to an LLM that generates the system's response.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 48 of its 63 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

48 of 63 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  12. 2018CoQA: A Conversational Question Answering Challengecited by 683
36 more
  1. 2018QuAC : Question Answering in Contextcited by 381
  2. 2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 148
  3. 2020Open-Retrieval Conversational Question Answeringcited by 99
  4. 2020Question Rewriting for Conversational Question Answeringcited by 93
  5. 2020doc2dial: A Goal-Oriented Document-Grounded Dialogue Datasetcited by 88
  6. 2020Few-Shot Generative Conversational Query Rewritingcited by 85
  7. 2023Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Searchcited by 75
  8. 2021Few-Shot Conversational Dense Retrievalcited by 72
  9. 2023Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewritingcited by 69
  10. 2021CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learningcited by 68
  11. 2019BERT with History Answer Embedding for Conversational Question Answeringcited by 67
  12. 2023ConvGQR: Generative Query Reformulation for Conversational Searchcited by 59
  13. 2022Dialog Inpainting: Turning Documents into Dialogscited by 56
  14. 2021MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documentscited by 53
  15. 2020Query Resolution for Conversational Search with Limited Supervisioncited by 52
  16. 2020Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Modelscited by 45
  17. 2020DoQA -- Accessing Domain-Specific FAQs via Conversational QAcited by 42
  18. 2018FlowQA: Grasping Flow in History for Conversational Machine Comprehensioncited by 33
  19. 2018SDNet: Contextualized Attention-based Deep Network for Conversational Question Answeringcited by 30
  20. 2023Learning to Relate to Previous Turns in Conversational Searchcited by 26
  21. 2024Ask Optimal Questions: Aligning Large Language Models with Retriever's Preference in Conversationcited by 23
  22. 2021Towards Quantifiable Dialogue Coherence Evaluationcited by 21
  23. 2023Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representationscited by 20
  24. 2019FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehensioncited by 19
  25. 2022Generating Information-Seeking Conversations from Unlabeled Documentscited by 17
  26. 2024Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answerscited by 15
  27. 2024Interpreting Conversational Dense Retrieval by Rewriting-Enhanced Inversion of Session Embeddingcited by 13
  28. 2020Conversations with Search Engines: SERP-based Conversational Response Generationcited by 11
  29. 2022Zero-shot Query Contextualization for Conversational Searchcited by 9
  30. 2022QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevancecited by 9
  31. 2023Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sourcescited by 9
  32. 2024ConvSDG: Session Data Generation for Conversational Searchcited by 8
  33. 2023q2d: Turning Questions into Dialogs to Teach Models How to Searchcited by 5
  34. 2024Domain Adaptation for Dense Retrieval and Conversational Dense Retrieval through Self-Supervision by Meticulous Pseudo-Relevance Labelingcited by 5
  35. 2021Weakly-Supervised Open-Retrieval Conversational Question Answeringcited by 4
  36. 2024Synthesizing Conversations from Unlabeled Documents using Automatic Response Segmentationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Open-Domain Conversational Question Answering with Historical Answers
  2. 2025A Survey of the State-of-the-Art in Conversational Question Answering Systems
  3. 2024Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Action
  4. 2023CarExpert: Leveraging Large Language Models for In-Car Conversational Question Answering
  5. 2026Inferential Question Answering
  6. 2021Efficient Retrieval Augmented Generation from Unstructured Knowledge for Task-Oriented Dialog
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
  8. 2026RAC: Retrieval-Augmented Clarification for Faithful Conversational Search