# Harnessing Evolution of Multi-Turn Conversations for Effective Answer Retrieval

arXiv: https://arxiv.org/abs/1912.10554 · HTML: https://mlsanity.com/paper/1912.10554/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2019.

## Built on by (most cited first, 10 of 10)

- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [ChatGPT as your Personal Data Scientist](https://mlsanity.com/paper/2305.13657/) (2023)
- [Query Performance Prediction: From Ad-hoc to Conversational Search](https://mlsanity.com/paper/2305.10923/) (2023)
- [Context-Aware Target Apps Selection and Recommendation for Enhancing Personal Mobile Assistants](https://mlsanity.com/paper/2101.03394/) (2021)
- [Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA](https://mlsanity.com/paper/2409.15515/) (2024)
- [Engineering Conversational Search Systems: A Review of Applications, Architectures, and Functional Components](https://mlsanity.com/paper/2407.00997/) (2024)
- [Keyword Extraction for Improved Document Retrieval in Conversational Search](https://mlsanity.com/paper/2109.05979/) (2021)
- [Proactive Guidance of Multi-Turn Conversation in Industrial Search](https://mlsanity.com/paper/2505.24251/) (2025)
- [MICROS: Mixed-Initiative ConveRsatiOnal Systems Workshop](https://mlsanity.com/paper/2101.10219/) (2021)
- [Rewriting Conversational Utterances with Instructed Large Language Models](https://mlsanity.com/paper/2410.07797/) (2024)

## Abstract (arXiv)

With the improvements in speech recognition and voice generation technologies over the last years, a lot of companies have sought to develop conversation understanding systems that run on mobile phones or smart home devices through natural language interfaces. Conversational assistants, such as Google Assistant and Microsoft Cortana, can help users to complete various types of tasks. This requires an accurate understanding of the user's information need as the conversation evolves into multiple turns. Finding relevant context in a conversation's history is challenging because of the complexity of natural language and the evolution of a user's information need. In this work, we present an extensive analysis of language, relevance, dependency of user utterances in a multi-turn information-seeking conversation. To this aim, we have annotated relevant utterances in the conversations released by the TREC CaST 2019 track. The annotation labels determine which of the previous utterances in a conversation can be used to improve the current one. Furthermore, we propose a neural utterance relevance model based on BERT fine-tuning, outperforming competitive baselines. We study and compare the performance of multiple retrieval models, utilizing different strategies to incorporate the user's context. The experimental results on both classification and retrieval tasks show that our proposed approach can effectively identify and incorporate the conversation context. We show that processing the current utterance using the predicted relevant utterance leads to a 38% relative improvement in terms of nDCG@20. Finally, to foster research in this area, we have released the dataset of the annotations.