# Keyword Extraction for Improved Document Retrieval in Conversational Search

arXiv: https://arxiv.org/abs/2109.05979 · HTML: https://mlsanity.com/paper/2109.05979/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [A Deep Relevance Matching Model for Ad-hoc Retrieval](https://mlsanity.com/paper/1711.08611/) (2017)
- [SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications](https://mlsanity.com/paper/1704.02853/) (2017)
- [Learning to Match Using Local and Distributed Representations of Text for Web Search](https://mlsanity.com/paper/1610.08136/) (2016)
- [ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)](https://mlsanity.com/paper/2009.11352/) (2020)
- [Query Resolution for Conversational Search with Limited Supervision](https://mlsanity.com/paper/2005.11723/) (2020)
- [Identifying Well-formed Natural Language Questions](https://mlsanity.com/paper/1808.09419/) (2018)
- [Analyzing and Learning from User Interactions for Search Clarification](https://mlsanity.com/paper/2006.00166/) (2020)
- [Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Search](https://mlsanity.com/paper/2006.07548/) (2020)
- [Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Search](https://mlsanity.com/paper/2008.03717/) (2020)
- [User Engagement Prediction for Clarification in Search](https://mlsanity.com/paper/2102.04163/) (2021)
- [Context-Aware Target Apps Selection and Recommendation for Enhancing Personal Mobile Assistants](https://mlsanity.com/paper/2101.03394/) (2021)
- [Harnessing Evolution of Multi-Turn Conversations for Effective Answer Retrieval](https://mlsanity.com/paper/1912.10554/) (2019)
- [Ranking Clarifying Questions Based on Predicted User Engagement](https://mlsanity.com/paper/2103.06192/) (2021)
- [Target Apps Selection: Towards a Unified Search Framework for Mobile Devices](https://mlsanity.com/paper/1805.02211/) (2018)

## Built on by (most cited first, 1 of 1)

- [Keyword Extraction in Scientific Documents](https://mlsanity.com/paper/2207.01888/) (2022)

## Abstract (arXiv)

Recent research has shown that mixed-initiative conversational search, based on the interaction between users and computers to clarify and improve a query, provides enormous advantages. Nonetheless, incorporating additional information provided by the user from the conversation poses some challenges. In fact, further interactions could confuse the system as a user might use words irrelevant to the information need but crucial for correct sentence construction in the context of multi-turn conversations. To this aim, in this paper, we have collected two conversational keyword extraction datasets and propose an end-to-end document retrieval pipeline incorporating them. Furthermore, we study the performance of two neural keyword extraction models, namely, BERT and sequence to sequence, in terms of extraction accuracy and human annotation. Finally, we study the effect of keyword extraction on the end-to-end neural IR performance and show that our approach beats state-of-the-art IR models. We make the two datasets publicly available to foster research in this area.