# Pre-Training for Query Rewriting in A Spoken Language Understanding System

arXiv: https://arxiv.org/abs/2002.05607 · HTML: https://mlsanity.com/paper/2002.05607/

## Key facts

- Cited by 6 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 13 Feb 2020.

## Built on by (most cited first, 6 of 6)

- [Correcting Automated and Manual Speech Transcription Errors using Warped Language Models](https://mlsanity.com/paper/2103.14580/) (2021)
- [Graph Meets LLM: A Novel Approach to Collaborative Filtering for Robust Conversational Understanding](https://mlsanity.com/paper/2305.14449/) (2023)
- [User-Initiated Repetition-Based Recovery in Multi-Utterance Dialogue Systems](https://mlsanity.com/paper/2108.01208/) (2021)
- [PENTATRON: PErsonalized coNText-Aware Transformer for Retrieval-based cOnversational uNderstanding](https://mlsanity.com/paper/2210.12308/) (2022)
- [KG-ECO: Knowledge Graph Enhanced Entity Correction for Query Rewriting](https://mlsanity.com/paper/2302.10454/) (2023)
- [Few-Shot Adaptation for Parsing Contextual Utterances with LLMs](https://mlsanity.com/paper/2309.10168/) (2023)

## Abstract (arXiv)

Query rewriting (QR) is an increasingly important technique to reduce customer friction caused by errors in a spoken language understanding pipeline, where the errors originate from various sources such as speech recognition errors, language understanding errors or entity resolution errors. In this work, we first propose a neural-retrieval based approach for query rewriting. Then, inspired by the wide success of pre-trained contextual language embeddings, and also as a way to compensate for insufficient QR training data, we propose a language-modeling (LM) based approach to pre-train query embeddings on historical user conversation data with a voice assistant. In addition, we propose to use the NLU hypotheses generated by the language understanding system to augment the pre-training. Our experiments show pre-training provides rich prior information and help the QR task achieve strong performance. We also show joint pre-training with NLU hypotheses has further benefit. Finally, after pre-training, we find a small set of rewrite pairs is enough to fine-tune the QR model to outperform a strong baseline by full training on all QR training data.