# Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunities

arXiv: https://arxiv.org/abs/2304.13149 · HTML: https://mlsanity.com/paper/2304.13149/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling](https://mlsanity.com/paper/1906.07241/) (2019)
- [Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entities](https://mlsanity.com/paper/1912.09251/) (2019)
- [RescoreBERT: Discriminative Speech Recognition Rescoring with BERT](https://mlsanity.com/paper/2202.01094/) (2022)
- [Personalization Strategies for End-to-End Speech Recognition Systems](https://mlsanity.com/paper/2102.07739/) (2021)
- [An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models](https://mlsanity.com/paper/1909.06678/) (2019)
- [Innovative Bert-based Reranking Language Models for Speech Recognition](https://mlsanity.com/paper/2104.04950/) (2021)
- [Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2010.13878/) (2020)
- [Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devices](https://mlsanity.com/paper/2207.08988/) (2022)
- [Latent Relation Language Models](https://mlsanity.com/paper/1908.07690/) (2019)
- [Scalable Multi Corpora Neural Language Models for ASR](https://mlsanity.com/paper/1907.01677/) (2019)
- [Scalable language model adaptation for spoken dialogue systems](https://mlsanity.com/paper/1812.04647/) (2018)
- [An Empirical Study of Efficient ASR Rescoring with Transformers](https://mlsanity.com/paper/1910.11450/) (2019)
- [Connecting and Comparing Language Model Interpolation Techniques](https://mlsanity.com/paper/1908.09738/) (2019)
- [Predicting Entity Popularity to Improve Spoken Entity Recognition by Virtual Assistants](https://mlsanity.com/paper/2005.12816/) (2020)
- [Error-driven Pruning of Language Models for Virtual Assistants](https://mlsanity.com/paper/2102.07219/) (2021)
- [Language model fusion for streaming end to end speech recognition](https://mlsanity.com/paper/2104.04487/) (2021)
- [A Discriminative Entity-Aware Language Model for Virtual Assistants](https://mlsanity.com/paper/2106.11292/) (2021)
- [Space-Efficient Representation of Entity-centric Query Language Models](https://mlsanity.com/paper/2206.14885/) (2022)
- [On Modeling ASR Word Confidence](https://mlsanity.com/paper/1907.09636/) (2019)
- [UserLibri: A Dataset for ASR Personalization Using Only Text](https://mlsanity.com/paper/2207.00706/) (2022)
- [ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks](https://mlsanity.com/paper/2207.01893/) (2022)

## Built on by (most cited first, 4 of 4)

- [Transformer-based Model for ASR N-Best Rescoring and Rewriting](https://mlsanity.com/paper/2406.08207/) (2024)
- [FASA: a Flexible and Automatic Speech Aligner for Extracting High-quality Aligned Children Speech Data](https://mlsanity.com/paper/2406.17926/) (2024)
- [SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise](https://mlsanity.com/paper/2602.12783/) (2026)
- [Phonetically-Augmented Discriminative Rescoring for Voice Search Error Correction](https://mlsanity.com/paper/2506.06117/) (2025)

## Abstract (arXiv)

Virtual assistants are becoming increasingly important speech-driven Information Retrieval platforms that assist users with various tasks. We discuss open problems and challenges with respect to modeling spoken information queries for virtual assistants, and list opportunities where Information Retrieval methods and research can be applied to improve the quality of virtual assistant speech recognition. We discuss how query domain classification, knowledge graphs and user interaction data, and query personalization can be helpful to improve the accurate recognition of spoken information domain queries. Finally, we also provide a brief overview of current problems and challenges in speech recognition.