# Server-side Rescoring of Spoken Entity-centric Knowledge Queries for Virtual Assistants

arXiv: https://arxiv.org/abs/2311.01398 · HTML: https://mlsanity.com/paper/2311.01398/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 2 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [RescoreBERT: Discriminative Speech Recognition Rescoring with BERT](https://mlsanity.com/paper/2202.01094/) (2022)
- [Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpus](https://mlsanity.com/paper/2008.10491/) (2020)
- [SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition](https://mlsanity.com/paper/1910.01992/) (2019)
- [Improving Rare Word Recognition with LM-aware MWER Training](https://mlsanity.com/paper/2204.07553/) (2022)
- [An Empirical Study of Efficient ASR Rescoring with Transformers](https://mlsanity.com/paper/1910.11450/) (2019)
- [Sentence-Select: Large-Scale Language Model Data Selection for Rare-Word Speech Recognition](https://mlsanity.com/paper/2203.05008/) (2022)
- [On-device neural speech synthesis](https://mlsanity.com/paper/2109.08710/) (2021)
- [Predicting Entity Popularity to Improve Spoken Entity Recognition by Virtual Assistants](https://mlsanity.com/paper/2005.12816/) (2020)
- [Space-Efficient Representation of Entity-centric Query Language Models](https://mlsanity.com/paper/2206.14885/) (2022)
- [ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks](https://mlsanity.com/paper/2207.01893/) (2022)

## Built on by (most cited first, 2 of 2)

- [Transformer-based Model for ASR N-Best Rescoring and Rewriting](https://mlsanity.com/paper/2406.08207/) (2024)
- [Phonetically-Augmented Discriminative Rescoring for Voice Search Error Correction](https://mlsanity.com/paper/2506.06117/) (2025)

## Abstract (arXiv)

On-device Virtual Assistants (VAs) powered by Automatic Speech Recognition (ASR) require effective knowledge integration for the challenging entity-rich query recognition. In this paper, we conduct an empirical study of modeling strategies for server-side rescoring of spoken information domain queries using various categories of Language Models (LMs) (N-gram word LMs, sub-word neural LMs). We investigate the combination of on-device and server-side signals, and demonstrate significant WER improvements of 23%-35% on various entity-centric query subpopulations by integrating various server-side LMs compared to performing ASR on-device only. We also perform a comparison between LMs trained on domain data and a GPT-3 variant offered by OpenAI as a baseline. Furthermore, we also show that model fusion of multiple server-side LMs trained from scratch most effectively combines complementary strengths of each model and integrates knowledge learned from domain-specific data to a VA ASR system.