# Transformer-based Model for ASR N-Best Rescoring and Rewriting

arXiv: https://arxiv.org/abs/2406.08207 · HTML: https://mlsanity.com/paper/2406.08207/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01818/) (2017)
- [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019)
- [A spelling correction model for end-to-end speech recognition](https://mlsanity.com/paper/1902.07178/) (2019)
- [Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model](https://mlsanity.com/paper/1910.10697/) (2019)
- [RescoreBERT: Discriminative Speech Recognition Rescoring with BERT](https://mlsanity.com/paper/2202.01094/) (2022)
- [Transformer Based Deliberation for Two-Pass Speech Recognition](https://mlsanity.com/paper/2101.11577/) (2021)
- [An Empirical Study of Efficient ASR Rescoring with Transformers](https://mlsanity.com/paper/1910.11450/) (2019)
- [Connecting and Comparing Language Model Interpolation Techniques](https://mlsanity.com/paper/1908.09738/) (2019)
- [Improving Deliberation by Text-Only and Semi-Supervised Training](https://mlsanity.com/paper/2206.14716/) (2022)
- [Synthetic Query Generation using Large Language Models for Virtual Assistants](https://mlsanity.com/paper/2406.06729/) (2024)
- [Error-driven Pruning of Language Models for Virtual Assistants](https://mlsanity.com/paper/2102.07219/) (2021)
- [Space-Efficient Representation of Entity-centric Query Language Models](https://mlsanity.com/paper/2206.14885/) (2022)
- [Lattention: Lattice-attention in ASR rescoring](https://mlsanity.com/paper/2111.10157/) (2021)
- [Acoustic Model Fusion for End-to-end Speech Recognition](https://mlsanity.com/paper/2310.07062/) (2023)
- [Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunities](https://mlsanity.com/paper/2304.13149/) (2023)
- [Server-side Rescoring of Spoken Entity-centric Knowledge Queries for Virtual Assistants](https://mlsanity.com/paper/2311.01398/) (2023)

## Built on by (most cited first, 3 of 3)

- [ProGRes: Prompted Generative Rescoring on ASR n-Best](https://mlsanity.com/paper/2409.00217/) (2024)
- [Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing](https://mlsanity.com/paper/2509.16622/) (2025)
- [Phonetically-Augmented Discriminative Rescoring for Voice Search Error Correction](https://mlsanity.com/paper/2506.06117/) (2025)

## Abstract (arXiv)

Voice assistants increasingly use on-device Automatic Speech Recognition (ASR) to ensure speed and privacy. However, due to resource constraints on the device, queries pertaining to complex information domains often require further processing by a search engine. For such applications, we propose a novel Transformer based model capable of rescoring and rewriting, by exploring full context of the N-best hypotheses in parallel. We also propose a new discriminative sequence training objective that can work well for both rescore and rewrite tasks. We show that our Rescore+Rewrite model outperforms the Rescore-only baseline, and achieves up to an average 8.6% relative Word Error Rate (WER) reduction over the ASR system by itself.