# PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding

arXiv: https://arxiv.org/abs/2406.07823 · HTML: https://mlsanity.com/paper/2406.07823/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 30 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Non-Autoregressive Machine Translation with Latent Alignments](https://mlsanity.com/paper/2004.07437/) (2020)
- [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://mlsanity.com/paper/1811.04719/) (2018)
- [Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2010.03546/) (2020)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Deliberation Model Based Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/2003.07962/) (2020)
- [Conversational Semantic Parsing](https://mlsanity.com/paper/2009.13655/) (2020)
- [Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces](https://mlsanity.com/paper/2008.06173/) (2020)
- [STOP: A dataset for Spoken Task Oriented Semantic Parsing](https://mlsanity.com/paper/2207.10643/) (2022)
- [Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2104.07275/) (2021)
- [Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog](https://mlsanity.com/paper/2104.04923/) (2021)
- [Don't Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding](https://mlsanity.com/paper/2010.03714/) (2020)
- [FANS: Fusing ASR and NLU for on-device SLU](https://mlsanity.com/paper/2111.00400/) (2021)
- [Retrieve-and-Fill for Scenario-based Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2202.00901/) (2022)
- [Non-autoregressive Error Correction for CTC-based ASR with Phone-conditioned Masked LM](https://mlsanity.com/paper/2209.04062/) (2022)

## Abstract (arXiv)

Spoken Language Understanding (SLU) is a critical component of voice assistants; it consists of converting speech to semantic parses for task execution. Previous works have explored end-to-end models to improve the quality and robustness of SLU models with Deliberation, however these models have remained autoregressive, resulting in higher latencies. In this work we introduce PRoDeliberation, a novel method leveraging a Connectionist Temporal Classification-based decoding strategy as well as a denoising objective to train robust non-autoregressive deliberation models. We show that PRoDeliberation achieves the latency reduction of parallel decoding (2-10x improvement over autoregressive models) while retaining the ability to correct Automatic Speech Recognition (ASR) mistranscriptions of autoregressive deliberation systems. We further show that the design of the denoising training allows PRoDeliberation to overcome the limitations of small ASR devices, and we provide analysis on the necessity of each component of the system.