# End-to-end Spoken Language Understanding with Tree-constrained Pointer Generator

arXiv: https://arxiv.org/abs/2210.16554 · HTML: https://mlsanity.com/paper/2210.16554/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 29 Oct 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [Class LM and word mapping for contextual biasing in End-to-End ASR](https://mlsanity.com/paper/2007.05609/) (2020)
- [Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2010.04284/) (2020)
- [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021)
- [End-to-End Neural Transformer Based Spoken Language Understanding](https://mlsanity.com/paper/2008.10984/) (2020)
- [Instant One-Shot Word-Learning for Context-Specific Neural Sequence-to-Sequence Speech Recognition](https://mlsanity.com/paper/2107.02268/) (2021)
- [Contextual Speech Recognition with Difficult Negative Training Examples](https://mlsanity.com/paper/1810.12170/) (2018)
- [End-to-end contextual speech recognition using class language models and a token passing decoder](https://mlsanity.com/paper/1812.02142/) (2018)
- [Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2011.09044/) (2020)
- [Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2104.07253/) (2021)
- [Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition](https://mlsanity.com/paper/2207.00857/) (2022)
- [Speech to Text Adaptation: Towards an Efficient Cross-Modal Distillation](https://mlsanity.com/paper/2005.08213/) (2020)
- [Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2205.09058/) (2022)
- [Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understanding](https://mlsanity.com/paper/2102.06750/) (2021)
- [Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech](https://mlsanity.com/paper/2105.04719/) (2021)
- [Neural-FST Class Language Model for End-to-End Speech Recognition](https://mlsanity.com/paper/2201.11867/) (2022)

## Built on by (most cited first, 6 of 6)

- [Can Contextual Biasing Remain Effective with Whisper and GPT-2?](https://mlsanity.com/paper/2306.01942/) (2023)
- [Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2305.18824/) (2023)
- [Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Data](https://mlsanity.com/paper/2307.01764/) (2023)
- [Speech-based Slot Filling using Large Language Models](https://mlsanity.com/paper/2311.07418/) (2023)
- [An efficient text augmentation approach for contextualized Mandarin speech recognition](https://mlsanity.com/paper/2406.09950/) (2024)
- [Is one brick enough to break the wall of spoken dialogue state tracking?](https://mlsanity.com/paper/2311.04923/) (2023)

## Abstract (arXiv)

End-to-end spoken language understanding (SLU) suffers from the long-tail word problem. This paper exploits contextual biasing, a technique to improve the speech recognition of rare words, in end-to-end SLU systems. Specifically, a tree-constrained pointer generator (TCPGen), a powerful and efficient biasing model component, is studied, which leverages a slot shortlist with corresponding entities to extract biasing lists. Meanwhile, to bias the SLU model output slot distribution, a slot probability biasing (SPB) mechanism is proposed to calculate a slot distribution from TCPGen. Experiments on the SLURP dataset showed consistent SLU-F1 improvements using TCPGen and SPB, especially on unseen entities. On a new split by holding out 5 slot types for the test, TCPGen with SPB achieved zero-shot learning with an SLU-F1 score over 50% compared to baselines which can not deal with it. In addition to slot filling, the intent classification accuracy was also improved.