# Device Directedness with Contextual Cues for Spoken Dialog Systems

arXiv: https://arxiv.org/abs/2211.13280 · HTML: https://mlsanity.com/paper/2211.13280/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization](https://mlsanity.com/paper/2012.06659/) (2020)
- [Device-directed Utterance Detection](https://mlsanity.com/paper/1808.02504/) (2018)
- [Improving End-to-End Speech-to-Intent Classification with Reptile](https://mlsanity.com/paper/2008.01994/) (2020)
- [End-to-End Spoken Language Understanding for Generalized Voice Assistants](https://mlsanity.com/paper/2106.09009/) (2021)
- [Representation learning through cross-modal conditional teacher-student training for speech emotion recognition](https://mlsanity.com/paper/2112.00158/) (2021)
- [Exploring attention mechanism for acoustic-based classification of speech utterances into system-directed and non-system-directed](https://mlsanity.com/paper/1902.00570/) (2019)
- [Sequential End-to-End Intent and Slot Label Classification and Localization](https://mlsanity.com/paper/2106.04660/) (2021)

## Built on by (most cited first, 1 of 1)

- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)

## Abstract (arXiv)

In this work, we define barge-in verification as a supervised learning task where audio-only information is used to classify user spoken dialogue into true and false barge-ins. Following the success of pre-trained models, we use low-level speech representations from a self-supervised representation learning model for our downstream classification task. Further, we propose a novel technique to infuse lexical information directly into speech representations to improve the domain-specific language information implicitly learned during pre-training. Experiments conducted on spoken dialog data show that our proposed model trained to validate barge-in entirely from speech representations is faster by 38% relative and achieves 4.5% relative F1 score improvement over a baseline LSTM model that uses both audio and Automatic Speech Recognition (ASR) 1-best hypotheses. On top of this, our best proposed model with lexically infused representations along with contextual features provides a further relative improvement of 5.7% in the F1 score but only 22% faster than the baseline.