# Multi-task Learning with Cross Attention for Keyword Spotting

arXiv: https://arxiv.org/abs/2107.07634 · HTML: https://mlsanity.com/paper/2107.07634/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1703.05390/) (2017)
- [End-to-End Streaming Keyword Spotting](https://mlsanity.com/paper/1812.02802/) (2018)
- [Streaming Small-Footprint Keyword Spotting using Sequence-to-Sequence Models](https://mlsanity.com/paper/1710.09617/) (2017)
- [A Cascade Architecture for Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1712.03603/) (2017)
- [Predicting detection filters for small footprint open-vocabulary keyword spotting](https://mlsanity.com/paper/1912.07575/) (2019)
- [Hybrid Transformer/CTC Networks for Hardware Efficient Voice Triggering](https://mlsanity.com/paper/2008.02323/) (2020)
- [Stacked 1D convolutional networks for end-to-end small footprint voice trigger detection](https://mlsanity.com/paper/2008.03405/) (2020)
- [Progressive Voice Trigger Detection: Accuracy vs Latency](https://mlsanity.com/paper/2010.15446/) (2020)
- [Multi-task Learning for Voice Trigger Detection](https://mlsanity.com/paper/2001.09519/) (2020)
- [Small-footprint Keyword Spotting Using Deep Neural Network and Connectionist Temporal Classifier](https://mlsanity.com/paper/1709.03665/) (2017)

## Built on by (most cited first, 1 of 1)

- [Improving Voice Trigger Detection with Metric Learning](https://mlsanity.com/paper/2204.02455/) (2022)

## Abstract (arXiv)

Keyword spotting (KWS) is an important technique for speech applications, which enables users to activate devices by speaking a keyword phrase. Although a phoneme classifier can be used for KWS, exploiting a large amount of transcribed data for automatic speech recognition (ASR), there is a mismatch between the training criterion (phoneme recognition) and the target task (KWS). Recently, multi-task learning has been applied to KWS to exploit both ASR and KWS training data. In this approach, an output of an acoustic model is split into two branches for the two tasks, one for phoneme transcription trained with the ASR data and one for keyword classification trained with the KWS data. In this paper, we introduce a cross attention decoder in the multi-task learning framework. Unlike the conventional multi-task learning approach with the simple split of the output layer, the cross attention decoder summarizes information from a phonetic encoder by performing cross attention between the encoder outputs and a trainable query sequence to predict a confidence score for the KWS task. Experimental results on KWS tasks show that the proposed approach achieves a 12% relative reduction in the false reject ratios compared to the conventional multi-task learning with split branches and a bi-directional long short-team memory decoder.