# Transformer-based encoder-encoder architecture for Spoken Term Detection

arXiv: https://arxiv.org/abs/2211.01089 · HTML: https://mlsanity.com/paper/2211.01089/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Keyword Transformer: A Self-Attention Model for Keyword Spotting](https://mlsanity.com/paper/2104.00769/) (2021)
- [Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches](https://mlsanity.com/paper/1611.02550/) (2016)
- [Wake Word Detection with Streaming Transformers](https://mlsanity.com/paper/2102.04488/) (2021)
- [Spoken Term Detection and Relevance Score Estimation using Dot-Product of Pronunciation Embeddings](https://mlsanity.com/paper/2210.11895/) (2022)
- [Exploring Capabilities of Monolingual Audio Transformers using Large Datasets in Automatic Speech Recognition of Czech](https://mlsanity.com/paper/2206.07627/) (2022)
- [CaTT-KWS: A Multi-stage Customized Keyword Spotting Framework based on Cascaded Transducer-Transformer](https://mlsanity.com/paper/2207.01267/) (2022)
- [Deep LSTM Spoken Term Detection using Wav2Vec 2.0 Recognizer](https://mlsanity.com/paper/2210.11885/) (2022)
- [DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks](https://mlsanity.com/paper/2204.08688/) (2022)

## Abstract (arXiv)

The paper presents a method for spoken term detection based on the Transformer architecture. We propose the encoder-encoder architecture employing two BERT-like encoders with additional modifications, including convolutional and upsampling layers, attention masking, and shared parameters. The encoders project a recognized hypothesis and a searched term into a shared embedding space, where the score of the putative hit is computed using the calibrated dot product. In the experiments, we used the Wav2Vec 2.0 speech recognizer, and the proposed system outperformed a baseline method based on deep LSTMs on the English and Czech STD datasets based on USC Shoah Foundation Visual History Archive (MALACH).