# Learning Efficient Representations for Keyword Spotting with Triplet Loss

arXiv: https://arxiv.org/abs/2101.04792 · HTML: https://mlsanity.com/paper/2101.04792/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 12 Jan 2021.

## Builds on (most cited first, 21 of 21)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014)
- [Hello Edge: Keyword Spotting on Microcontrollers](https://mlsanity.com/paper/1711.07128/) (2017)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [Deep Residual Learning for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1710.10361/) (2017)
- [Deep Speaker: an End-to-End Neural Speaker Embedding System](https://mlsanity.com/paper/1705.02304/) (2017)
- [Streaming keyword spotting on mobile devices](https://mlsanity.com/paper/2005.06720/) (2020)
- [A neural attention model for speech command recognition](https://mlsanity.com/paper/1808.08929/) (2018)
- [MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition](https://mlsanity.com/paper/2004.08531/) (2020)
- [Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1703.05390/) (2017)
- [Streaming Small-Footprint Keyword Spotting using Sequence-to-Sequence Models](https://mlsanity.com/paper/1710.09617/) (2017)
- [TristouNet: Triplet Loss for Speaker Turn Embedding](https://mlsanity.com/paper/1609.04301/) (2016)
- [Max-Pooling Loss Training of Long Short-Term Memory Networks for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1705.02411/) (2017)
- [Metric Learning for Keyword Spotting](https://mlsanity.com/paper/2005.08776/) (2020)
- [Honk: A PyTorch Reimplementation of Convolutional Neural Networks for Keyword Spotting](https://mlsanity.com/paper/1710.06554/) (2017)
- [Howl: A Deployed, Open-Source Wake Word Detection System](https://mlsanity.com/paper/2008.09606/) (2020)
- [Triplet Network with Attention for Speaker Diarization](https://mlsanity.com/paper/1808.01535/) (2018)
- [End-to-end Triplet Loss based Emotion Embedding System for Speech Emotion Recognition](https://mlsanity.com/paper/2010.06200/) (2020)

## Built on by (most cited first, 24 of 24)

- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [Keyword Transformer: A Self-Attention Model for Keyword Spotting](https://mlsanity.com/paper/2104.00769/) (2021)
- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks](https://mlsanity.com/paper/2303.00733/) (2023)
- [Towards Stealthy Backdoor Attacks against Speech Recognition via Elements of Sound](https://mlsanity.com/paper/2307.08208/) (2023)
- [Leveraging Large Language Models for Exploiting ASR Uncertainty](https://mlsanity.com/paper/2309.04842/) (2023)
- [LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units](https://mlsanity.com/paper/2402.04882/) (2024)
- [Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding](https://mlsanity.com/paper/2308.06472/) (2023)
- [Few-Shot Open-Set Learning for On-Device Customization of KeyWord Spotting Systems](https://mlsanity.com/paper/2306.02161/) (2023)
- [UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions](https://mlsanity.com/paper/2310.02973/) (2023)
- [Self-Learning for Personalized Keyword Spotting on Ultra-Low-Power Audio Sensors](https://mlsanity.com/paper/2408.12481/) (2024)
- [FedorAS: Federated Architecture Search under system heterogeneity](https://mlsanity.com/paper/2206.11239/) (2022)
- [Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition](https://mlsanity.com/paper/2110.03894/) (2021)
- [Comparison of semi-supervised deep learning algorithms for audio classification](https://mlsanity.com/paper/2102.08183/) (2021)
- [Attention-Free Keyword Spotting](https://mlsanity.com/paper/2110.07749/) (2021)
- [Speech Augmentation Based Unsupervised Learning for Keyword Spotting](https://mlsanity.com/paper/2205.14329/) (2022)
- [Finite Gaussian Neurons: Defending against adversarial attacks by making neural networks say "I don't know"](https://mlsanity.com/paper/2306.07796/) (2023)
- [Enhancing Few-shot Keyword Spotting Performance through Pre-Trained Self-supervised Speech Models](https://mlsanity.com/paper/2506.17686/) (2025)
- [Language models for longitudinal analysis of abusive content in Billboard Music Charts](https://mlsanity.com/paper/2510.06266/) (2025)
- [Low-Resource Spoken Language Identification Using Self-Attentive Pooling and Deep 1D Time-Channel Separable Convolutions](https://mlsanity.com/paper/2106.00052/) (2021)
- [Learning Audio Representations with MLPs](https://mlsanity.com/paper/2203.08490/) (2022)
- [Contrastive Augmentation: An Unsupervised Learning Approach for Keyword Spotting in Speech Technology](https://mlsanity.com/paper/2409.00356/) (2024)
- [TriG-NER: Triplet-Grid Framework for Discontinuous Named Entity Recognition](https://mlsanity.com/paper/2411.01839/) (2024)
- [Edgeworth Expansion for Semi-hard Triplet Loss](https://mlsanity.com/paper/2503.12893/) (2025)

## Abstract (arXiv)

In the past few years, triplet loss-based metric embeddings have become a de-facto standard for several important computer vision problems, most no-tably, person reidentification. On the other hand, in the area of speech recognition the metric embeddings generated by the triplet loss are rarely used even for classification problems. We fill this gap showing that a combination of two representation learning techniques: a triplet loss-based embedding and a variant of kNN for classification instead of cross-entropy loss significantly (by 26% to 38%) improves the classification accuracy for convolutional networks on a LibriSpeech-derived LibriWords datasets. To do so, we propose a novel phonetic similarity based triplet mining approach. We also improve the current best published SOTA for Google Speech Commands dataset V1 10+2 -class classification by about 34%, achieving 98.55% accuracy, V2 10+2-class classification by about 20%, achieving 98.37% accuracy, and V2 35-class classification by over 50%, achieving 97.0% accuracy.