# Joint Multimodal Contrastive Learning for Robust Spoken Term Detection and Keyword Spotting

arXiv: https://arxiv.org/abs/2512.14115 · HTML: https://mlsanity.com/paper/2512.14115/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers](https://mlsanity.com/paper/2108.11996/) (2021)
- [Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder](https://mlsanity.com/paper/1603.00982/) (2016)
- [Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models](https://mlsanity.com/paper/1811.00403/) (2018)
- [Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches](https://mlsanity.com/paper/1611.02550/) (2016)
- [Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1706.03818/) (2017)
- [Multi-view Recurrent Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1611.04496/) (2016)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings](https://mlsanity.com/paper/2012.02221/) (2020)
- [Improved acoustic word embeddings for zero-resource languages using multilingual transfer](https://mlsanity.com/paper/2006.02295/) (2020)
- [Multilingual Distributed Representations without Word Alignment](https://mlsanity.com/paper/1312.6173/) (2013)

## Abstract (arXiv)

Acoustic Word Embeddings (AWEs) improve the efficiency of speech retrieval tasks such as Spoken Term Detection (STD) and Keyword Spotting (KWS). However, existing approaches suffer from limitations, including unimodal supervision, disjoint optimization of audio-audio and audio-text alignment, and the need for task-specific models. To address these shortcomings, we propose a joint multimodal contrastive learning framework that unifies both acoustic and cross-modal supervision in a shared embedding space. Our approach simultaneously optimizes: (i) audio-text contrastive learning, inspired by the CLAP loss, to align audio and text representations and (ii) audio-audio contrastive learning, via Deep Word Discrimination (DWD) loss, to enhance intra-class compactness and inter-class separation. The proposed method outperforms existing AWE baselines on word discrimination task while flexibly supporting both STD and KWS. To our knowledge, this is the first comprehensive approach of its kind.