# Adversarial Deep Metric Learning for Cross-Modal Audio-Text Alignment in Open-Vocabulary Keyword Spotting

arXiv: https://arxiv.org/abs/2505.16735 · HTML: https://mlsanity.com/paper/2505.16735/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 25 of 25)

- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Deep Residual Learning for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1710.10361/) (2017)
- [SphereFace2: Binary Classification is All You Need for Deep Face Recognition](https://mlsanity.com/paper/2108.01513/) (2021)
- [Query-by-Example Keyword Spotting system using Multi-head Attention and Softtriple Loss](https://mlsanity.com/paper/2102.07061/) (2021)
- [Multi-view Recurrent Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1611.04496/) (2016)
- [Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting](https://mlsanity.com/paper/2206.15400/) (2022)
- [Multilingual Jointly Trained Acoustic and Written Word Embeddings](https://mlsanity.com/paper/2006.14007/) (2020)
- [Matching Latent Encoding for Audio-Text based Keyword Spotting](https://mlsanity.com/paper/2306.05245/) (2023)
- [PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywords](https://mlsanity.com/paper/2308.16511/) (2023)
- [Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding](https://mlsanity.com/paper/2308.06472/) (2023)
- [Additional Shared Decoder on Siamese Multi-view Encoders for Learning Acoustic Word Embeddings](https://mlsanity.com/paper/1910.00341/) (2019)
- [Exploring Binary Classification Loss For Speaker Verification](https://mlsanity.com/paper/2307.08205/) (2023)
- [Phoneme-Level Contrastive Learning for User-Defined Keyword Spotting with Flexible Enrollment](https://mlsanity.com/paper/2412.20805/) (2024)
- [Asymmetric Proxy Loss for Multi-View Acoustic Word Embeddings](https://mlsanity.com/paper/2203.16080/) (2022)
- [AdaMS: Deep Metric Learning with Adaptive Margin and Adaptive Scale for Acoustic Word Discrimination](https://mlsanity.com/paper/2210.14564/) (2022)
- [MM-KWS: Multi-modal Prompts for Multilingual User-defined Keyword Spotting](https://mlsanity.com/paper/2406.07310/) (2024)
- [Relational Proxy Loss for Audio-Text based Keyword Spotting](https://mlsanity.com/paper/2406.05314/) (2024)
- [Text-aware Speech Separation for Multi-talker Keyword Spotting](https://mlsanity.com/paper/2406.12447/) (2024)
- [Learning acoustic word embeddings with phonetically associated triplet network](https://mlsanity.com/paper/1811.02736/) (2018)
- [Text-Aware Adapter for Few-Shot Keyword Spotting](https://mlsanity.com/paper/2412.18142/) (2024)
- [Bridging the Gap between Audio and Text using Parallel-attention for User-defined Keyword Spotting](https://mlsanity.com/paper/2408.03593/) (2024)

## Built on by (most cited first, 1 of 1)

- [MATE: Matryoshka Audio-Text Embeddings for Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2601.14012/) (2026)

## Abstract (arXiv)

For text enrollment-based open-vocabulary keyword spotting (KWS), acoustic and text embeddings are typically compared at either the phoneme or utterance level. To facilitate this, we optimize acoustic and text encoders using deep metric learning (DML), enabling direct comparison of multi-modal embeddings in a shared embedding space. However, the inherent heterogeneity between audio and text modalities presents a significant challenge. To address this, we propose Modality Adversarial Learning (MAL), which reduces the domain gap in heterogeneous modality representations. Specifically, we train a modality classifier adversarially to encourage both encoders to generate modality-invariant embeddings. Additionally, we apply DML to achieve phoneme-level alignment between audio and text, and conduct extensive comparisons across various DML objectives. Experiments on the Wall Street Journal (WSJ) and LibriPhrase datasets demonstrate the effectiveness of the proposed approach.