mlsanityData to 7 Oct 2026

MM-KWS: Multi-modal Prompts for Multilingual User-defined Keyword Spotting

arXiv 2406.07310 ↗First posted 11 Jun 2024eess.AS cs.CL cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose MM-KWS, a novel approach to user-defined keyword spotting leveraging multi-modal enrollments of text and speech templates. Unlike previous methods that focus solely on either text or speech features, MM-KWS extracts phoneme, text, and speech embeddings from both modalities. These embeddings are then compared with the query speech embedding to detect the target keywords. To ensure the applicability of MM-KWS across diverse languages, we utilize a feature extractor incorporating several multilingual pre-trained models. Subsequently, we validate its effectiveness on Mandarin and English tasks. In addition, we have integrated advanced data augmentation tools for hard case mining to enhance MM-KWS in distinguishing confusable words. Experimental results on the LibriPhrase and WenetPhrase datasets demonstrate that MM-KWS outperforms prior methods significantly.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

21 of 27 references.

References · 21

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  5. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  6. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  7. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  8. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  9. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 92
  10. 2021Deep Spoken Keyword Spotting: An Overviewcited by 43
  11. 2021Query-by-Example Keyword Spotting system using Multi-head Attention and Softtriple Losscited by 25
  12. 2022Learning Audio-Text Agreement for Open-vocabulary Keyword Spottingcited by 20
9 more
  1. 2019Query-by-example on-device keyword spottingcited by 19
  2. 2020g2pM: A Neural Grapheme-to-Phoneme Conversion Package for Mandarin Chinese Based on a New Open Benchmark Datasetcited by 15
  3. 2023Matching Latent Encoding for Audio-Text based Keyword Spottingcited by 15
  4. 2023PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywordscited by 12
  5. 2023Flexible Keyword Spotting based on Homogeneous Audio-Text Embeddingcited by 10
  6. 2020Stacked 1D convolutional networks for end-to-end small footprint voice trigger detectioncited by 8
  7. 2023Open-vocabulary Keyword-spotting with Adaptive Instance Normalizationcited by 8
  8. 2022Fully Unsupervised Training of Few-shot Keyword Spottingcited by 5
  9. 2021Auto-KWS 2021 Challenge: Task, Datasets, and Baselinescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Adversarial Deep Metric Learning for Cross-Modal Audio-Text Alignment in Open-Vocabulary Keyword Spottingcited by 1
  2. 2024SLiCK: Exploiting Subsequences for Length-Constrained Keyword Spottingcited by 0
  3. 2025GraphemeAug: A Systematic Approach to Synthesized Hard Negative Keyword Spotting Examplescited by 0
  4. 2026ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosodycited by 0