mlsanityData to 13 Oct 2026

The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language

arXiv 2311.08323 ↗First posted 14 Nov 2023cs.CL cs.SD eess.AScited by 14

Not on the map. Why?

Abstract (arXiv)

In this project, we demonstrate that phoneme-based models for speech processing can achieve strong crosslinguistic generalizability to unseen languages. We curated the IPAPACK, a massively multilingual speech corpora with phonemic transcriptions, encompassing more than 115 languages from diverse language families, selectively checked by linguists. Based on the IPAPACK, we propose CLAP-IPA, a multi-lingual phoneme-speech contrastive embedding model capable of open-vocabulary matching between arbitrary speech signals and phonemic sequences. The proposed model was tested on 95 unseen languages, showing strong generalizability across languages. Temporal alignments between phonemes and speech signals also emerged from contrastive training, enabling zeroshot forced alignment in unseen languages. We further introduced a neural forced aligner IPA-ALIGNER by finetuning CLAP-IPA with the Forward-Sum loss to learn better phone-to-audio alignment. Evaluation results suggest that IPA-ALIGNER can generalize to unseen languages without adaptation.

Key facts. Cited by 14 papers in the map (data through 13 Oct 2026). Builds on 32 of its 69 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

32 of 69 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,004
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,359
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 713
  6. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 637
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 594
  8. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 500
  9. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 459
  10. 2023Scaling Speech Technology to 1,000+ Languagescited by 402
  11. 2020Universal Phone Recognition with a Multilingual Allophone Systemcited by 81
  12. 2021Simple and Effective Zero-shot Cross-lingual Phoneme Recognitioncited by 79
20 more
  1. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 75
  2. 2017Deep Residual Learning for Small-Footprint Keyword Spottingcited by 68
  3. 2021Keyword Transformer: A Self-Attention Model for Keyword Spottingcited by 60
  4. 2022ByT5 model for massively multilingual grapheme-to-phoneme conversioncited by 49
  5. 2020Streaming keyword spotting on mobile devicescited by 47
  6. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
  7. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  8. 2021Few-Shot Keyword Spotting in Any Languagecited by 27
  9. 2021Phone-to-audio alignment without text: A Semi-supervised Approachcited by 25
  10. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  11. 2020fugashi, a Tool for Tokenizing Japanese in Pythoncited by 17
  12. 2023Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributescited by 13
  13. 2023PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywordscited by 12
  14. 2023Flexible Keyword Spotting based on Homogeneous Audio-Text Embeddingcited by 10
  15. 2020A Corpus for Large-Scale Phonetic Typologycited by 9
  16. 2022g2pW: A Conditional Weighted Softmax BERT for Polyphone Disambiguation in Mandarincited by 7
  17. 2023Multilingual Query-by-Example Keyword Spotting with Metric Learning and Phoneme-to-Embedding Mappingcited by 3
  18. 2023Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognitioncited by 3
  19. 2022Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingscited by 2
  20. 2023CLARA: Multilingual Contrastive Learning for Audio Representation Acquisitioncited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 7.

14 papers.

Sort:

  1. 2025ZIPA: A family of efficient models for multilingual phone recognitioncited by 18
  2. 2025POWSM: A Phonetic Open Whisper-Style Speech Foundation Modelcited by 14
  3. 2024From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemescited by 11
  4. 2025IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modelingcited by 6
  5. 2026An Empirical Recipe for Universal Phone Recognitioncited by 5
  6. 2026PRiSM: Benchmarking Phone Realization in Speech Modelscited by 4
  7. 2026Multilingual Dysarthric Speech Assessment Using Universal Phone Recognition and Language-Specific Phonemic Contrast Modelingcited by 4
  8. 2026VoXtream2: Full-stream TTS with dynamic speaking rate controlcited by 3
  9. 2025Happiness is Sharing a Vocabulary: A Study of Transliteration Methodscited by 2
  10. 2024Efficiently Identifying Low-Quality Language Subsets in Multilingual Datasets: A Case Study on a Large-Scale Multilingual Audio Datasetcited by 1
  11. 2025Handling Korean Out-of-Vocabulary Words with Phoneme Representation Learningcited by 0
  12. 2026Does language matter for spoken word classification? A multilingual generative meta-learning approachcited by 0
2 more
  1. 2026Evaluating Bias in Phoneme-Based Automatic Speech Recognition Systems: An Analysis of IPA Transcription Modelscited by 0
  2. 2026Phoneme- and Word-Level Metrics Using Self-Supervised Speech Representations for Forced Alignment Evaluationcited by 0