mlsanityData to 7 Oct 2026

CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval

arXiv 2412.13071 ↗First posted 17 Dec 2024cs.CL cs.IR cs.SDcited by 2

Not on the map. Why?

Abstract (arXiv)

This study introduces CLASP (Contrastive Language-Speech Pretraining), a multilingual, multimodal representation tailored for audio-text information retrieval. CLASP leverages the synergy between spoken content and textual data. During training, we utilize our newly introduced speech-text dataset, which encompasses 15 diverse categories ranging from fiction to religion. CLASP's audio component integrates audio spectrograms with a pre-trained self-supervised speech model, while its language encoding counterpart employs a sentence encoder pre-trained on over 100 languages. This unified lightweight model bridges the gap between various modalities and languages, enhancing its effectiveness in handling and retrieving multilingual and multimodal data. Our evaluations across multiple languages demonstrate that CLASP establishes new benchmarks in HITS@1, MRR, and meanR metrics, outperforming traditional ASR-based retrieval methods that rely on transcribing speech into text for subsequent text retrieval, especially in specific scenarios.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

31 of 45 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  10. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  11. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  12. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
19 more
  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  3. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  4. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  5. 2022Audio Self-supervised Learning: A Surveycited by 65
  6. 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 61
  7. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  8. 2020FroDO: From Detections to 3D Objectscited by 51
  9. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
  10. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  11. 2018Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spacescited by 34
  12. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  13. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  14. 2022Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancementcited by 20
  15. 2022SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languagescited by 16
  16. 2021Optimizing Alignment of Speech and Language Latent Spaces for End-to-End Speech Recognition and Understandingcited by 11
  17. 2023Few-shot Joint Multimodal Aspect-Sentiment Analysis Based on Generative Multimodal Promptcited by 9
  18. 2020Acoustic Neighbor Embeddingscited by 3
  19. 2023Adaptive Knowledge Distillation between Text and Speech Pre-trained Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  2. 2026Don't Just Listen, Try Planning: Graph-based Retrieval-Generation Agent for Long-form Audio Meeting Understandingcited by 0