mlsanityData to 14 Oct 2026

SENSE models: an open source solution for multilingual and multimodal semantic-based tasks

arXiv 2509.12093 ↗First posted 15 Sep 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper introduces SENSE (Shared Embedding for N-lingual Speech and tExt), an open-source solution inspired by the SAMU-XLSR framework and conceptually similar to Meta AI's SONAR models. These approaches rely on a teacher-student framework to align a self-supervised speech encoder with the language-agnostic continuous representations of a text encoder at the utterance level. We describe how the original SAMU-XLSR method has been updated by selecting a stronger teacher text model and a better initial speech encoder. The source code for training and using SENSE models has been integrated into the SpeechBrain toolkit, and the first SENSE model we trained has been publicly released. We report experimental results on multilingual and multimodal semantic tasks, where our SENSE model achieves highly competitive performance. Finally, this study offers new insights into how semantics are captured in such semantically aligned speech encoders.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

33 of 46 references.

References · 33

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  8. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  10. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  11. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  12. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
21 more
  1. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  2. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  3. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  4. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  5. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  6. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  7. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  8. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
  9. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  10. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 71
  11. 2024Open-Source Conversational AI with SpeechBrain 1.0cited by 56
  12. 2020BARThez: a Skilled Pretrained French Sequence-to-Sequence Modelcited by 40
  13. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  14. 2016Attentive Pooling Networkscited by 28
  15. 2024Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyondcited by 20
  16. 2020On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognitioncited by 19
  17. 2018End-to-end named entity extraction from speechcited by 11
  18. 2022End-to-end model for named entity recognition from speech without paired training datacited by 11
  19. 2022On the Use of Semantically-Aligned Speech Representations for Spoken Language Understandingcited by 7
  20. 2024Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialectcited by 7
  21. 2024MSNER: A Multilingual Speech Dataset for Named Entity Recognitioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Pantagruel: Unified Self-Supervised Encoders for French Text and Speechcited by 2
  2. 2026Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarizationcited by 0
  3. 2026Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encodercited by 0
  4. 2026SLURP-TN : Resource for Tunisian Dialect Spoken Language Understandingcited by 0
  5. 2026FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddingscited by 0
  6. 2026NADI 2026: The Second Multidialectal Arabic Speech Processing Shared Taskcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022An Analysis of Semantically-Aligned Speech-Text Embeddings
  2. 2025Multi-Sense Embeddings for Language Models and Knowledge Distillation
  3. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
  4. 2026Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech
  5. 2024Spirit LM: Interleaved Spoken and Written Language Model
  6. 2023Improved Cross-Lingual Transfer Learning For Automatic Speech Translation
  7. 2025Mellow: a small audio language model for reasoning
  8. 2025Instituto de Telecomunicacc~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning