mlsanityData to 7 Oct 2026

SONAR: Sentence-Level Multimodal and Language-Agnostic Representations

arXiv 2308.11466 ↗First posted 22 Aug 2023cs.CLcited by 105

Not on the map. Why?

Abstract (arXiv)

We introduce SONAR, a new multilingual and multimodal fixed-size sentence embedding space. Our single text encoder, covering 200 languages, substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks. Speech segments can be embedded in the same SONAR embedding space using language-specific speech encoders trained in a teacher-student setting on speech transcription data. Our encoders outperform existing speech encoders on similarity search tasks. We also provide a text decoder for 200 languages, which allows us to perform text-to-text and speech-to-text machine translation, including for zero-shot language and modality combinations. Our text-to-text results are competitive compared to the state-of-the-art NLLB~1B model, despite the fixed-size bottleneck representation. Our zero-shot speech-to-text translation results compare favorably with strong supervised baselines such as Whisper.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 22 Aug 2023.

Builds on

24 of 32 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  10. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  11. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  12. 2018Universal Sentence Encodercited by 485
12 more
  1. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  2. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  3. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  4. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  6. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  7. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  8. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  9. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  10. 2019Improving Zero-shot Translation with Language-Independent Constraintscited by 33
  11. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  12. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 38; 2025: 34; 2026: 31.

105 papers.

Sort:

By area:

  1. 2024The Faiss librarycited by 370
  2. 2024Generative Representational Instruction Tuningcited by 227
  3. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  4. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  5. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  6. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
  7. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 26
  8. 2024In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translationcited by 21
  9. 2025BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translationcited by 21
  10. 2025Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Modelscited by 17
  11. 2024Speech Retrieval-Augmented Generation without Automatic Speech Recognitioncited by 14
  12. 2023SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERTcited by 13
38 more
  1. 2024Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Modelscited by 13
  2. 2025Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translationcited by 13
  3. 2024Cross-Lingual Transfer Learning for Speech Translationcited by 11
  4. 2024SpeechQE: Estimating the Quality of Direct Speech Translationcited by 10
  5. 2025GLAP: General contrastive audio-text pretraining across domains and languagescited by 10
  6. 2025Conditions for Catastrophic Forgetting in Multilingual Translationcited by 10
  7. 2024How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representationscited by 8
  8. 2024Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialectcited by 7
  9. 2024Bridging Language Gaps in Audio-Text Retrievalcited by 6
  10. 2025ATRI: Mitigating Multilingual Audio Text Retrieval Inconsistencies by Reducing Data Distribution Errorscited by 6
  11. 2025Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impactcited by 6
  12. 2025SENSE models: an open source solution for multilingual and multimodal semantic-based taskscited by 6
  13. 2024Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systemscited by 5
  14. 2024Enhancing Multilingual Voice Toxicity Detection with Speech-Text Alignmentcited by 5
  15. 2025Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generationcited by 5
  16. 2025ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonalitycited by 5
  17. 2026Limited Linguistic Diversity in Embodied AI Datasetscited by 5
  18. 2024Improving Multi-lingual Alignment Through Soft Contrastive Learningcited by 4
  19. 2024News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendationcited by 4
  20. 2024Exploring Retrieval Augmented Generation in Arabiccited by 4
  21. 2024Empowering Low-Resource Language ASR via Large-Scale Pseudo Labelingcited by 4
  22. 2024Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languagescited by 4
  23. 2024LCFO: Long Context and Long Form Output Dataset and Benchmarkingcited by 4
  24. 2025LLM Reasoning for Machine Translation: Synthetic Data Generation over Thinking Tokenscited by 4
  25. 2024Towards Red Teaming in Multimodal and Multilingual Translationcited by 3
  26. 2024uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimescited by 3
  27. 2024Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approachcited by 3
  28. 2025SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularizationcited by 3
  29. 2025SONAR-SLT: Multilingual Sign Language Translation via Language-Agnostic Sentence Embedding Supervisioncited by 3
  30. 2024A New Benchmark for Evaluating Automatic Speech Recognition in the Arabic Call Domaincited by 2
  31. 2024Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignmentcited by 2
  32. 2024Multi-Target Cross-Lingual Summarization: a novel task and a language-neutral approachcited by 2
  33. 2024On the Role of Speech Data in Reducing Toxicity Detection Biascited by 2
  34. 2025On the Robust Approximation of ASR Metricscited by 2
  35. 2025In-context Language Learning for Endangered Languages in Speech Recognitioncited by 2
  36. 2025Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?cited by 2
  37. 2025Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialectscited by 2
  38. 2025ParaScopes: What do Language Models Activations Encode About Future Text?cited by 2

50 most cited of 105 All 105 (JSON)