mlsanityData to 14 Oct 2026

SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?

arXiv 2508.12243 ↗First posted 17 Aug 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Multilingual text embeddings are often assumed to encode meaning in a perspective-independent semantic space, yielding stable similarity judgments across tasks and languages. Our results show that this assumption does not hold in practice. We introduce SEA-BED, a large-scale benchmark covering 10 Southeast Asian (SEA) languages and diverse embedding tasks, designed to systematically examine how embedding performance varies across tasks, languages, and language-task combinations. Across extensive evaluations, we observe that no single model performs uniformly well across SEA languages; task difficulty differs markedly within languages, and success on one task does not reliably generalize to others. Language-task analyses further reveal highly non-uniform performance landscapes, where performance varies across different language-task combinations. These findings call for closer attention to performance measurements that provide an expansive view across languages and tasks to uncover inconsistencies in semantic representation. Based on these observations, we provide insights for future model development, including data, algorithmic, and architectural considerations.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 62 of its 144 references that are in the map. First posted on arXiv 17 Aug 2025.

Builds on

62 of 144 references.

References · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  9. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  10. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
38 more
  1. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  2. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  4. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  5. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  7. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  8. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  9. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  10. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  11. 2024Generative Representational Instruction Tuningcited by 233
  12. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
  13. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  14. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 141
  15. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
  16. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  17. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 128
  18. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  19. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  20. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  21. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  22. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 69
  23. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  24. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  25. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 58
  26. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  27. 2020QED: A Framework and Dataset for Explanations in Question Answeringcited by 55
  28. 2023PyThaiNLP: Thai Natural Language Processing in Pythoncited by 28
  29. 2024The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embeddingcited by 26
  30. 2021A Large-scale Dataset for Hate Speech Detection on Vietnamese Social Media Textscited by 25
  31. 2021IndoNLI: A Natural Language Inference Dataset for Indonesiancited by 24
  32. 2019Emotion Recognition for Vietnamese Social Media Textcited by 23
  33. 2021Intent Detection and Slot Filling for Vietnamesecited by 19
  34. 2020Improving Bi-LSTM Performance for Indonesian Sentiment Analysis Using Paragraph Vectorcited by 18
  35. 2021Constructive and Toxic Speech Detection for Open-domain Social Media Comments in Vietnamesecited by 17
  36. 2020Towards Computational Linguistics in Minangkabau Language: Studies on Sentiment Analysis and Machine Translationcited by 16
  37. 2023QASiNa: Religious Domain Question Answering using Sirah Nabawiyahcited by 15
  38. 2020Liputan6: A Large-scale Indonesian Dataset for Text Summarizationcited by 14

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languagescited by 3
  2. 2026GraphRAG for Engineering Diagrams: ChatP&ID Enables LLM Interaction with P&IDscited by 1