mlsanityData to 7 Oct 2026

mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks

arXiv 2506.08400 ↗First posted 10 Jun 2025cs.CL cs.LG cs.SDcited by 7

Not on the map. Why?

Abstract (arXiv)

Large Language models (LLMs) have demonstrated impressive performance on a wide range of tasks, including in multimodal settings such as speech. However, their evaluation is often limited to English and a few high-resource languages. For low-resource languages, there is no standardized evaluation benchmark. In this paper, we address this gap by introducing mSTEB, a new benchmark to evaluate the performance of LLMs on a wide range of tasks covering language identification, text classification, question answering, and translation tasks on both speech and text modalities. We evaluated the performance of leading LLMs such as Gemini 2.0 Flash and GPT-4o (Audio) and state-of-the-art open models such as Qwen 2 Audio and Gemma 3 27B. Our evaluation shows a wide gap in performance between high-resource and low-resource languages, especially for languages spoken in Africa and Americas/Oceania. Our findings show that more investment is needed to address their under-representation in LLMs coverage.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

46 of 55 references.

References · 46

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2025Gemma 3 Technical Reportcited by 1,686
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
34 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  2. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  3. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  4. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  5. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  6. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  7. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  8. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  9. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  10. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  11. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  12. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  13. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  14. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  15. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  16. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  17. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  18. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  19. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  20. 2023AfroBench: How Good are Large Language Models on African Languages?cited by 43
  21. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  22. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  23. 2023GlotLID: Language Identification for Low-Resource Languagescited by 37
  24. 2022IndicXNLI: Evaluating Multilingual Inference for Indian Languagescited by 26
  25. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  26. 2021IndoNLI: A Natural Language Inference Dataset for Indonesiancited by 24
  27. 2023ChatGPT MT: Competitive for High- (but not Low-) Resource Languagescited by 23
  28. 2019Evaluation of Sentence Representations in Polishcited by 18
  29. 2022JamPatoisNLI: A Jamaican Patois Natural Language Inference Datasetcited by 11
  30. 2024On the Evaluation of Speech Foundation Models for Spoken Language Understandingcited by 9
  31. 2024Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languagescited by 9
  32. 2024A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpuscited by 8
  33. 2024SpeechTaxi: On Multilingual Semantic Speech Classificationcited by 6
  34. 2025Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understandingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  2. 2025The Token Tax: Systematic Bias in Multilingual Tokenizationcited by 13
  3. 2026GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Modelscited by 2
  4. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  5. 2025No Text Needed: Forecasting MT Quality and Inequity from Fertility and Metadatacited by 0
  6. 2026AfriVox-v2: A Domain-Verticalized Benchmark for In-the-Wild African Speech Recognitioncited by 0
  7. 2026mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?cited by 0