mlsanityData to 14 Oct 2026

Benchmarking Sociolinguistic Diversity in Swahili NLP: A Taxonomy-Guided Approach

arXiv 2508.14051 ↗First posted 6 Aug 2025cs.CL cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce the first taxonomy-guided evaluation of Swahili NLP, addressing gaps in sociolinguistic diversity. Drawing on health-related psychometric tasks, we collect a dataset of 2,170 free-text responses from Kenyan speakers. The data exhibits tribal influences, urban vernacular, code-mixing, and loanwords. We develop a structured taxonomy and use it as a lens for examining model prediction errors across pre-trained and instruction-tuned language models. Our findings advance culturally grounded evaluation frameworks and highlight the role of sociolinguistic variation in shaping model performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 57 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

23 of 57 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  8. 2018A comparative study of fairness-enhancing interventions in machine learningcited by 208
  9. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 196
  10. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  11. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  12. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 135
11 more
  1. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 102
  2. 2020Low-resource Languages: A Review of Past Work and Future Challengescited by 91
  3. 2018Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processingcited by 63
  4. 2018Modeling Empathy and Distress in Reaction to News Storiescited by 52
  5. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 50
  6. 2019The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the xAUC Metriccited by 41
  7. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  8. 2024The Sociolinguistic Foundations of Language Modelingcited by 13
  9. 2024Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalizationcited by 7
  10. 2024State of NLP in Kenya: A Surveycited by 7
  11. 2021Sentiment Classification in Swahili Language Using Multilingual BERTcited by 6

Built on by

Not cited yet.