mlsanityData to 14 Oct 2026

Position: AI Leaderboards Are Underserving the Global South: A Case Study from India

arXiv 2608.18117 ↗First posted 4 Jul 2026cs.AI econ.GN q-fin.ECcited by 0

Not on the map. Why?

Abstract (arXiv)

This position paper argues that AI leaderboards are structurally ill-suited to serving the Global South because they lack independent governance, conflict-of-interest policies, and mechanisms for metric evolution. The barrier is not missing data; high-quality regional benchmarks already exist: IndicSUPERB, MILU, and LAHAJA for India; IrokoBench for Africa; AlGhafa for Arabic. The barrier is institutional design. Global leaderboards do not include these benchmarks, and no governance mechanism compels them to do so. Commercial pressure corrects leaderboard failures when paying customers in the Global North are affected. The Global South lacks equivalent leverage. Without governance, failures affecting Hindi, Swahili, or Arabic speakers persist indefinitely as documented but unaddressed gaps. Using India as a case study (1.4 billion people, 22 scheduled languages, high-quality benchmarks, but no trusted aggregation), we report findings from a consultation with 58 AI practitioners showing consistent preference for formal governance and disclosure-based conflict management. The solution is not more data but better institutions: regional leaderboards with independent governance from the start.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 44 references that are in the map. First posted on arXiv 4 Jul 2026.

Builds on

19 of 44 references.

References · 19

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2016Using Deep Learning for Image-Based Plant Disease Detectioncited by 219
  3. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  4. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 124
  5. 2025The Leaderboard Illusioncited by 66
  6. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  7. 2025Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and ASTcited by 60
  8. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 56
  9. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuningcited by 48
  10. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  11. 2022IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languagescited by 40
  12. 2020The Problem with Metrics is a Fundamental Problem for AIcited by 37
7 more
  1. 2025Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluationcited by 23
  2. 2023Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASRcited by 21
  3. 2023Svarah: Evaluating English ASR Systems on Indian Accentscited by 16
  4. 2024LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systemscited by 10
  5. 2025TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Storiescited by 10
  6. 2025Investigating Hallucination in Conversations for Low Resource Languagescited by 2
  7. 2023Current policies governing editorial conflicts of interest are ineffectivecited by 1

Built on by

Not cited yet.