mlsanityData to 7 Oct 2026

LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages

arXiv 2305.14263 ↗First posted 23 May 2023cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Knowing the language of an input text/audio is a necessary first step for using almost every NLP tool such as taggers, parsers, or translation systems. Language identification is a well-studied problem, sometimes even considered solved; in reality, due to lack of data and computational challenges, current systems cannot accurately identify most of the world's 7000 languages. To tackle this bottleneck, we first compile a corpus, MCS-350, of 50K multilingual and parallel children's stories in 350+ languages. MCS-350 can serve as a benchmark for language identification of short texts and for 1400+ new translation directions in low-resource Indian and African languages. Second, we propose a novel misprediction-resolution hierarchical model, LIMIt, for language identification that reduces error by 55% (from 0.71 to 0.32) on our compiled children's stories dataset and by 40% (from 0.23 to 0.14) on the FLORES-200 benchmark. Our method can expand language identification coverage into low-resource languages by relying solely on systemic misprediction patterns, bypassing the need to retrain large models from scratch.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 96 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

37 of 96 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  4. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  6. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  7. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  8. 2019Massively Multilingual Neural Machine Translationcited by 246
  9. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  10. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  11. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  12. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
25 more
  1. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  2. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  3. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  4. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  5. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 74
  6. 2020Balancing Training for Multilingual Neural Machine Translationcited by 71
  7. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  8. 2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
  9. 2018Automatic Language Identification in Texts: A Surveycited by 40
  10. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 38
  11. 2022Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Taskscited by 28
  12. 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24
  13. 2022Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languagescited by 20
  14. 2020Rediscovering the Slavic Continuum in Representations Emerging from Neural Models of Spoken Language Identificationcited by 14
  15. 2020A Comprehensive Survey of Multilingual Neural Machine Translationcited by 12
  16. 2021Comparing Approaches to Dravidian Language Identificationcited by 11
  17. 2020Discriminating Between Similar Nordic Languagescited by 10
  18. 2021Adaptive Sparse Transformer for Multilingual Translationcited by 8
  19. 2022Better Quality Estimation for Low Resource Corpus Miningcited by 8
  20. 2017LIDE: Language Identification from Text Documentscited by 3
  21. 2018Discriminating between Indo-Aryan Languages Using SVM Ensemblescited by 2
  22. 2020Combining Deep Learning and String Kernels for the Localization of Swiss German Tweetscited by 2
  23. 2022Annotating Norwegian Language Varieties on Twitter for Part-of-Speechcited by 2
  24. 2020Vulgaris: Analysis of a Corpus for Middle-Age Varieties of Italian Languagecited by 1
  25. 2021UnibucKernel: Geolocating Swiss German Jodels Using Ensemble Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2026OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Reportcited by 4
  2. 2024Script-Agnostic Language Identificationcited by 0
  3. 2025Developing a Mixed-Methods Pipeline for Community-Oriented Digitization of Kwak'wala Legacy Textscited by 0
  4. 2025ILID: Native Script Language Identification for Indian Languagescited by 0
  5. 2026AfroScope: A Framework for Studying the Linguistic Landscape of Africacited by 0
  6. 2026How Should We Model the Probability of a Language?cited by 0