mlsanityData to 14 Oct 2026

ConLID: Supervised Contrastive Learning for Low-Resource Language Identification

arXiv 2506.15304 ↗First posted 18 Jun 2025cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Language identification (LID) is a critical step in curating multilingual LLM pretraining corpora from web crawls. While many studies on LID model training focus on collecting diverse training data to improve performance, low-resource languages -- often limited to single-domain data, such as the Bible -- continue to perform poorly. To resolve these imbalance and bias issues, we propose a novel supervised contrastive learning (SCL) approach to learn domain-invariant representations for low-resource languages. We show that our approach improves LID performance on out-of-domain data for low-resource languages by 3.2 percentage points, while maintaining its performance for the high-resource languages.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

20 of 35 references.

References · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  8. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 276
  9. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  10. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  11. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
  12. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
8 more
  1. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  2. 2023An Open Dataset and Model for Language Identificationcited by 50
  3. 2020Mapping Languages: The Corpus of Global Language Usecited by 42
  4. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 39
  5. 2022Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resourcescited by 25
  6. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 16
  7. 2024Geographically-Informed Language Identificationcited by 6
  8. 2024From N-grams to Pre-trained Multilingual Models For Language Identificationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Datacited by 6
  2. 2025DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switchingcited by 3
  3. 2026What Language is This? Ask Your Tokenizercited by 2
  4. 2026Robust Language Identification for Romansh Varietiescited by 2
  5. 2026AfroScope: A Framework for Studying the Linguistic Landscape of Africacited by 0
  6. 2026MixLID: Token-Level Language Identification in Mixed-Language Textcited by 0