mlsanityData to 7 Oct 2026

Code-Switched Language Identification is Harder Than You Think

arXiv 2402.01505 ↗First posted 2 Feb 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Code switching (CS) is a very common phenomenon in written and spoken communication but one that is handled poorly by many natural language processing applications. Looking to the application of building CS corpora, we explore CS language identification (LID) for corpus building. We make the task more realistic by scaling it to more languages and considering models with simpler architectures for faster inference. We also reformulate the task as a sentence-level multi-label tagging problem to make it more tractable. Having defined the task, we investigate three reasonable models for this task and define metrics which better reflect desired performance. We present empirical evidence that no current approach is adequate and finally provide recommendations for future work in this area.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 19 of its 33 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

19 of 33 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  3. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  4. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  5. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  6. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 101
  7. 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 92
  8. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  9. 2019A Survey of Code-switched Speech and Language Processingcited by 73
  10. 2023A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologiescited by 59
  11. 2023An Open Dataset and Model for Language Identificationcited by 49
  12. 2021ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversationcited by 43
7 more
  1. 2020Mapping Languages: The Corpus of Global Language Usecited by 42
  2. 2018Automatic Language Identification in Texts: A Surveycited by 40
  3. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 38
  4. 2022The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challengescited by 35
  5. 2019Overview for the Second Shared Task on Language Identification in Code-Switched Datacited by 33
  6. 2017LanideNN: Multilingual Language Identification on Character Windowcited by 10
  7. 2022Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONEScited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switchingcited by 3
  2. 2025SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMscited by 2
  3. 2026Code-Switching Spoken Language Identification as Multi-Label Set Predictioncited by 0