mlsanityData to 14 Oct 2026

Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation

arXiv 2310.20470 ↗First posted 31 Oct 2023cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Multilingualism is widespread around the world and code-switching (CSW) is a common practice among different language pairs/tuples across locations and regions. However, there is still not much progress in building successful CSW systems, despite the recent advances in Massive Multilingual Language Models (MMLMs). We investigate the reasons behind this setback through a critical study about the existing CSW data sets (68) across language pairs in terms of the collection and preparation (e.g. transcription and annotation) stages. This in-depth analysis reveals that a) most CSW data involves English ignoring other language pairs/tuples b) there are flaws in terms of representativeness in data collection and preparation stages due to ignoring the location based, socio-demographic and register variation in CSW. In addition, lack of clarity on the data selection and filtering stages shadow the representativeness of CSW data sets. We conclude by providing a short check-list to improve the representativeness for forthcoming studies involving CSW data collection and preparation.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 39 of its 124 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

39 of 124 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  7. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  8. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  9. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  10. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
  11. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 102
  12. 2019A Survey of Code-switched Speech and Language Processingcited by 75
27 more
  1. 2020Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Textcited by 67
  2. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  3. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  4. 2019Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?cited by 64
  5. 2023A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologiescited by 59
  6. 2019Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentencescited by 55
  7. 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
  8. 2021ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversationcited by 44
  9. 2021Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASRcited by 41
  10. 2020PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine Translationcited by 38
  11. 2021QASR: QCRI Aljazeera Speech Resource -- A Large Scale Annotated Arabic Speech Corpuscited by 38
  12. 2017Like trainer, like bot? Inheritance of bias in algorithmic content moderationcited by 35
  13. 2020A New Dataset for Natural Language Inference from Code-mixed Conversationscited by 29
  14. 2021From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Textcited by 29
  15. 2021Arabic Code-Switching Speech Recognition using Monolingual Datacited by 17
  16. 2020Learning to Recognize Code-switched Speech Without Forgetting Monolingual Speech Recognitioncited by 10
  17. 2017Joining Hands: Exploiting Monolingual Treebanks for Parsing of Code-mixing Datacited by 7
  18. 2017All that is English may be Hindi: Enhancing language identification through automatic ranking of likeliness of word borrowing in social mediacited by 7
  19. 2017Dual Language Models for Code Switched Speech Recognitioncited by 5
  20. 2018Intent Detection for code-mix utterances in task oriented dialogue systemscited by 5
  21. 2021GupShup: An Annotated Corpus for Abstractive Summarization of Open-Domain Code-Switched Conversationscited by 5
  22. 2022Borrowing or Codeswitching? Annotating for Finer-Grained Distinctions in Language Mixingcited by 5
  23. 2019Creating a Large Multi-Layered Representational Repository of Linguistic Code Switched Arabic Datacited by 4
  24. 2020Semi-supervised acoustic modelling for five-lingual code-switched ASR using automatically-segmented soap opera speechcited by 4
  25. 2019CodeSwitch-Reddit: Exploration of Written Multilingual Discourse in Online Discussion Forumscited by 3
  26. 2020Semi-supervised acoustic and language model training for English-isiZulu code-switched speech recognitioncited by 3
  27. 2019Large-Scale Speaker Diarization of Radio Broadcast Archivescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  2. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  3. 2024TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMscited by 24
  4. 2025PIER: A Novel Metric for Evaluating What Matters in Code-Switchingcited by 11
  5. 2025Minimal Pair-Based Evaluation of Code-Switchingcited by 8
  6. 2024Does Generative AI speak Nigerian-Pidgin?: Issues about Representativeness and Bias for Multilingualism in LLMscited by 5
  7. 2025Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalitiescited by 5
  8. 2024Grammatical Error Correction for Code-Switched Sentences by Learners of Englishcited by 2
  9. 2024Leveraging LLMs for Translating and Classifying Mental Health Datacited by 2
  10. 2025Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiencycited by 2
  11. 2024Building Multilingual Datasets for Predicting Mental Health Severity through LLMs: Prospects and Challengescited by 1
  12. 2025A Typology of Synthetic Datasets for Dialogue Processing in Clinical Contextscited by 1
4 more
  1. 2026Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languagescited by 1
  2. 2026Code Mixologist : A Practitioner's Guide to Building Code-Mixed LLMscited by 0
  3. 2026PolERo: Studying Political Evasion in Romaniancited by 0
  4. 2026TurEngMix: A Text Corpus and Benchmark for Turkish-English Code-Mixed Language Identification and Named Entity Recognitioncited by 0