No Language Data Left Behind: A Comparative Study of CJK Language Datasets in the Hugging Face Ecosystem
Not on the map. Why?
Abstract (arXiv)
Recent advances in Natural Language Processing (NLP) have underscored the crucial role of high-quality datasets in building large language models (LLMs). However, while extensive resources and analyses exist for English, the landscape for East Asian languages - particularly Chinese, Japanese, and Korean (CJK) - remains fragmented and underexplored, despite these languages together serving over 1.6 billion speakers. To address this gap, we investigate the HuggingFace ecosystem from a cross-linguistic perspective, focusing on how cultural norms, research environments, and institutional practices shape dataset availability and quality. Drawing on more than 3,300 datasets, we employ quantitative and qualitative methods to examine how these factors drive distinct creation and curation patterns across Chinese, Japanese, and Korean NLP communities. Our findings highlight the large-scale and often institution-driven nature of Chinese datasets, grassroots community-led development in Korean NLP, and an entertainment- and subculture-focused emphasis on Japanese collections. By uncovering these patterns, we reveal practical strategies for enhancing dataset documentation, licensing clarity, and cross-lingual resource sharing - ultimately guiding more effective and culturally attuned LLM development in East Asia. We conclude by discussing best practices for future dataset curation and collaboration, aiming to strengthen resource development across all three languages.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 6 Jul 2025.
Builds on
19 of 31 references.
References · 19
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
- 2018Datasheets for Datasetscited by 1,386
- 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
- 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
- 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
- 2021Datasets: A Community Library for Natural Language Processingcited by 530
- 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
7 more
- 2021KLUE: Korean Language Understanding Evaluationcited by 114
- 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
- 2024Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on Hugging Facecited by 27
- 2023TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processescited by 19
- 2022K-MHaS: A Multi-label Hate Speech Detection Dataset in Korean Online News Commentcited by 13
- 2020Open Korean Corpora: A Practical Reportcited by 6
- 2023KoMultiText: Large-Scale Korean Text Dataset for Classifying Biased Speech in Real-World Online Servicescited by 3
Built on by
Not cited yet.