mlsanityData to 14 Oct 2026

GlotScript: A Resource and Tool for Low Resource Writing System Identification

arXiv 2309.13320 ↗First posted 23 Sep 2023cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

We present GlotScript, an open resource and tool for low resource writing system identification. GlotScript-R is a resource that provides the attested writing systems for more than 7,000 languages. It is compiled by aggregating information from existing writing system resources. GlotScript-T is a writing system identification tool that covers all 161 Unicode 15.0 scripts. For an input text, it returns its script distribution where scripts are identified by ISO 15924 codes. We also present two use cases for GlotScript. First, we demonstrate that GlotScript can help cleaning multilingual corpora such as mC4 and OSCAR. Second, we analyze the tokenization of a number of language models such as GPT-4 using GlotScript and provide insights on the coverage of low resource scripts and languages by each language model. We hope that GlotScript will become a useful resource for work on low resource languages in the NLP community. GlotScript-R and GlotScript-T are available at https://github.com/cisnlp/GlotScript.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 16 of its 38 references that are in the map. First posted on arXiv 23 Sep 2023.

Builds on

16 of 38 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  10. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  11. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  12. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
4 more
  1. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  2. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
  3. 2020OCR Post Correction for Endangered Language Textscited by 31
  4. 2017Natural Language Processing with Small Feed-Forward Networkscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 4; 2026: 4.

15 papers.

Sort:

By area:

  1. 2023GlotLID: Language Identification for Low-Resource Languagescited by 39
  2. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  3. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  4. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 16
  5. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 16
  6. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  7. 2025Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Mattercited by 10
  8. 2024data2lang2vec: Data Driven Typological Features Completioncited by 4
  9. 2024TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Modelscited by 3
  10. 2024GIRT-Model: Automated Generation of Issue Report Templatescited by 2
  11. 2025SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMscited by 2
  12. 2026GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scriptscited by 2
3 more
  1. 2026QQ: A Language Metadata Toolkit for Multilingual NLPcited by 1
  2. 2025GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Modelscited by 0
  3. 2026Improving Language Identification for Code-Switched Utterances with Integer Linear Programmingcited by 0