# Unicode Normalization and Grapheme Parsing of Indic Languages

arXiv: https://arxiv.org/abs/2306.01743 · HTML: https://mlsanity.com/paper/2306.01743/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 53 references that are in the map.
- First posted on arXiv 11 May 2023.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Yara Parser: A Fast and Accurate Dependency Parser](https://mlsanity.com/paper/1503.06733/) (2015)
- [A Large Multi-Target Dataset of Common Bengali Handwritten Graphemes](https://mlsanity.com/paper/2010.00170/) (2020)
- [NumtaDB - Assembled Bengali Handwritten Digits](https://mlsanity.com/paper/1806.02452/) (2018)
- [Orthographic Syllable as basic unit for SMT between Related Languages](https://mlsanity.com/paper/1610.00634/) (2016)
- [BanglaLekha-Isolated: A Comprehensive Bangla Handwritten Character Dataset](https://mlsanity.com/paper/1703.10661/) (2017)

## Built on by (most cited first, 1 of 1)

- [DETAIL Matters: Measuring the Impact of Prompt Specificity on Reasoning in Large Language Models](https://mlsanity.com/paper/2512.02246/) (2025)

## Abstract (arXiv)

Writing systems of Indic languages have orthographic syllables, also known as complex graphemes, as unique horizontal units. A prominent feature of these languages is these complex grapheme units that comprise consonants/consonant conjuncts, vowel diacritics, and consonant diacritics, which, together make a unique Language. Unicode-based writing schemes of these languages often disregard this feature of these languages and encode words as linear sequences of Unicode characters using an intricate scheme of connector characters and font interpreters. Due to this way of using a few dozen Unicode glyphs to write thousands of different unique glyphs (complex graphemes), there are serious ambiguities that lead to malformed words. In this paper, we are proposing two libraries: i) a normalizer for normalizing inconsistencies caused by a Unicode-based encoding scheme for Indic languages and ii) a grapheme parser for Abugida text. It deconstructs words into visually distinct orthographic syllables or complex graphemes and their constituents. Our proposed normalizer is a more efficient and effective tool than the previously used IndicNLP normalizer. Moreover, our parser and normalizer are also suitable tools for general Abugida text processing as they performed well in our robust word-based and NLP experiments. We report the pipeline for the scripts of 7 languages in this work and develop the framework for the integration of more scripts.