# Which Encoding is the Best for Text Classification in Chinese, English, Japanese and Korean?

arXiv: https://arxiv.org/abs/1708.02657 · HTML: https://mlsanity.com/paper/1708.02657/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2017.

## Built on by (most cited first, 10 of 10)

- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information](https://mlsanity.com/paper/2106.16038/) (2021)
- [InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling](https://mlsanity.com/paper/2304.03544/) (2023)
- [Towards the TopMost: A Topic Modeling System Toolkit](https://mlsanity.com/paper/2309.06908/) (2023)
- [Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis](https://mlsanity.com/paper/2305.13654/) (2023)
- [Expanding Scope: Adapting English Adversarial Attacks to Chinese](https://mlsanity.com/paper/2306.04874/) (2023)
- [Bit-level BPE: Below the byte boundary](https://mlsanity.com/paper/2506.07541/) (2025)
- [Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training](https://mlsanity.com/paper/2305.18760/) (2023)
- [byteSteady: Fast Classification Using Byte-Level n-Gram Embeddings](https://mlsanity.com/paper/2106.13302/) (2021)
- [Chinese-Jev: Bringing System One Model to Chinese-Language Tasks](https://mlsanity.com/paper/2609.36965/) (2026)

## Abstract (arXiv)

This article offers an empirical study on the different ways of encoding Chinese, Japanese, Korean (CJK) and English languages for text classification. Different encoding levels are studied, including UTF-8 bytes, characters, words, romanized characters and romanized words. For all encoding levels, whenever applicable, we provide comparisons with linear models, fastText and convolutional networks. For convolutional networks, we compare between encoding mechanisms using character glyph images, one-hot (or one-of-n) encoding, and embedding. In total there are 473 models, using 14 large-scale text classification datasets in 4 languages including Chinese, English, Japanese and Korean. Some conclusions from these results include that byte-level one-hot encoding based on UTF-8 consistently produces competitive results for convolutional networks, that word-level n-grams linear models are competitive even without perfect word segmentation, and that fastText provides the best result using character-level n-gram encoding but can overfit when the features are overly rich.