# CINO: A Chinese Minority Pre-trained Language Model

arXiv: https://arxiv.org/abs/2202.13558 · HTML: https://mlsanity.com/paper/2202.13558/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 28 Feb 2022.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks](https://mlsanity.com/paper/1909.00964/) (2019)
- [ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora](https://mlsanity.com/paper/2012.15674/) (2020)
- [IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP](https://mlsanity.com/paper/2011.00677/) (2020)
- [VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2010.16046/) (2020)
- [Multilingual BERT Post-Pretraining Alignment](https://mlsanity.com/paper/2010.12547/) (2020)

## Built on by (most cited first, 11 of 11)

- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Facilitating large language model Russian adaptation with Learned Embedding Propagation](https://mlsanity.com/paper/2412.21140/) (2024)
- [Pay Attention to the Robustness of Chinese Minority Language Models! Syllable-level Textual Adversarial Attack on Tibetan Script](https://mlsanity.com/paper/2412.02323/) (2024)
- [TiBERT: Tibetan Pre-trained Language Model](https://mlsanity.com/paper/2205.07303/) (2022)
- [MediaGPT : A Large Language Model For Chinese Media](https://mlsanity.com/paper/2307.10930/) (2023)
- [TSCheater: Generating High-Quality Tibetan Adversarial Texts via Visual Similarity](https://mlsanity.com/paper/2412.02371/) (2024)
- [MiLMo:Minority Multilingual Pre-trained Language Model](https://mlsanity.com/paper/2212.01779/) (2022)
- [PEFTT: Parameter-Efficient Fine-Tuning for low-resource Tibetan pre-trained language models](https://mlsanity.com/paper/2309.12109/) (2023)
- [Vocabulary Customization for Efficient Domain-Specific LLM Deployment](https://mlsanity.com/paper/2509.26124/) (2025)
- [Leveraging Retrieval-Augmented Generation for Culturally Inclusive Hakka Chatbots: Design Insights and User Perceptions](https://mlsanity.com/paper/2410.15572/) (2024)
- [AttnBoost: Retail Supply Chain Sales Insights via Gradient Boosting Perspective](https://mlsanity.com/paper/2509.10506/) (2025)

## Abstract (arXiv)

Multilingual pre-trained language models have shown impressive performance on cross-lingual tasks. It greatly facilitates the applications of natural language processing on low-resource languages. However, there are still some languages that the current multilingual models do not perform well on. In this paper, we propose CINO (Chinese Minority Pre-trained Language Model), a multilingual pre-trained language model for Chinese minority languages. It covers Standard Chinese, Yue Chinese, and six other ethnic minority languages. To evaluate the cross-lingual ability of the multilingual model on ethnic minority languages, we collect documents from Wikipedia and news websites, and construct two text classification datasets, WCM (Wiki-Chinese-Minority) and CMNews (Chinese-Minority-News). We show that CINO notably outperforms the baselines on various classification tasks. The CINO model and the datasets are publicly available at http://cino.hfl-rc.com.