# Exploring Cross-Lingual Knowledge Transfer via Transliteration-Based MLM Fine-Tuning for Critically Low-resource Chakma Language

arXiv: https://arxiv.org/abs/2510.09032 · HTML: https://mlsanity.com/paper/2510.09032/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla](https://mlsanity.com/paper/2101.00204/) (2021)
- [Extending Multilingual BERT to Low-Resource Languages](https://mlsanity.com/paper/2004.13640/) (2020)
- [How to Adapt Your Pretrained Multilingual Model to 1600 Languages](https://mlsanity.com/paper/2106.02124/) (2021)
- [How Does Quantization Affect Multilingual LLMs?](https://mlsanity.com/paper/2407.03211/) (2024)
- [Tokenization Matters: Improving Zero-Shot NER for Indic Languages](https://mlsanity.com/paper/2504.16977/) (2025)
- [Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya](https://mlsanity.com/paper/2006.07698/) (2020)
- [RoundTripOCR: A Data Generation Technique for Enhancing Post-OCR Error Correction in Low-Resource Devanagari Languages](https://mlsanity.com/paper/2412.15248/) (2024)
- [Linguistic Entity Masking to Improve Cross-Lingual Representation of Multilingual Language Models for Low-Resource Languages](https://mlsanity.com/paper/2501.05700/) (2025)

## Abstract (arXiv)

As an Indo-Aryan language with limited available data, Chakma remains largely underrepresented in language models. In this work, we introduce a novel corpus of contextually coherent Bangla-transliterated Chakma, curated from Chakma literature, and validated by native speakers. Using this dataset, we fine-tune six encoder-based transformer models, including multilingual (mBERT, XLM-RoBERTa, DistilBERT), regional (BanglaBERT, IndicBERT), and monolingual English (DeBERTaV3) variants on masked language modeling (MLM) tasks. Our experiments show that fine-tuned multilingual models outperform their pre-trained counterparts when adapted to Bangla-transliterated Chakma, achieving up to 73.54% token accuracy and a perplexity as low as 2.90. Our analysis further highlights the impact of data quality on model performance and shows the limitations of OCR pipelines for morphologically rich Indic scripts. Our research demonstrates that Bangla-transliterated Chakma can be very effective for transfer learning for Chakma language, and we release our dataset to encourage further research on multilingual language modeling for low-resource languages.