mlsanityData to 14 Oct 2026

Exploring Cross-Lingual Knowledge Transfer via Transliteration-Based MLM Fine-Tuning for Critically Low-resource Chakma Language

arXiv 2510.09032 ↗First posted 10 Oct 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

As an Indo-Aryan language with limited available data, Chakma remains largely underrepresented in language models. In this work, we introduce a novel corpus of contextually coherent Bangla-transliterated Chakma, curated from Chakma literature, and validated by native speakers. Using this dataset, we fine-tune six encoder-based transformer models, including multilingual (mBERT, XLM-RoBERTa, DistilBERT), regional (BanglaBERT, IndicBERT), and monolingual English (DeBERTaV3) variants on masked language modeling (MLM) tasks. Our experiments show that fine-tuned multilingual models outperform their pre-trained counterparts when adapted to Bangla-transliterated Chakma, achieving up to 73.54% token accuracy and a perplexity as low as 2.90. Our analysis further highlights the impact of data quality on model performance and shows the limitations of OCR pipelines for morphologically rich Indic scripts. Our research demonstrates that Bangla-transliterated Chakma can be very effective for transfer learning for Chakma language, and we release our dataset to encourage further research on multilingual language modeling for low-resource languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

22 of 42 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  9. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  10. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  11. 2019How multilingual is Multilingual BERT?cited by 645
  12. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
10 more
  1. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  2. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  3. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 125
  4. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  5. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  6. 2024How Does Quantization Affect Multilingual LLMs?cited by 43
  7. 2025Tokenization Matters: Improving Zero-Shot NER for Indic Languagescited by 12
  8. 2020Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinyacited by 8
  9. 2024RoundTripOCR: A Data Generation Technique for Enhancing Post-OCR Error Correction in Low-Resource Devanagari Languagescited by 3
  10. 2025Linguistic Entity Masking to Improve Cross-Lingual Representation of Multilingual Language Models for Low-Resource Languagescited by 1

Built on by

Not cited yet.