# AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese

arXiv: https://arxiv.org/abs/2305.13641 · HTML: https://mlsanity.com/paper/2305.13641/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 51 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [GottBERT: a pure German Language Model](https://mlsanity.com/paper/2012.02110/) (2020)
- [PMIndia -- A Collection of Parallel Corpora of Languages of India](https://mlsanity.com/paper/2001.09907/) (2020)
- [KinyaBERT: a Morphology-aware Kinyarwanda Language Model](https://mlsanity.com/paper/2203.08459/) (2022)
- [Stolen Probability: A Structural Weakness of Neural Language Models](https://mlsanity.com/paper/2005.02433/) (2020)
- [EstBERT: A Pretrained Language-Specific BERT for Estonian](https://mlsanity.com/paper/2011.04784/) (2020)
- [2 * n is better than n2: Decomposing Event Coreference Resolution into Two Tractable Problems](https://mlsanity.com/paper/2305.05672/) (2023)
- [Zero-shot Neural Transfer for Cross-lingual Entity Linking](https://mlsanity.com/paper/1811.04154/) (2018)
- [AsNER -- Annotated Dataset and Baseline for Assamese Named Entity recognition](https://mlsanity.com/paper/2207.03422/) (2022)
- [Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT](https://mlsanity.com/paper/2203.09055/) (2022)
- [On the cross-lingual transferability of multilingual prototypical models across NLU tasks](https://mlsanity.com/paper/2207.09157/) (2022)

## Built on by (most cited first, 4 of 4)

- [Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges](https://mlsanity.com/paper/2509.11570/) (2025)
- [Dynamic Epistemic Friction in Dialogue](https://mlsanity.com/paper/2506.10934/) (2025)
- [Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language](https://mlsanity.com/paper/2410.03718/) (2024)
- [Okay, Let's Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation](https://mlsanity.com/paper/2404.03196/) (2024)

## Abstract (arXiv)

Despite their successes in NLP, Transformer-based language models still require extensive computing resources and suffer in low-resource or low-compute settings. In this paper, we present AxomiyaBERTa, a novel BERT model for Assamese, a morphologically-rich low-resource language (LRL) of Eastern India. AxomiyaBERTa is trained only on the masked language modeling (MLM) task, without the typical additional next sentence prediction (NSP) objective, and our results show that in resource-scarce settings for very low-resource languages like Assamese, MLM alone can be successfully leveraged for a range of tasks. AxomiyaBERTa achieves SOTA on token-level tasks like Named Entity Recognition and also performs well on "longer-context" tasks like Cloze-style QA and Wiki Title Prediction, with the assistance of a novel embedding disperser and phonological signals respectively. Moreover, we show that AxomiyaBERTa can leverage phonological signals for even more challenging tasks, such as a novel cross-document coreference task on a translated version of the ECB+ corpus, where we present a new SOTA result for an LRL. Our source code and evaluation scripts may be found at https://github.com/csu-signal/axomiyaberta.