# Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages

arXiv: https://arxiv.org/abs/2509.02160 · HTML: https://mlsanity.com/paper/2509.02160/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 2 Sep 2025.

## Builds on (most cited first, 21 of 21)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Paloma: A Benchmark for Evaluating Language Model Fit](https://mlsanity.com/paper/2312.10523/) (2023)
- [Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks](https://mlsanity.com/paper/2009.08445/) (2020)
- [Meta-Learning to Improve Pre-Training](https://mlsanity.com/paper/2111.01754/) (2021)
- [Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark](https://mlsanity.com/paper/2311.09122/) (2023)
- [Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resources](https://mlsanity.com/paper/1911.06161/) (2019)
- [Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation](https://mlsanity.com/paper/2207.03509/) (2022)
- [E-NER -- An Annotated Named Entity Recognition Corpus of Legal Text](https://mlsanity.com/paper/2212.09306/) (2022)
- [Developing a Named Entity Recognition Dataset for Tagalog](https://mlsanity.com/paper/2311.07161/) (2023)
- [Is Pre-training Truly Better Than Meta-Learning?](https://mlsanity.com/paper/2306.13841/) (2023)
- [Batayan: A Filipino NLP benchmark for evaluating Large Language Models](https://mlsanity.com/paper/2502.14911/) (2025)
- [Learning Dynamics of Meta-Learning in Small Model Pretraining](https://mlsanity.com/paper/2508.02189/) (2025)
- [FilBench: Can LLMs Understand and Generate Filipino?](https://mlsanity.com/paper/2508.03523/) (2025)
- [CebuaNER: A New Baseline Cebuano Named Entity Recognition Model](https://mlsanity.com/paper/2310.00679/) (2023)

## Built on by (most cited first, 3 of 3)

- [Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research](https://mlsanity.com/paper/2509.16413/) (2025)
- [Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark](https://mlsanity.com/paper/2604.12744/) (2026)
- [Beetle: A Bilingual Model Suite for Modelling Second-Language Processing](https://mlsanity.com/paper/2609.22633/) (2026)

## Abstract (arXiv)

Named-entity recognition (NER) in low-resource languages is usually tackled by finetuning very large multilingual LMs, an option that is often infeasible in memory- or latency-constrained settings. We ask whether small decoder LMs can be pretrained so that they adapt quickly and transfer zero-shot to languages unseen during pretraining. To this end we replace part of the autoregressive objective with first-order model-agnostic meta-learning (MAML). Tagalog and Cebuano are typologically similar yet structurally different in their actor/non-actor voice systems, and hence serve as a challenging test-bed. Across four model sizes (11 M - 570 M) MAML lifts zero-shot micro-F1 by 2-6 pp under head-only tuning and 1-3 pp after full tuning, while cutting convergence time by up to 8%. Gains are largest for single-token person entities that co-occur with Tagalog case particles si/ni, highlighting the importance of surface anchors.