Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages
Not on the map. Why?
Abstract (arXiv)
Named-entity recognition (NER) in low-resource languages is usually tackled by finetuning very large multilingual LMs, an option that is often infeasible in memory- or latency-constrained settings. We ask whether small decoder LMs can be pretrained so that they adapt quickly and transfer zero-shot to languages unseen during pretraining. To this end we replace part of the autoregressive objective with first-order model-agnostic meta-learning (MAML). Tagalog and Cebuano are typologically similar yet structurally different in their actor/non-actor voice systems, and hence serve as a challenging test-bed. Across four model sizes (11 M - 570 M) MAML lifts zero-shot micro-F1 by 2-6 pp under head-only tuning and 1-3 pp after full tuning, while cutting convergence time by up to 8%. Gains are largest for single-token person entities that co-occur with Tagalog case particles si/ni, highlighting the importance of surface anchors.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 51 references that are in the map. First posted on arXiv 2 Sep 2025.
Builds on
21 of 51 references.
References · 21
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2020GLU Variants Improve Transformercited by 1,804
- 2019Root Mean Square Layer Normalizationcited by 1,480
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
- 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
- 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
- 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 54
- 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
- 2021Meta-Learning to Improve Pre-Trainingcited by 27
- 2023Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmarkcited by 26
9 more
- 2019Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resourcescited by 25
- 2022Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptationcited by 14
- 2022E-NER -- An Annotated Named Entity Recognition Corpus of Legal Textcited by 13
- 2023Developing a Named Entity Recognition Dataset for Tagalogcited by 7
- 2023Is Pre-training Truly Better Than Meta-Learning?cited by 6
- 2025Batayan: A Filipino NLP benchmark for evaluating Large Language Modelscited by 5
- 2025Learning Dynamics of Meta-Learning in Small Model Pretrainingcited by 3
- 2025FilBench: Can LLMs Understand and Generate Filipino?cited by 3
- 2023CebuaNER: A New Baseline Cebuano Named Entity Recognition Modelcited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.