# What Matters When Building Universal Multilingual Named Entity Recognition Models?

arXiv: https://arxiv.org/abs/2601.06347 · HTML: https://mlsanity.com/paper/2601.06347/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 9 Jan 2026.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [Rethinking embedding coupling in pre-trained language models](https://mlsanity.com/paper/2010.12821/) (2020)
- [Universal Information Extraction as Unified Semantic Matching](https://mlsanity.com/paper/2301.03282/) (2023)
- [LMDX: Language Model-based Document Information Extraction and Localization](https://mlsanity.com/paper/2309.10952/) (2023)
- [NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data](https://mlsanity.com/paper/2402.15343/) (2024)
- [Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark](https://mlsanity.com/paper/2311.09122/) (2023)
- [Large Language Models for Biomedical Knowledge Graph Construction: Information extraction from EMR notes](https://mlsanity.com/paper/2301.12473/) (2023)
- [What Matters for Neural Cross-Lingual Named Entity Recognition: An Empirical Analysis](https://mlsanity.com/paper/1909.03598/) (2019)
- [NER Retriever: Zero-Shot Named Entity Retrieval with Type-Aware Embeddings](https://mlsanity.com/paper/2509.04011/) (2025)

## Abstract (arXiv)

Recent progress in universal multilingual named entity recognition (NER) has been driven by multilingual transformer models, task-specific architectures, custom loss functions, and large-scale training datasets. However, despite substantial prior work, we find that many design decisions for such models are made without systematic justification, with individual components evaluated only in combination rather than in isolation. We argue that this impedes progress in the field by making it difficult to identify which choices improve multilingual generalization. In this work, we conduct an extensive empirical evaluation on transformer backbones, architectures, training objectives, data composition, and threshold selection for zero-shot, multilingual NER models. Building on these findings, we present Otter, a small encoder model that achieves consistent improvements over strong multilingual NER baselines, outperforming similarly sized models by 5.7 percentage points in F1. Further, it remains within 1.4 points of Qwen3-32B and 5.4 points below Gemma3-27B, while being roughly 90x smaller and one to two orders of magnitude faster at inference.