# Towards Lingua Franca Named Entity Recognition with BERT

arXiv: https://arxiv.org/abs/1912.01389 · HTML: https://mlsanity.com/paper/1912.01389/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2019.

## Built on by (most cited first, 11 of 11)

- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Zero-shot cross-lingual transfer language selection using linguistic similarity](https://mlsanity.com/paper/2301.13720/) (2023)
- [Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks](https://mlsanity.com/paper/2108.08375/) (2021)
- [Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/2106.00241/) (2021)
- [Cross-Lingual Named Entity Recognition Using Parallel Corpus: A New Approach Using XLM-RoBERTa Alignment](https://mlsanity.com/paper/2101.11112/) (2021)
- [Can BERT Dig It? -- Named Entity Recognition for Information Retrieval in the Archaeology Domain](https://mlsanity.com/paper/2106.07742/) (2021)
- [Target-Oriented Fine-tuning for Zero-Resource Named Entity Recognition](https://mlsanity.com/paper/2107.10523/) (2021)
- [Evaluating Generalizability of Fine-Tuned Models for Fake News Detection](https://mlsanity.com/paper/2205.07154/) (2022)
- [ProKD: An Unsupervised Prototypical Knowledge Distillation Network for Zero-Resource Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/2301.08855/) (2023)
- [Extracting Impact Model Narratives from Social Services' Text](https://mlsanity.com/paper/2204.09557/) (2022)
- [DA-Net: A Disentangled and Adaptive Network for Multi-Source Cross-Lingual Transfer Learning](https://mlsanity.com/paper/2403.04158/) (2024)

## Abstract (arXiv)

Information extraction is an important task in NLP, enabling the automatic extraction of data for relational database filling. Historically, research and data was produced for English text, followed in subsequent years by datasets in Arabic, Chinese (ACE/OntoNotes), Dutch, Spanish, German (CoNLL evaluations), and many others. The natural tendency has been to treat each language as a different dataset and build optimized models for each. In this paper we investigate a single Named Entity Recognition model, based on a multilingual BERT, that is trained jointly on many languages simultaneously, and is able to decode these languages with better accuracy than models trained only on one language. To improve the initial model, we study the use of regularization strategies such as multitask learning and partial gradient updates. In addition to being a single model that can tackle multiple languages (including code switch), the model could be used to make zero-shot predictions on a new language, even ones for which training data is not available, out of the box. The results show that this model not only performs competitively with monolingual models, but it also achieves state-of-the-art results on the CoNLL02 Dutch and Spanish datasets, OntoNotes Arabic and Chinese datasets. Moreover, it performs reasonably well on unseen languages, achieving state-of-the-art for zero-shot on three CoNLL languages.