# An Empirical Study of Pre-trained Transformers for Arabic Information Extraction

arXiv: https://arxiv.org/abs/2004.14519 · HTML: https://mlsanity.com/paper/2004.14519/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 3 of 3)

- [Frustratingly Easy Label Projection for Cross-lingual Transfer](https://mlsanity.com/paper/2211.15613/) (2022)
- [DaN+: Danish Nested Named Entities and Lexical Normalization](https://mlsanity.com/paper/2105.11301/) (2021)
- [Cross-lingual hate speech detection based on multilingual domain-specific word embeddings](https://mlsanity.com/paper/2104.14728/) (2021)

## Abstract (arXiv)

Multilingual pre-trained Transformers, such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020a), have been shown to enable the effective cross-lingual zero-shot transfer. However, their performance on Arabic information extraction (IE) tasks is not very well studied. In this paper, we pre-train a customized bilingual BERT, dubbed GigaBERT, that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning. We study GigaBERT's effectiveness on zero-short transfer across four IE tasks: named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction. Our best model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT (Antoun et al., 2020) in both the supervised and zero-shot transfer settings. We have made our pre-trained models publicly available at https://github.com/lanwuwei/GigaBERT.