# Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation

arXiv: https://arxiv.org/abs/1906.01569 · HTML: https://mlsanity.com/paper/1906.01569/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2019.

## Built on by (most cited first, 14 of 14)

- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [Substructure Substitution: Structured Data Augmentation for NLP](https://mlsanity.com/paper/2101.00411/) (2021)
- [Why Can't Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversity](https://mlsanity.com/paper/2302.06488/) (2023)
- [To Augment or Not to Augment? A Comparative Study on Text Augmentation Techniques for Low-Resource NLP](https://mlsanity.com/paper/2111.09618/) (2021)
- [Semi-automatic Generation of Multilingual Datasets for Stance Detection in Twitter](https://mlsanity.com/paper/2101.11978/) (2021)
- [Empirical Error Modeling Improves Robustness of Noisy Neural Sequence Labeling](https://mlsanity.com/paper/2105.11872/) (2021)
- [Average Is Not Enough: Caveats of Multilingual Evaluation](https://mlsanity.com/paper/2301.01269/) (2023)
- [LOCORE: Image Re-ranking with Long-Context Sequence Modeling](https://mlsanity.com/paper/2503.21772/) (2025)
- [Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Set](https://mlsanity.com/paper/2503.10515/) (2025)
- [Optimal Size-Performance Tradeoffs: Weighing PoS Tagger Models](https://mlsanity.com/paper/2104.07951/) (2021)
- [Multinational Address Parsing: A Zero-Shot Evaluation](https://mlsanity.com/paper/2112.04008/) (2021)
- [Progressive Sentiment Analysis for Code-Switched Text Data](https://mlsanity.com/paper/2210.14380/) (2022)
- [Deepparse : An Extendable, and Fine-Tunable State-Of-The-Art Library for Parsing Multinational Street Addresses](https://mlsanity.com/paper/2311.11846/) (2023)
- [GWPT: A Green Word-Embedding-based POS Tagger](https://mlsanity.com/paper/2401.07475/) (2024)

## Abstract (arXiv)

Pretrained contextual and non-contextual subword embeddings have become available in over 250 languages, allowing massively multilingual NLP. However, while there is no dearth of pretrained embeddings, the distinct lack of systematic evaluations makes it difficult for practitioners to choose between them. In this work, we conduct an extensive evaluation comparing non-contextual subword embeddings, namely FastText and BPEmb, and a contextual representation method, namely BERT, on multilingual named entity recognition and part-of-speech tagging. We find that overall, a combination of BERT, BPEmb, and character representations works best across languages and tasks. A more detailed analysis reveals different strengths and weaknesses: Multilingual BERT performs well in medium- to high-resource languages, but is outperformed by non-contextual subword embeddings in a low-resource setting.