# Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss

arXiv: https://arxiv.org/abs/1604.05529 · HTML: https://mlsanity.com/paper/1604.05529/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2016.

## Built on by (most cited first, 30 of 30)

- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [A Systematic Review of Reproducibility Research in Natural Language Processing](https://mlsanity.com/paper/2103.07929/) (2021)
- [Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning](https://mlsanity.com/paper/2104.08676/) (2021)
- [Conformal prediction for text infilling and part-of-speech prediction](https://mlsanity.com/paper/2111.02592/) (2021)
- [Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Training](https://mlsanity.com/paper/2109.02284/) (2021)
- [Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users](https://mlsanity.com/paper/2205.03018/) (2022)
- [De-identification of Privacy-related Entities in Job Postings](https://mlsanity.com/paper/2105.11223/) (2021)
- [AnnoBERT: Effectively Representing Multiple Annotators' Label Choices to Improve Hate Speech Detection](https://mlsanity.com/paper/2212.10405/) (2022)
- [Restoring Hebrew Diacritics Without a Dictionary](https://mlsanity.com/paper/2105.05209/) (2021)
- [Disentangling Uncertainty in Machine Translation Evaluation](https://mlsanity.com/paper/2204.06546/) (2022)
- [Integrating Approaches to Word Representation](https://mlsanity.com/paper/2109.04876/) (2021)
- [Deep Learning Transformer Architecture for Named Entity Recognition on Low Resourced Languages: State of the art results](https://mlsanity.com/paper/2111.00830/) (2021)
- [Towards a Deep Multi-layered Dialectal Language Analysis: A Case Study of African-American English](https://mlsanity.com/paper/2206.08978/) (2022)
- [Multitask Learning for Class-Imbalanced Discourse Classification](https://mlsanity.com/paper/2101.00389/) (2021)
- [Optimal Size-Performance Tradeoffs: Weighing PoS Tagger Models](https://mlsanity.com/paper/2104.07951/) (2021)
- [Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units](https://mlsanity.com/paper/2106.04935/) (2021)
- [Thread Detection and Response Generation using Transformers with Prompt Optimisation](https://mlsanity.com/paper/2403.05931/) (2024)
- [Coloring the Black Box: What Synesthesia Tells Us about Character Embeddings](https://mlsanity.com/paper/2101.10565/) (2021)
- [Benchmarking zero-shot and few-shot approaches for tokenization, tagging, and dependency parsing of Tagalog text](https://mlsanity.com/paper/2208.01814/) (2022)
- [High-Resource Methodological Bias in Low-Resource Investigations](https://mlsanity.com/paper/2211.07534/) (2022)
- [Heidelberg-Boston @ SIGTYP 2024 Shared Task: Enhancing Low-Resource Language Analysis With Character-Aware Hierarchical Transformers](https://mlsanity.com/paper/2405.20145/) (2024)
- [Indirect Question Answering in English, German and Bavarian: A Challenging Task for High- and Low-Resource Languages Alike](https://mlsanity.com/paper/2603.15130/) (2026)
- [Computational linguistics and Natural Language Processing](https://mlsanity.com/paper/2206.07026/) (2022)
- [Part-of-Speech Tagging of Odia Language Using statistical and Deep Learning-Based Approaches](https://mlsanity.com/paper/2207.03256/) (2022)
- [Smart Agriculture : A Novel Multilevel Approach for Agricultural Risk Assessment over Unstructured Data](https://mlsanity.com/paper/2211.12515/) (2022)
- [Cross-Domain Evaluation of POS Taggers: From Wall Street Journal to Fandom Wiki](https://mlsanity.com/paper/2304.13989/) (2023)
- [A Global Context Mechanism for Sequence Labeling](https://mlsanity.com/paper/2305.19928/) (2023)
- [When Do Discourse Markers Affect Computational Sentence Understanding?](https://mlsanity.com/paper/2309.00368/) (2023)
- [An Innovative CGL-MHA Model for Sarcasm Sentiment Recognition Using the MindSpore Framework](https://mlsanity.com/paper/2411.01264/) (2024)
- [Refining Syntactic Distinctions Using Decision Trees: A Paper on Postnominal 'That' in Complement vs. Relative Clauses](https://mlsanity.com/paper/2509.14261/) (2025)

## Abstract (arXiv)

Bidirectional long short-term memory (bi-LSTM) networks have recently proven successful for various NLP sequence modeling tasks, but little is known about their reliance to input representations, target languages, data set size, and label noise. We address these issues and evaluate bi-LSTMs with word, character, and unicode byte embeddings for POS tagging. We compare bi-LSTMs to traditional POS taggers across languages and data sizes. We also present a novel bi-LSTM model, which combines the POS tagging loss function with an auxiliary loss function that accounts for rare words. The model obtains state-of-the-art performance across 22 languages, and works especially well for morphologically complex languages. Our analysis suggests that bi-LSTMs are less sensitive to training data size and label corruptions (at small noise levels) than previously assumed.