# Optimal Size-Performance Tradeoffs: Weighing PoS Tagger Models

arXiv: https://arxiv.org/abs/2104.07951 · HTML: https://mlsanity.com/paper/2104.07951/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages](https://mlsanity.com/paper/1710.02187/) (2017)
- [A Comparison of LSTM and BERT for Small Corpus](https://mlsanity.com/paper/2009.05451/) (2020)
- [Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss](https://mlsanity.com/paper/1604.05529/) (2016)
- [Robust Multilingual Part-of-Speech Tagging via Adversarial Training](https://mlsanity.com/paper/1711.04903/) (2017)
- [Polyglot: Distributed Word Representations for Multilingual NLP](https://mlsanity.com/paper/1307.1662/) (2013)
- [Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings](https://mlsanity.com/paper/1805.08237/) (2018)
- [Controlling Information Capacity of Binary Neural Network](https://mlsanity.com/paper/2008.01438/) (2020)
- [Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation](https://mlsanity.com/paper/1906.01569/) (2019)
- [Near-lossless Binarization of Word Embeddings](https://mlsanity.com/paper/1803.09065/) (2018)
- [Embedding Compression with Isotropic Iterative Quantization](https://mlsanity.com/paper/2001.05314/) (2020)
- [Cross-Lingual Morphological Tagging for Low-Resource Languages](https://mlsanity.com/paper/1606.04279/) (2016)
- [Learning Recurrent Binary/Ternary Weights](https://mlsanity.com/paper/1809.11086/) (2018)
- [A Grounded Unsupervised Universal Part-of-Speech Tagger for Low-Resource Languages](https://mlsanity.com/paper/1904.05426/) (2019)
- [Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings](https://mlsanity.com/paper/2002.00733/) (2020)
- [Compressing Neural Language Models by Sparse Word Representations](https://mlsanity.com/paper/1610.03950/) (2016)
- [Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing](https://mlsanity.com/paper/2010.03088/) (2020)

## Built on by (most cited first, 2 of 2)

- [A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021](https://mlsanity.com/paper/2106.04216/) (2021)
- [Exploring Graph Representations of Logical Forms for Language Modeling](https://mlsanity.com/paper/2505.14523/) (2025)

## Abstract (arXiv)

Improvement in machine learning-based NLP performance are often presented with bigger models and more complex code. This presents a trade-off: better scores come at the cost of larger tools; bigger models tend to require more during training and inference time. We present multiple methods for measuring the size of a model, and for comparing this with the model's performance. In a case study over part-of-speech tagging, we then apply these techniques to taggers for eight languages and present a novel analysis identifying which taggers are size-performance optimal. Results indicate that some classical taggers place on the size-performance skyline across languages. Further, although the deep models have highest performance for multiple scores, it is often not the most complex of these that reach peak performance.