# Subword Pooling Makes a Difference

arXiv: https://arxiv.org/abs/2102.10864 · HTML: https://mlsanity.com/paper/2102.10864/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Multilingual Constituency Parsing with Self-Attention and Pre-Training](https://mlsanity.com/paper/1812.11760/) (2018)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019)
- [Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms](https://mlsanity.com/paper/1805.09843/) (2018)

## Built on by (most cited first, 18 of 18)

- [FiNER: Financial Numeric Entity Recognition for XBRL Tagging](https://mlsanity.com/paper/2203.06482/) (2022)
- [Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5](https://mlsanity.com/paper/2410.11627/) (2024)
- [A Principled Framework for Evaluating on Typologically Diverse Languages](https://mlsanity.com/paper/2407.05022/) (2024)
- [CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitization](https://mlsanity.com/paper/2510.08829/) (2025)
- [Morphosyntactic probing of multilingual BERT models](https://mlsanity.com/paper/2306.06205/) (2023)
- [Evaluating Transferability of BERT Models on Uralic Languages](https://mlsanity.com/paper/2109.06327/) (2021)
- [Breaking the Representation Bottleneck of Chinese Characters: Neural Machine Translation with Stroke Sequence Modeling](https://mlsanity.com/paper/2211.12781/) (2022)
- [An Information Extraction Study: Take In Mind the Tokenization!](https://mlsanity.com/paper/2303.15100/) (2023)
- [A Latent-Variable Model for Intrinsic Probing](https://mlsanity.com/paper/2201.08214/) (2022)
- [Learning to Look Inside: Augmenting Token-Based Encoders with Character-Level Information](https://mlsanity.com/paper/2108.00391/) (2021)
- [UM6P-CS at SemEval-2022 Task 11: Enhancing Multilingual and Code-Mixed Complex Named Entity Recognition via Pseudo Labels using Multilingual Transformer](https://mlsanity.com/paper/2204.13515/) (2022)
- [Challenges and Opportunities in Text Generation Explainability](https://mlsanity.com/paper/2405.08468/) (2024)
- [BiVert: Bidirectional Vocabulary Evaluation using Relations for Machine Translation](https://mlsanity.com/paper/2403.03521/) (2024)
- [Impact of Subword Pooling Strategy on Cross-lingual Event Detection](https://mlsanity.com/paper/2302.11365/) (2023)
- [A Comparative Analysis of Static Word Embeddings for Hungarian](https://mlsanity.com/paper/2505.07809/) (2025)
- [Designing and Contextualising Probes for African Languages](https://mlsanity.com/paper/2505.10081/) (2025)
- [Probing Character-level Transformers for the Spanish L-shaped Morphome](https://mlsanity.com/paper/2608.03452/) (2026)
- [Strong Multilingual Privacy Tagging at Encoder Speed](https://mlsanity.com/paper/2609.38630/) (2026)

## Abstract (arXiv)

Contextual word-representations became a standard in modern natural language processing systems. These models use subword tokenization to handle large vocabularies and unknown words. Word-level usage of such systems requires a way of pooling multiple subwords that correspond to a single word. In this paper we investigate how the choice of subword pooling affects the downstream performance on three tasks: morphological probing, POS tagging and NER, in 9 typologically diverse languages. We compare these in two massively multilingual models, mBERT and XLM-RoBERTa. For morphological tasks, the widely used `choose the first subword' is the worst strategy and the best results are obtained by using attention over the subwords. For POS tagging both of these strategies perform poorly and the best choice is to use a small LSTM over the subwords. The same strategy works best for NER and we show that mBERT is better than XLM-RoBERTa in all 9 languages. We publicly release all code, data and the full result tables at urlhttps://github.com/juditacs/subword-choice.