# Neural Token Segmentation for High Token-Internal Complexity

arXiv: https://arxiv.org/abs/2203.10845 · HTML: https://mlsanity.com/paper/2203.10845/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 24 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Neural Modeling for Named Entities and Morphology (NEMO^2)](https://mlsanity.com/paper/2007.15620/) (2020)
- [Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF](https://mlsanity.com/paper/1704.01314/) (2017)
- [Universal Word Segmentation: Implementation and Interpretation](https://mlsanity.com/paper/1807.02974/) (2018)
- [Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging](https://mlsanity.com/paper/1709.00616/) (2017)

## Built on by (most cited first, 4 of 4)

- [Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All](https://mlsanity.com/paper/2211.15199/) (2022)
- [Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrew](https://mlsanity.com/paper/2311.00658/) (2023)
- [A Truly Joint Neural Architecture for Segmentation and Parsing](https://mlsanity.com/paper/2402.02564/) (2024)
- [Multilingual Sequence-to-Sequence Models for Hebrew NLP](https://mlsanity.com/paper/2212.09682/) (2022)

## Abstract (arXiv)

Tokenizing raw texts into word units is an essential pre-processing step for critical tasks in the NLP pipeline such as tagging, parsing, named entity recognition, and more. For most languages, this tokenization step straightforward. However, for languages with high token-internal complexity, further token-to-word segmentation is required. Previous canonical segmentation studies were based on character-level frameworks, with no contextualised representation involved. Contextualized vectors a la BERT show remarkable results in many applications, but were not shown to improve performance on linguistic segmentation per se. Here we propose a novel neural segmentation model which combines the best of both worlds, contextualised token representation and char-level decoding, which is particularly effective for languages with high token-internal complexity and extreme morphological ambiguity. Our model shows substantial improvements in segmentation accuracy on Hebrew and Arabic compared to the state-of-the-art, and leads to further improvements on downstream tasks such as Part-of-Speech Tagging, Dependency Parsing and Named-Entity Recognition, over existing pipelines. When comparing our segmentation-first pipeline with joint segmentation and labeling in the same settings, we show that, contrary to pre-neural studies, the pipeline performance is superior.