# H-Net++: Hierarchical Dynamic Chunking for Tokenizer-Free Language Modelling in Morphologically-Rich Languages

arXiv: https://arxiv.org/abs/2508.05628 · HTML: https://mlsanity.com/paper/2508.05628/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers](https://mlsanity.com/paper/2305.07185/) (2023)
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021)
- [Character-Level Language Modeling with Deeper Self-Attention](https://mlsanity.com/paper/1808.04444/) (2018)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [ParsBERT: Transformer-based Model for Persian Language Understanding](https://mlsanity.com/paper/2005.12515/) (2020)
- [CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters](https://mlsanity.com/paper/2010.10392/) (2020)
- [Dynamic Chunking for End-to-End Hierarchical Sequence Modeling](https://mlsanity.com/paper/2507.07955/) (2025)
- [Sequence Modeling via Segmentations](https://mlsanity.com/paper/1702.07463/) (2017)
- [Adaptive Summaries: A Personalized Concept-based Summarization Approach by Learning from Users' Feedback](https://mlsanity.com/paper/2012.13387/) (2020)
- [SumRecom: A Personalized Summarization Approach by Learning from Users' Feedback](https://mlsanity.com/paper/2408.07294/) (2024)

## Built on by (most cited first, 1 of 1)

- [Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model](https://mlsanity.com/paper/2605.30080/) (2026)

## Abstract (arXiv)

Byte-level language models eliminate fragile tokenizers but face computational challenges in morphologically-rich languages (MRLs), where words span many bytes. We propose H-NET++, a hierarchical dynamic-chunking model that learns linguistically-informed segmentation through end-to-end training. Key innovations include: (1) a lightweight Transformer context-mixer (1.9M parameters) for cross-chunk attention, (2) a two-level latent hyper-prior for document-level consistency, (3) specialized handling of orthographic artifacts (e.g. Persian ZWNJ), and (4) curriculum-based training with staged sequence lengths. On a 1.4B-token Persian corpus, H-NET++ achieves state-of-the-art results: 0.159 BPB reduction versus BPE-based GPT-2-fa (12% better compression), 5.4pp gain on ParsGLUE, 53% improved robustness to ZWNJ corruption, and 73.8% F1 on gold morphological boundaries. Our learned chunks align with Persian morphology without explicit supervision, demonstrating that hierarchical dynamic chunking provides an effective tokenizer-free solution for MRLs while maintaining computational efficiency.