# Frequency-Ordered Tokenization for Better Text Compression

arXiv: https://arxiv.org/abs/2602.22958 · HTML: https://mlsanity.com/paper/2602.22958/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 33 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 3 of 3)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)

## Built on by (most cited first, 1 of 1)

- [Token-Native Storage: Read and Write in your Agent's Language](https://mlsanity.com/paper/2608.02376/) (2026)

## Abstract (arXiv)

We present frequency-ordered tokenization (FOT), a simple preprocessing technique that improves lossless text compression by exploiting the power-law frequency distribution of natural language (Zipf's law). FOT segments text into the vocabulary of a pretrained Byte Pair Encoding (BPE) tokenizer, renumbers tokens by frequency with a fixed table, and writes them with a short byte code before any standard compressor. Because the FOT stream is less than half the size of the text, compressors produce smaller output and run faster at the same time. On enwik8 (100 MB of Wikipedia), FOT reduces the compressed size by 7.96 percentage points (pp) for zlib, 2.24 pp for xz (LZMA), 1.96 pp for brotli and 1.44 pp for zstd, while making total compression, including the transform, 1.6x, 3.2x, 1.8x and 5.0x faster. Gains hold from 100 KB to 1 GB (enwik9) and across texts of diverse languages, and outperform the classical Word Replacing Transform. On enwik9, FOT with tuned xz sets a new state of the art among LZ-based compressors on the Large Text Compression Benchmark. The method can be implemented in under 50 lines of code.