mlsanityData to 14 Oct 2026

zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss

arXiv 2610.13605 ↗First posted 9 Oct 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Dynamic vocabularies built on-the-fly via Lempel-Ziv-Welch (LZW) compression let a language model represent the same text in fewer tokens, but prior work shows that this efficiency comes at a cost to reasoning and perplexity. We show that this quality gap is not a fundamental price of compression but can be substantially closed through targeted improvements to the compression layer and LZW codec, without modifying the underlying pretrained language model. Our analysis reveals four key issues: (1) LZW's compression algorithm creates an implicit conflict between the input embedding and the output embedding, a tension that a single shared embedding cannot resolve; (2) naively applying RoPE to LZW-compressed sequences ignores that merged tokens can span multiple tokens; (3) merging certain tokens (notably digits) disproportionately hurts downstream accuracy; and (4) dynamic vocabularies introduce an inherent segmentation ambiguity that confounds standard evaluation. We address these through decoupled input/output dynamic embedding layer, size-aware RoPE positioning, and a modified LZW codec supporting selective merges and multi-view segmentations. Across four models, our recipe improves perplexity on every corpus we evaluated as well as mathematical reasoning accuracy, closing roughly half the quality gap between zip2zip and the uncompressed baseline at a similar compression ratio on general text. This represents a significant step toward lossless adaptive-vocabulary compression for language models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 42 of its 46 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

42 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023In-context Autoencoder for Context Compression in a Large Language Model13 Jul
  2. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models6 Oct

Other references · 40

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  8. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  9. 2023Adapting Language Models to Compress Contextscited by 356
  10. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  11. 2016Using the Output Embedding to Improve Language Modelscited by 272
  12. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
28 more
  1. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  2. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  3. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
  4. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  5. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  6. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  7. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 102
  8. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  9. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 92
  10. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  11. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  12. 2025SuperBPE: Space Travel for Language Modelscited by 63
  13. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 34
  14. 2024Zero-Shot Tokenizer Transfercited by 31
  15. 2021You should evaluate your language model on marginal likelihood over tokenisationscited by 30
  16. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 28
  17. 2023Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyondcited by 23
  18. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  19. 2023Should you marginalize over possible tokenizations?cited by 18
  20. 2024Multi-word Tokenization for Sequence Compressioncited by 17
  21. 2024Training LLMs over Neurally Compressed Textcited by 17
  22. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 16
  23. 2024Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Modelscited by 15
  24. 2024Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMscited by 12
  25. 2023Learn Your Tokens: Word-Pooled Tokenization for Language Modelingcited by 8
  26. 2023Toucan: Token-Aware Character Level Language Modelingcited by 8
  27. 2024Generation with Dynamic Vocabularycited by 6
  28. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity
  2. 2026Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Models
  3. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
  4. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  5. 2026Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven
  6. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models
  7. 2026SemanticFold: Latent Sequence Compression SeparatesLanguage Modeling, Decodability, and Reasoning
  8. 2025Lossless Token Sequence Compression via Meta-Tokens