# Chunk-Distilled Language Modeling

arXiv: https://arxiv.org/abs/2501.00343 · HTML: https://mlsanity.com/paper/2501.00343/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2024.

## Built on by (most cited first, 1 of 1)

- [FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenization](https://mlsanity.com/paper/2512.04952/) (2025)

## Abstract (arXiv)

We introduce Chunk-Distilled Language Modeling (CD-LM), an approach to text generation that addresses two challenges in current large language models (LLMs): the inefficiency of token-level generation, and the difficulty of adapting to new data and knowledge. Our method combines deep network-based LLMs with a straightforward retrieval module, which allows the generation of multi-token text chunks at a single decoding step. Our retrieval framework enables flexible construction of model- or domain-specific datastores, either leveraging the internal knowledge of existing models, or incorporating expert insights from human-annotated corpora. This adaptability allows for enhanced control over the language model's distribution without necessitating additional training. We present the CD-LM formulation along with performance metrics demonstrating its ability to improve language model performance and efficiency across a diverse set of downstream tasks. Code and data will be made publicly available.