# SpeLLM: Character-Level Multi-Head Decoding

arXiv: https://arxiv.org/abs/2507.16323 · HTML: https://mlsanity.com/paper/2507.16323/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 30 references that are in the map.
- First posted on arXiv 22 Jul 2025.

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://mlsanity.com/paper/2305.13707/) (2023)
- [MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers](https://mlsanity.com/paper/2305.07185/) (2023)
- [Byte Latent Transformer: Patches Scale Better Than Tokens](https://mlsanity.com/paper/2412.09871/) (2024)
- [Efficient softmax approximation for GPUs](https://mlsanity.com/paper/1609.04309/) (2016)
- [Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers](https://mlsanity.com/paper/2103.09301/) (2021)
- [Improving Multilingual Models with Language-Clustered Vocabularies](https://mlsanity.com/paper/2010.12777/) (2020)
- [Fast Vocabulary Transfer for Language Model Compression](https://mlsanity.com/paper/2402.09977/) (2024)
- [What do tokens know about their characters and how do they know it?](https://mlsanity.com/paper/2206.02608/) (2022)
- [Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens](https://mlsanity.com/paper/2108.11193/) (2021)
- [FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling](https://mlsanity.com/paper/2502.14856/) (2025)
- [T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings](https://mlsanity.com/paper/2406.19223/) (2024)
- [Headless Language Models: Learning without Predicting with Contrastive Weight Tying](https://mlsanity.com/paper/2309.08351/) (2023)
- [Vocabulary-level Memory Efficiency for Language Model Fine-tuning](https://mlsanity.com/paper/2309.08708/) (2023)
- [LLM Vocabulary Compression for Low-Compute Environments](https://mlsanity.com/paper/2411.06371/) (2024)

## Built on by (most cited first, 2 of 2)

- [Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic](https://mlsanity.com/paper/2510.17001/) (2025)
- [More Than Words: Compositional Tokenization for Efficient Language Models](https://mlsanity.com/paper/2610.05597/) (2026)

## Abstract (arXiv)

Scaling LLM vocabulary is often used to reduce input sequence length and alleviate attention's quadratic cost. Yet, current LLM architectures impose a critical bottleneck to this procedure: the output projection layer scales linearly with vocabulary size, rendering substantial expansion impractical. We propose SpeLLM, a method that decouples input and output vocabularies by predicting character-level strings through multiple output heads. In SpeLLM, each of the k linear heads predicts a single character simultaneously, enabling the model to represent a much larger output space using smaller, independent linear heads. We present a self-distillation approach for converting a standard LLM to a SpeLLM. Our experiments with four pre-trained LLMs show their SpeLLM variants achieve competitive performance on downstream tasks while reducing runtime by 5.1% on average across models. Our approach provides a potential avenue for reducing LLM costs, while increasing support for underrepresented languages and domains.