# Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes

arXiv: https://arxiv.org/abs/2605.09751 · HTML: https://mlsanity.com/paper/2605.09751/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 10 May 2026.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers](https://mlsanity.com/paper/2305.07185/) (2023)
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021)
- [Character-Aware Neural Language Models](https://mlsanity.com/paper/1508.06615/) (2015)
- [MaskBit: Embedding-free Image Generation via Bit Tokens](https://mlsanity.com/paper/2409.16211/) (2024)
- [Efficient softmax approximation for GPUs](https://mlsanity.com/paper/1609.04309/) (2016)
- [Hash Embeddings for Efficient Word Representations](https://mlsanity.com/paper/1709.03933/) (2017)
- [Compressing Word Embeddings via Deep Compositional Code Learning](https://mlsanity.com/paper/1711.01068/) (2017)
- [DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling](https://mlsanity.com/paper/1911.12385/) (2019)
- [All Word Embeddings from One Embedding](https://mlsanity.com/paper/2004.12073/) (2020)
- [Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations](https://mlsanity.com/paper/2507.04886/) (2025)
- [LightRNN: Memory and Computation-Efficient Recurrent Neural Networks](https://mlsanity.com/paper/1610.09893/) (2016)

## Abstract (arXiv)

We study whether a decoder-only language model requires an independently trainable input vector for every token. For a vocabulary of size V, an injective fixed-length binary identifier requires K=lceillog2 Vrceil bits. We replace the usual trainable V× dmodel input table with fixed minimal binary token codes and a parameter-free tiled lift to model width. With V=65,536 and dmodel=1024, this supplies each token as a fixed 16-bit code and removes 67.1M trainable parameters, approximately 12.5% of the untied learned-input baseline. We also study a table-free implementation with one fixed invertible affine recoding over mathbbF216. Across three training seeds, 32-layer models trained on approximately 16-17B tokens obtain mean held-out perplexities of 2.44 for the learned-input baseline, 2.36 for canonical binary codes, and 2.39 for affine-recoded codes. These descriptive results do not establish statistical superiority or equivalence. Standardized evaluation of released base checkpoints with the LM Evaluation Harness adds commonsense, knowledge, and language-modeling benchmarks. The three paper checkpoints show broadly similar, task-dependent performance, while external SmolLM2 reference models are substantially stronger on many tasks. Our conclusion is therefore limited to the studied regime: a free trainable token-indexed input table is not required to learn nontrivial language modeling. The Transformer still learns continuous representations, and the output vocabulary projection remains standard and trainable.