Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes

We study whether a decoder-only language model requires an independently trainable input vector for every token. For a vocabulary of size V, an injective fixed-length binary identifier requires K=lceillog2 Vrceil bits. We replace the usual trainable V× dmodel input table with fixed minimal binary token codes and a parameter-free tiled lift to model width. With V=65,536 and dmodel=1024, this supplies each token as a fixed 16-bit code and removes 67.1M trainable parameters, approximately 12.5% of the untied learned-input baseline. We also study a table-free implementation with one fixed invertible affine recoding over mathbbF216. Across three training seeds, 32-layer models trained on approximately 16-17B tokens obtain mean held-out perplexities of 2.44 for the learned-input baseline, 2.36 for canonical binary codes, and 2.39 for affine-recoded codes. These descriptive results do not establish statistical superiority or equivalence. Standardized evaluation of released base checkpoints with the LM Evaluation Harness adds commonsense, knowledge, and language-modeling benchmarks. The three paper checkpoints show broadly similar, task-dependent performance, while external SmolLM2 reference models are substantially stronger on many tasks. Our conclusion is therefore limited to the studied regime: a free trainable token-indexed input table is not required to learn nontrivial language modeling. The Transformer still learns continuous representations, and the output vocabulary projection remains standard and trainable.