Spelling Bee Embeddings for Language Modeling
Not on the map. Why?
Abstract (arXiv)
We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on spelling, but also across standard benchmarks. We conduct scaling studies for models with 40M to 800M parameters, which suggest that the improvements are equivalent to needing about 8% less compute and data to achieve the same test loss.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 22 references that are in the map. First posted on arXiv 25 Jan 2026.
Builds on
17 of 22 references.
References · 17
- 2017Attention Is All You Needcited by 60,361
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2020GLU Variants Improve Transformercited by 1,804
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2020On Layer Normalization in the Transformer Architecturecited by 922
- 2018Adaptive Input Representations for Neural Language Modelingcited by 226
- 2022Memorizing Transformerscited by 226
- 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
- 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
5 more
- 2021Self-attention Does Not Need O(n2) Memorycited by 154
- 2022Efficient Transformers with Dynamic Token Poolingcited by 76
- 2024Kolmogorov-Arnold Transformercited by 56
- 2024Why Do Large Language Models (LLMs) Struggle to Count Letters?cited by 38
- 2024Large Language Models Lack Understanding of Character Composition of Wordscited by 22
Built on by
Not cited yet.