mlsanityData to 7 Oct 2026

Scaling Embedding Layers in Language Models

arXiv 2502.01637 ↗First posted 3 Feb 2025cs.CL cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

We propose SCONE (Scalable, Contextualized, Offloaded, N-gram Embedding), a new method for extending input embedding layers to enhance language model performance. To avoid increased decoding costs, SCONE retains the original vocabulary while introducing embeddings for a set of frequent n-grams. These embeddings provide contextualized representation for each input token and are learned with a separate model during training. After training, embeddings are precomputed and stored in off-accelerator memory; during inference, querying them has minimal impact on latency due to the low complexity of embedding lookups. SCONE enables two new scaling strategies: increasing the number of n-gram embeddings and scaling the model used to learn them, both while maintaining fixed accelerator usage during inference (in terms of FLOPS and memory). We show that scaling both aspects enables a model with 1B accelerator-resident parameters to outperform a 1.9B-parameter baseline across diverse corpora, while using only about half the FLOPS and accelerator memory during inference.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

57 of 74 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2017Billion-scale similarity search with GPUscited by 2,477
38 more
  1. 2018Deep contextualized word representationscited by 2,285
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  8. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  12. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  13. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  14. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  15. 2014Memory Networkscited by 354
  16. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  17. 2024Nemotron-4 340B Technical Reportcited by 143
  18. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 137
  19. 2015Character-Aware Neural Language Modelscited by 129
  20. 2019Large Memory Layers with Product Keyscited by 120
  21. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  22. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  23. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  24. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  25. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  26. 2022Formal Algorithms for Transformerscited by 70
  27. 2024Mixture of A Million Expertscited by 70
  28. 2016Efficient softmax approximation for GPUscited by 67
  29. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 65
  30. 2021Scaling Scaling Laws with Board Gamescited by 62
  31. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  32. 2024Compression Represents Intelligence Linearlycited by 56
  33. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  34. 2021Learnable Embedding Sizes for Recommender Systemscited by 42
  35. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  36. 2024Memory Layers at Scalecited by 39
  37. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 27
  38. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 13.

19 papers.

Sort:

By area:

  1. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  2. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  3. 2025Exploiting Vocabulary Frequency Imbalance in Language Model Pre-trainingcited by 7
  4. 2025Lossless Token Sequence Compression via Meta-Tokenscited by 4
  5. 2025DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Modelscited by 4
  6. 2026Proxy Compression for Language Modelingcited by 3
  7. 2025VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Modelscited by 2
  8. 2026L3: Large Lookup Layerscited by 2
  9. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 2
  10. 2026Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scalingcited by 1
  11. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  12. 2026Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMscited by 1
7 more
  1. 2025Dynamic Injection of Entity Knowledge into Dense Retrieverscited by 0
  2. 2025Mixture of Lookup Key-Value Expertscited by 0
  3. 2026Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Modelscited by 0
  4. 2026NGM: A Plug-and-Play Training-Free Memory Module for LLMscited by 0
  5. 2026MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookupcited by 0
  6. 2026Large language models in medical time series analysiscited by 0
  7. 2026Forking: Sudden Overfitting Under Replaycited by 0