mlsanityData to 7 Oct 2026

Super Tiny Language Models

arXiv 2405.14159 ↗First posted 23 May 2024cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

The rapid advancement of large language models (LLMs) has led to significant improvements in natural language processing but also poses challenges due to their high computational and energy demands. This paper introduces a series of research efforts focused on Super Tiny Language Models (STLMs), which aim to deliver high performance with significantly reduced parameter counts. We explore innovative techniques such as byte-level tokenization with a pooling mechanism, weight tying, and efficient training strategies. These methods aim to significantly reduce reduce the parameter count compared to traditional models -- in future works, we aim to build on these in a way that maintains and improves upon the performance of base transformer models. This series of papers will explore into various subproblems, including tokenizer-free models, self-play based training, and alternative training objectives. We will target models with 10M, 50M, and 100M parameters. Our ultimate goal is to make high-performance language models more accessible and practical for a wide range of applications.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

38 of 51 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
26 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2024Mixtral of Expertscited by 1,785
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  13. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  14. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  15. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  16. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  17. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  18. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  19. 2016Using the Output Embedding to Improve Language Modelscited by 266
  20. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  21. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  22. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 37
  23. 2024MobiLlama: Towards Accurate and Lightweight Fully Transparent GPTcited by 34
  24. 2019Tensorized Embedding Layers for Efficient Model Compressioncited by 27
  25. 2024Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Textscited by 26
  26. 2023Dropout Reduces Underfittingcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  2. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  3. 2025Parameterized Synthetic Text Generation with SimpleStoriescited by 9
  4. 2025StochasTok: Improving Fine-Grained Subword Understanding in LLMscited by 6
  5. 2025All is Not Lost: LLM Recovery without Checkpointscited by 1
  6. 2026Stochasticity in Tokenisation Improves Robustnesscited by 1
  7. 2025Toward Agentic Environments: GenAI and the Convergence of AI, Sustainability, and Human-Centric Spacescited by 0