mlsanityData to 7 Oct 2026

SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model

arXiv 2502.02737 ↗First posted 4 Feb 2025cs.CLcited by 307

Not on the map. Why?

Cited by 49 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While large language models have facilitated breakthroughs in many applications of artificial intelligence, their inherent largeness makes them computationally expensive and challenging to deploy in resource-constrained settings. In this paper, we document the development of SmolLM2, a state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a multi-stage training process that mixes web text with specialized math, code, and instruction-following data. We additionally introduce new specialized datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing datasets to be problematically small or low-quality. To inform our design decisions, we perform both small-scale ablations as well as a manual refinement process that updates the dataset mixing rates at each stage based on the performance at the previous stage. Ultimately, we demonstrate that SmolLM2 outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To facilitate future research on LM development as well as applications of small LMs, we release both SmolLM2 as well as all of the datasets we prepared in the course of this project.

Key facts. Cited by 307 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 132; 2026: 175.

307 papers.

Sort:

By area:

  1. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  3. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  4. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  5. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  6. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  7. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  8. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 81
  9. 2025Scaling Language-Free Visual Representation Learningcited by 62
  10. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  11. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  12. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
38 more
  1. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 36
  2. 2025TiDAR: Think in Diffusion, Talk in Autoregressioncited by 35
  3. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
  4. 2026DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulationcited by 33
  5. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 32
  6. 2025BitNet b1.58 2B4T Technical Reportcited by 30
  7. 2025Parallel Scaling Law for Language Modelscited by 29
  8. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  9. 2025Mellow: a small audio language model for reasoningcited by 27
  10. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuningcited by 27
  11. 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 20
  12. 2025Mid-Training of Large Language Models: A Surveycited by 18
  13. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Modelscited by 16
  14. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 16
  15. 2025OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learningcited by 15
  16. 2025A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clonecited by 15
  17. 2025Soft-Masked Diffusion Language Modelscited by 15
  18. 2025Machine-generated text detection prevents language model collapsecited by 14
  19. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Themcited by 14
  20. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 13
  21. 2025T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Modelscited by 13
  22. 2025Krikri: Advancing Open Large Language Models for Greekcited by 13
  23. 2025NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communitiescited by 13
  24. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Modelscited by 13
  25. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 12
  26. 2026dLLM: Simple Diffusion Language Modelingcited by 12
  27. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 11
  28. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  29. 2026Shaping capabilities with token-level data filteringcited by 11
  30. 2026Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?cited by 11
  31. 2025Rainbow Padding: Mitigating Early Termination in Instruction-Tuned Diffusion LLMscited by 10
  32. 2026Arcee Trinity Large Technical Reportcited by 10
  33. 2025TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarkingcited by 9
  34. 2025Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarkscited by 9
  35. 2025MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generationcited by 9
  36. 2025A*-Thought: Efficient Reasoning via Bidirectional Compression for Low-Resource Settingscited by 9
  37. 2025Mixture of Cognitive Reasoners: Modular Reasoning with Brain-Like Specializationcited by 9
  38. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 9

50 most cited of 307 All 307 (JSON)