mlsanityData to 7 Oct 2026

Small Language Models: Survey, Measurements, and Insights

arXiv 2409.15790 ↗First posted 24 Sep 2024cs.CL cs.AI cs.LGcited by 109

Not on the map. Why?

Cited by 23 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Small language models (SLMs), despite their widespread adoption in modern smart devices, have received significantly less academic attention compared to their large language model (LLM) counterparts, which are predominantly deployed in data centers and cloud environments. While researchers continue to improve the capabilities of LLMs in the pursuit of artificial general intelligence, SLM research aims to make machine intelligence more accessible, affordable, and efficient for everyday tasks. Focusing on transformer-based, decoder-only language models with 100M-5B parameters, we survey 70 state-of-the-art open-source SLMs, analyzing their technical innovations across three axes: architectures, training datasets, and training algorithms. In addition, we evaluate their capabilities in various domains, including commonsense reasoning, mathematics, in-context learning, and long context. To gain further insight into their on-device runtime costs, we benchmark their inference latency and memory footprints. Through in-depth analysis of our benchmarking data, we offer valuable insights to advance research in this field.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Builds on 54 of its 77 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

54 of 77 references.

References · 54

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  8. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  9. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  10. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  11. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  12. 2020The Pushshift Reddit Datasetcited by 289
  13. 2022Teaching Small Language Models to Reasoncited by 279
  14. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  15. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  16. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  17. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  18. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  19. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  20. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  21. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  22. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  23. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  24. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  25. 2024Fast On-device LLM Inference with NPUscited by 58
  26. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  27. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  28. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  29. 2024Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoningcited by 31
  30. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
  31. 2023Mobile Foundation Model as Firmwarecited by 25
  32. 2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 19
  33. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  34. 2024Small Language Models are Good Too: An Empirical Study of Zero-Shot Classificationcited by 17
  35. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 12
  36. 2023Weight-Inherited Distillation for Task-Agnostic BERT Compressioncited by 8
  37. 2024Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanationscited by 8
  38. 2024Small Language Models for Application Interactions: A Case Studycited by 5

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 62; 2026: 39.

109 papers.

Sort:

By area:

  1. 2025Small Language Models are the Future of Agentic AIcited by 254
  2. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  3. 2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 41
  4. 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 31
  5. 2025Towards Reasoning Ability of Small Language Modelscited by 29
  6. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  7. 2025Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agentscited by 24
  8. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  9. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  10. 2025T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Modelscited by 13
  11. 2025Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performancecited by 12
  12. 2025Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Studycited by 11
38 more
  1. 2025Enhancing Autonomous Driving Systems with On-Board Deployed Large Language Modelscited by 11
  2. 2025Edge-First Language Model Inference: Models, Metrics, and Tradeoffscited by 11
  3. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deploymentcited by 10
  4. 2025GPIoT: Tailoring Small Language Models for IoT Program Synthesis and Developmentcited by 10
  5. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  6. 2025SLM-SQL: An Exploration of Small Language Models for Text-to-SQLcited by 10
  7. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidancecited by 10
  8. 2025RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimizationcited by 9
  9. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approachcited by 8
  10. 2025Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exitscited by 8
  11. 2024PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-trainingcited by 7
  12. 2025Towards Achieving Concept Completeness for Textual Concept Bottleneck Modelscited by 6
  13. 2025Position: Enough of Scaling LLMs! Lets Focus on Downscalingcited by 5
  14. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  15. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  16. 2026CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patternscited by 5
  17. 2026EmoMAS: Emotion-Aware Multi-Agent System for High-Stakes Edge-Deployable Negotiation with Bayesian Orchestrationcited by 5
  18. 2024DroidCall: A Dataset for LLM-powered Android Intent Invocationcited by 4
  19. 2025Cost-Optimal Grouped-Query Attention for Long-Context Modelingcited by 4
  20. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  21. 2025Chatbot To Help Patients Understand Their Healthcited by 4
  22. 2025Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Modelscited by 4
  23. 2025GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devicescited by 3
  24. 2025Can Small Language Models Reliably Resist Jailbreak Attacks? A Comprehensive Evaluationcited by 3
  25. 2025Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inferencecited by 3
  26. 2025Recent Advances in Transformer and Large Language Models for UAV Applicationscited by 3
  27. 2025Menta: A Small Language Model for On-Device Mental Health Predictioncited by 3
  28. 2025Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Modelscited by 2
  29. 2025Every Software as an Agent: Blueprint and Case Studycited by 2
  30. 2025Beyond the Tip of Efficiency: Uncovering the Submerged Threats of Jailbreak Attacks in Small Language Modelscited by 2
  31. 2025S2GPT-PINNs: Sparse and Small models for PDEscited by 2
  32. 2025DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarizationcited by 2
  33. 2025Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptationcited by 2
  34. 2025EQ-Negotiator: Dynamic Emotional Personas Empower Small Language Models for Edge-Deployable Credit Negotiationcited by 2
  35. 2026Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspacescited by 2
  36. 2026CLAG: Adaptive Memory Organization via Agent-Driven Clustering for Small Language Model Agentscited by 2
  37. 2026To Isolate or to Score? Model-Adaptive Assessment for Cost-Efficient Multi-Agent RAGcited by 2
  38. 2024Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Modelscited by 1

50 most cited of 109 All 109 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Slm-mux: Orchestrating small language models for reasoning
  2. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
  3. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?
  4. 2025Small Language Models for Efficient Agentic Tool Calling: Outperforming Large Models with Targeted Fine-tuning
  5. 2025Enhancing Reasoning Capabilities of Small Language Models with Blueprints and Prompt Template Search
  6. 2024Can LLMs Compute with Reasons?
  7. 2023A Comprehensive Overview of Large Language Models
  8. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models