mlsanityData to 14 Oct 2026

A Survey of Small Language Models

arXiv 2410.20011 ↗First posted 25 Oct 2024cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources, making them ideal for various settings including on-device, mobile, edge devices, among many others. In this article, we present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques. We propose a novel taxonomy for categorizing the methods used to optimize SLMs, including model compression, pruning, and quantization techniques. We summarize the benchmark datasets that are useful for benchmarking SLMs along with the evaluation metrics commonly used. Additionally, we highlight key open challenges that remain to be addressed. Our survey aims to serve as a valuable resource for researchers and practitioners interested in developing and deploying small yet efficient language models.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 116 of its 147 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

116 of 147 references.

References · 116

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
38 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2020Reformer: The Efficient Transformercited by 1,668
  11. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  13. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  14. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  15. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  16. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  17. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  18. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  19. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  20. 2017Mixed Precision Trainingcited by 1,156
  21. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  22. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  23. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  24. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  25. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  26. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  27. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  28. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  29. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  30. 2019Are Sixteen Heads Really Better than One?cited by 816
  31. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  32. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  33. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  34. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  35. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  36. 2018CoQA: A Conversational Question Answering Challengecited by 683
  37. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  38. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637

50 most cited of 116 All 116 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 14.

31 papers.

Sort:

By area:

  1. 2025Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judgecited by 23
  2. 2026Skill-R1: Agent Skill Evolution via Reinforcement Learningcited by 15
  3. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deploymentcited by 10
  4. 2025Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Callingcited by 7
  5. 2025AI Propaganda factories with language modelscited by 4
  6. 2025Learnware of Language Models: Specialized Small Language Models Can Do Bigcited by 3
  7. 2025Effective Learning for Small Reasoning Models: An Empirical Study on 0.5B Reasoning LLMscited by 3
  8. 2025Menta: A Small Language Model for On-Device Mental Health Predictioncited by 3
  9. 2025Beyond the Tip of Efficiency: Uncovering the Submerged Threats of Jailbreak Attacks in Small Language Modelscited by 2
  10. 2025Advancing SLM Tool-Use Capability using Reinforcement Learningcited by 2
  11. 2024Do Large Language Models Advocate for Inferentialism?cited by 1
  12. 2025Win-k: Improved Membership Inference Attacks on Small Language Modelscited by 1
19 more
  1. 2026Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Modelscited by 1
  2. 2024Insights into resource utilization of code small language models serving with runtime engines and execution providerscited by 0
  3. 2025Optimization Problem Solving Can Transition to Evolutionary Agentic Workflowscited by 0
  4. 2025PWC-MoE: Privacy-Aware Wireless Collaborative Mixture of Expertscited by 0
  5. 2025Towards a Small Language Model Lifecycle Frameworkcited by 0
  6. 2025IMB: An Italian Medical Benchmark for Question Answeringcited by 0
  7. 2025Evaluating Small Language Models for Agentic On-Farm Decision Support Systemscited by 0
  8. 2026In-Browser Agents for Search Assistancecited by 0
  9. 2026Performance of Small Language Model Pretraining on FABRIC: An Empirical Studycited by 0
  10. 2026Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experimentcited by 0
  11. 2026FedDetox: Robust Federated SLM Alignment via On-Device Data Sanitizationcited by 0
  12. 2026Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insightscited by 0
  13. 2026Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systemscited by 0
  14. 2026In-Context Learning for Wound Classification with Small Multimodal Language Modelscited by 0
  15. 2026MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Modelscited by 0
  16. 2026Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimizationcited by 0
  17. 2026Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUscited by 0
  18. 2026Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimizationcited by 0
  19. 2026Evaluating Context Segmentation in Locally Deployable SLMs for Cybersecurity CTF Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation
  2. 2025SLMQuant:Benchmarking Small Language Model Quantization for Practical Deployment
  3. 2024Small Language Models: Survey, Measurements, and Insights
  4. 2024PanGu-π Pro:Rethinking Optimization and Architecture for Tiny Language Models
  5. 2023A Survey on Model Compression for Large Language Models
  6. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms
  7. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies