mlsanityData to 7 Oct 2026

Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)

arXiv 2501.05465 ↗First posted 3 Jan 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

As foundation AI models continue to increase in size, an important question arises - is massive scale the only path forward? This survey of about 160 papers presents a family of Small Language Models (SLMs) in the 1 to 8 billion parameter range that demonstrate smaller models can perform as well, or even outperform large models. We explore task agnostic, general purpose SLMs, task-specific SLMs and techniques to create SLMs that can guide the community to build models while balancing performance, efficiency, scalability and cost. Furthermore we define and characterize SLMs' effective sizes, representing increased capability with respect to LLMs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 110 of its 129 references that are in the map. First posted on arXiv 3 Jan 2025.

Builds on

110 of 129 references.

References · 110

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  13. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  14. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  15. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  16. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  17. 2020Knowledge Distillation: A Surveycited by 1,578
  18. 2019Root Mean Square Layer Normalizationcited by 1,425
  19. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  20. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  21. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  22. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  23. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  24. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  25. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  26. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  27. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  28. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  29. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  30. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  31. 2023StarCoder: may the source be with you!cited by 790
  32. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  33. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  34. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  35. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  36. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  37. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  38. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668

50 most cited of 110 All 110 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoringcited by 3
  2. 2025TeamMedAgents: Pareto-Efficient Multi-Agent Medical Reasoning Through Teamwork Theorycited by 1