mlsanityData to 14 Oct 2026

Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation

arXiv 2505.19529 ↗First posted 26 May 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Small Language Models (SLMs) have gained substantial attention due to their ability to execute diverse language tasks successfully while using fewer computer resources. These models are particularly ideal for deployment in limited environments, such as mobile devices, on-device processing, and edge systems. In this study, we present a complete assessment of SLMs, focussing on their design frameworks, training approaches, and techniques for lowering model size and complexity. We offer a novel classification system to organize the optimization approaches applied for SLMs, encompassing strategies like pruning, quantization, and model compression. Furthermore, we assemble SLM's studies of evaluation suite with some existing datasets, establishing a rigorous platform for measuring SLM capabilities. Alongside this, we discuss the important difficulties that remain unresolved in this sector, including trade-offs between efficiency and performance, and we suggest directions for future study. We anticipate this study to serve as a beneficial guide for researchers and practitioners who aim to construct compact, efficient, and high-performing language models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

35 of 47 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
23 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  2. 2017Mixed Precision Trainingcited by 1,156
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  4. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  8. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  11. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  12. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  13. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  14. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  15. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  16. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
  17. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  18. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 41
  19. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  20. 2024Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMscited by 24
  21. 2024The Price of Prompting: Profiling Energy Use in Large Language Models Inferencecited by 23
  22. 2024Understanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thoughtcited by 19
  23. 2024FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Onlycited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Small Language Models
  2. 2024Small Language Models: Survey, Measurements, and Insights
  3. 2025SLMQuant:Benchmarking Small Language Model Quantization for Practical Deployment
  4. 2024On-Device Language Models: A Comprehensive Review
  5. 2024PanGu-π Pro:Rethinking Optimization and Architecture for Tiny Language Models
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
  7. 2025Slm-mux: Orchestrating small language models for reasoning
  8. 2025Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation