mlsanityData to 14 Oct 2026

Towards a Small Language Model Lifecycle Framework

arXiv 2506.07695 ↗First posted 9 Jun 2025cs.SE cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Background: The growing demand for efficient and deployable language models has led to increased interest in Small Language Models (SLMs). However, existing research remains fragmented, lacking a unified lifecycle perspective. Objective: This study aims to define a comprehensive lifecycle framework for SLMs by synthesizing insights from academic literature and practitioner sources. Method: We conducted a comprehensive survey of 36 works, analyzing and categorizing lifecycle-relevant techniques. Results: We propose a modular lifecycle model structured into main, optional, and cross-cutting components. The model captures key interconnections across stages, supporting method reuse, co-adaptation, and lifecycle-awareness. Conclusion: Our framework provides a coherent foundation for developing and maintaining SLMs, bridging theory and practice, and guiding future research and tool development.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

38 of 49 references.

References · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2023A Survey of Large Language Modelscited by 2,576
  6. 2020Efficient Transformers: A Surveycited by 790
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  8. 2024Phi-4 Technical Reportcited by 711
  9. 2024Large Language Models: A Surveycited by 525
  10. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  11. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  12. 2023A Survey on Model Compression for Large Language Modelscited by 269
26 more
  1. 2024A Survey on Data Selection for Language Modelscited by 236
  2. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  4. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  5. 2023Efficient Large Language Models: A Surveycited by 186
  6. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  8. 2022Machine Learning Operations (MLOps): Overview, Definition, and Architecturecited by 128
  9. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 127
  10. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  11. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  12. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  13. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  14. 2024Datasets for Large Language Models: A Comprehensive Surveycited by 71
  15. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 67
  16. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  17. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  18. 2024A Survey on Transformer Compressioncited by 46
  19. 2017Guidelines for including grey literature and conducting multivocal literature reviews in software engineeringcited by 41
  20. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  21. 2024A Survey of Small Language Modelscited by 31
  22. 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
  23. 2024A Survey on LLM Inference-Time Self-Improvementcited by 17
  24. 2024Federated Distillation: A Surveycited by 15
  25. 2024Computational Bottlenecks of Training Small-scale Large Language Modelscited by 2
  26. 2023Toward End-to-End MLOps Tools Map: A Preliminary Study based on a Multivocal Literature Reviewcited by 1

Built on by

Not cited yet.