mlsanityData to 7 Oct 2026

A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

arXiv 2411.03350 ↗First posted 4 Nov 2024cs.CL cs.AI cs.LGcited by 125

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like PaLM 540B and Llama-3.1 405B face limitations due to large parameter sizes and computational demands, often requiring cloud API use which raises privacy concerns, limits real-time applications on edge devices, and increases fine-tuning costs. Additionally, LLMs often underperform in specialized domains such as healthcare and law due to insufficient domain-specific knowledge, necessitating specialized models. Therefore, Small Language Models (SLMs) are increasingly favored for their low inference latency, cost-effectiveness, efficient development, and easy customization and adaptability. These models are particularly well-suited for resource-limited environments and domain knowledge acquisition, addressing LLMs' challenges and proving ideal for applications that require localized data handling for privacy, minimal inference latency for efficiency, and domain knowledge acquisition through lightweight fine-tuning. The rising demand for SLMs has spurred extensive research and development. However, a comprehensive survey investigating issues related to the definition, acquisition, application, enhancement, and reliability of SLM remains lacking, prompting us to conduct a detailed survey on these topics. The definition of SLMs varies widely, thus to standardize, we propose defining SLMs by their capability to perform specialized tasks and suitability for resource-constrained settings, setting boundaries based on the minimal size for emergent abilities and the maximum size sustainable under resource constraints. For other aspects, we provide a taxonomy of relevant models/methods and develop general frameworks for each category to enhance and utilize SLMs effectively.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

23 of 34 references.

References · 23

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  10. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  11. 2022Efficiently Scaling Transformer Inferencecited by 529
  12. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
11 more
  1. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  3. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  4. 2023Self-Knowledge Guided Retrieval Augmentation for Large Language Modelscited by 69
  5. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  6. 2023Exploring the psychology of LLMs' Moral and Legal Reasoningcited by 43
  7. 2022CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender Systemcited by 38
  8. 2023MCC-KD: Multi-CoT Consistent Knowledge Distillationcited by 26
  9. 2023Test-Time Self-Adaptive Small Language Models for Question Answeringcited by 7
  10. 2024SLM as Guardian: Pioneering AI Safety with Small Language Modelscited by 7
  11. 2022COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 64; 2026: 56.

125 papers.

Sort:

By area:

  1. 2025Small Language Models are the Future of Agentic AIcited by 254
  2. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 50
  5. 2025Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learningcited by 37
  6. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  7. 2025Towards Reasoning Ability of Small Language Modelscited by 29
  8. 2025MiniRAG: Towards Extremely Simple Retrieval-Augmented Generationcited by 28
  9. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directionscited by 25
  10. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  11. 2025Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inferencecited by 18
  12. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Taskscited by 16
38 more
  1. 2026Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Modelscited by 15
  2. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  3. 2025Motion Forecasting for Autonomous Vehicles: A Surveycited by 8
  4. 2025Efficient Construction of Model Family through Progressive Training Using Model Expansioncited by 8
  5. 2025MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agentscited by 8
  6. 2024Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Datacited by 7
  7. 2025Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Callingcited by 7
  8. 2025The State of Large Language Models for African Languages: Progress and Challengescited by 7
  9. 2025Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learningcited by 7
  10. 2025KnowsLM: A framework for evaluation of small language models for knowledge augmentation and humanised conversationscited by 6
  11. 2025A Survey of Classification Tasks and Approaches for Legal Contractscited by 6
  12. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  13. 2025Slm-mux: Orchestrating small language models for reasoningcited by 5
  14. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  15. 2026CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patternscited by 5
  16. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  17. 2025The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMscited by 4
  18. 2026Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuningcited by 4
  19. 2025Can Small Language Models Reliably Resist Jailbreak Attacks? A Comprehensive Evaluationcited by 3
  20. 2025Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Familycited by 3
  21. 2025Learnware of Language Models: Specialized Small Language Models Can Do Bigcited by 3
  22. 2025Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Modelscited by 3
  23. 2025Pico: A Modular Framework for Hypothesis-Driven Small Language Model Researchcited by 3
  24. 2025ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answeringcited by 3
  25. 2025Menta: A Small Language Model for On-Device Mental Health Predictioncited by 3
  26. 2026Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environmentscited by 3
  27. 2026SWE-Prot'eg'e: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agentscited by 3
  28. 2026A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inferencecited by 3
  29. 2026Select to Think: Unlocking SLM Potential with Local Sufficiencycited by 3
  30. 2024Generative AI Toolkit -- a framework for increasing the quality of LLM-based applications over their whole life cyclecited by 2
  31. 2025Causal Distillation: Transferring Structured Explanations from Large to Compact Language Modelscited by 2
  32. 2025When LLMs Team Up: The Emergence of Collaborative Affective Computingcited by 2
  33. 2025Detecting Voice Phishing with Precision: Fine-Tuning Small Language Modelscited by 2
  34. 2025Retrieval-augmented reasoning with lean language modelscited by 2
  35. 2025Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagationcited by 2
  36. 2025DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarizationcited by 2
  37. 2025Mnemosyne: An Unsupervised, Human-Inspired Long-Term Memory Architecture for Edge-Based LLMscited by 2
  38. 2026Reasoning or Fluency? Dissecting Probabilistic Confidence in Best-of-N Selectioncited by 2

50 most cited of 125 All 125 (JSON)