mlsanityData to 14 Oct 2026

What is the Role of Small Models in the LLM Era: A Survey

arXiv 2409.06857 ↗First posted 10 Sep 2024cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities in various reasoning tasks, which leads to the development of increasingly large models. However, scaling up model sizes results in significantly higher computational costs and energy consumption, which makes these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior surveys. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition (or Complementarity). We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 161 of its 255 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

161 of 255 references.

References · 161

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  11. 2020Scaling Laws for Neural Language Modelscited by 6,586
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
38 more
  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  13. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  14. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  15. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  16. 2020Knowledge Distillation: A Surveycited by 1,603
  17. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  18. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  19. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  20. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  21. 2023LIMA: Less Is More for Alignmentcited by 1,100
  22. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  23. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  24. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  25. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  26. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  27. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  28. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  29. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  30. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  31. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  32. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  33. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  34. 2022A Survey on In-context Learningcited by 721
  35. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  36. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  37. 2024A Survey on Large Language Models for Code Generationcited by 658
  38. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609

50 most cited of 161 All 161 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 30; 2026: 6.

41 papers.

Sort:

By area:

  1. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 165
  2. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  3. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  4. 2024Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMscited by 17
  5. 2024Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understandingcited by 17
  6. 2025Query-Level Uncertainty in Large Language Modelscited by 14
  7. 2025Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Uglycited by 11
  8. 2026When Routing Collapses: On the Degenerate Convergence of LLM Routerscited by 10
  9. 2025A Strategic Coordination Framework of Small LLMs Matches Large LLMs in Data Synthesiscited by 9
  10. 2025BioAgents: Democratizing Bioinformatics Analysis with Multi-Agent Systemscited by 8
  11. 2025Automated CAD Modeling Sequence Generation from Text Descriptions via Transformer-Based Large Language Modelscited by 8
  12. 2025HoarePrompt: Structural Reasoning About Program Correctness in Natural Languagecited by 7
29 more
  1. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Modelscited by 7
  2. 2025Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignmentcited by 7
  3. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  4. 2025FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inferencecited by 4
  5. 2025Autonomous Radiotherapy Treatment Planning Using DOLA: A Privacy-Preserving, LLM-Based Optimization Agentcited by 4
  6. 2025AI Propaganda factories with language modelscited by 4
  7. 2026BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoningcited by 4
  8. 2024From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmapcited by 2
  9. 2025Knowledge Distillation: Enhancing Neural Network Compression with Integrated Gradientscited by 2
  10. 2025Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Realitycited by 2
  11. 2025Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Modelscited by 2
  12. 2025From Punchlines to Predictions: A Metric to Assess LLM Performance in Identifying Humor in Stand-Up Comedycited by 2
  13. 2025When LLMs Team Up: The Emergence of Collaborative Affective Computingcited by 2
  14. 2025Energy-Aware Code Generation with LLMs: Benchmarking Small vs. Large Language Models for Sustainable AI Programmingcited by 2
  15. 2025EQ-Negotiator: Dynamic Emotional Personas Empower Small Language Models for Edge-Deployable Credit Negotiationcited by 2
  16. 2025Pub-Guard-LLM: Detecting Retracted Biomedical Articles with Reliable Explanationscited by 1
  17. 2025LightRouter: Towards Efficient LLM Collaboration with Minimal Overheadcited by 1
  18. 2025NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Modelscited by 1
  19. 2025Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraintscited by 1
  20. 2025Analysing Environmental Efficiency in AI for X-Ray Diagnosiscited by 1
  21. 2026GraphSeek: Next-Generation Graph Analytics with LLMscited by 1
  22. 2026CogScale: Scalable Benchmark for Sequence Processingcited by 1
  23. 2026Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Modelscited by 1
  24. 2025Tracing Partisan Bias to Its Emotional Fingerprints: A Computational Approach to Mitigationcited by 0
  25. 2025Towards a Small Language Model Lifecycle Frameworkcited by 0
  26. 2025Model compression using knowledge distillation with integrated gradientscited by 0
  27. 2025SOLVE-Med: Specialized Orchestration for Leading Vertical Experts across Medical Specialtiescited by 0
  28. 2025Mirror, Mirror on the Wall -- Which is the Best Model of Them All?cited by 0
  29. 2026Scientific Knowledge-driven Decoding Constraints Improving the Reliability of LLMscited by 0