mlsanityData to 7 Oct 2026

Are Small Language Models Ready to Compete with Large Language Models for Practical Applications?

arXiv 2406.11402 ↗First posted 17 Jun 2024cs.CL cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

The rapid rise of Language Models (LMs) has expanded their use in several applications. Yet, due to constraints of model size, associated cost, or proprietary restrictions, utilizing state-of-the-art (SOTA) LLMs is not always feasible. With open, smaller LMs emerging, more applications can leverage their capabilities, but selecting the right LM can be challenging as smaller LMs do not perform well universally. This work tries to bridge this gap by proposing a framework to experimentally evaluate small, open LMs in practical settings through measuring semantic correctness of outputs across three practical aspects: task types, application domains, and reasoning types, using diverse prompt styles. It also conducts an in-depth comparison of 10 small, open LMs to identify the best LM and prompt style depending on specific application requirements using the proposed framework. We also show that if selected appropriately, they can outperform SOTA LLMs like DeepSeek-v2, GPT-4o, GPT-4o-mini, Gemini-1.5-Pro, and even compete with GPT-4o.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

28 of 47 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2022A Survey on In-context Learningcited by 717
16 more
  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  3. 2023The Falcon Series of Open Language Modelscited by 526
  4. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  5. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  6. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  7. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  8. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  9. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  10. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  11. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  12. 2020Unsupervised Multi-hop Question Answering by Question Generationcited by 38
  13. 2024Prompt Perturbation Consistency Learning for Robust Language Modelscited by 26
  14. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  15. 2024ChatDiet: Empowering Personalized Nutrition-Oriented Food Recommender Chatbots through an LLM-Augmented Frameworkcited by 20
  16. 2024Deciphering Political Entity Sentiment in News with Large Language Models: Zero-Shot and Few-Shot Strategiescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 3.

9 papers.

  1. 2026Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Modelscited by 15
  2. 2026Lightweight LLM Agent Memory with Small Language Modelscited by 12
  3. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Typescited by 9
  4. 2025Reproducibility Study of Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiationcited by 4
  5. 2025QA-prompting: Improving Summarization with Large Language Models using Question-Answeringcited by 4
  6. 2024Robust Privacy Amidst Innovation with Large Language Models Through a Critical Assessment of the Riskscited by 3
  7. 2025CARGO: A Framework for Confidence-Aware Routing of Large Language Modelscited by 3
  8. 2025VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Modelscited by 2
  9. 2026Relation Extraction Capabilities of LLMs on Clinical Text: A Bilingual Evaluation for English and Turkishcited by 0