mlsanityData to 14 Oct 2026

Mini-Giants: "Small" Language Models and Open Source Win-Win

arXiv 2307.08189 ↗First posted 17 Jul 2023cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

ChatGPT is phenomenal. However, it is prohibitively expensive to train and refine such giant models. Fortunately, small language models are flourishing and becoming more and more competent. We call them "mini-giants". We argue that open source community like Kaggle and mini-giants will win-win in many ways, technically, ethically and socially. In this article, we present a brief yet rich background, discuss how to attain small language models, present a comparative study of small language models and a brief discussion of evaluation methods, discuss the application scenarios where small language models are most needed in the real world, and conclude with discussion and outlook.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 65 of its 110 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

65 of 110 references.

References · 65

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  13. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  14. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  15. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  16. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  17. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  18. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  19. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  20. 2023BloombergGPT: A Large Language Model for Financecited by 939
  21. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  22. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  23. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  24. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  25. 2023Textbooks Are All You Needcited by 569
  26. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  27. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  28. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  29. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  30. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  31. 2023MusicLM: Generating Music From Textcited by 448
  32. 2023Language Models can Solve Computer Taskscited by 372
  33. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  34. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  35. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  36. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  37. 2023FinGPT: Open-Source Financial Large Language Modelscited by 253
  38. 2022Neural Networks and the Chomsky Hierarchycited by 217

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.

3 papers.

  1. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  2. 2023LLM-Twin: Mini-Giant Model-driven Beyond 5G Digital Twin Networking Framework with Semantic Secure Communication and Computationcited by 4
  3. 2026Bespoke Visual Assistance: What and How do Blind and Low-Vision People Create with Agentic Programming?cited by 0