mlsanityData to 7 Oct 2026

LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions

arXiv 2304.14402 ↗First posted 27 Apr 2023cs.CLcited by 124

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) with instruction fine-tuning demonstrate superior generative capabilities. However, these models are resource-intensive. To alleviate this issue, we explore distilling knowledge from instruction-tuned LLMs into much smaller ones. To this end, we carefully develop a large set of 2.58M instructions based on both existing and newly-generated instructions. In addition to being sizable, we design our instructions to cover a broad set of topics to ensure diversity. Extensive analysis of our instruction dataset confirms its diversity, and we generate responses for these instructions using gpt-3.5-turbo. Leveraging these instructions, we fine-tune a diverse herd of models, collectively referred to as LaMini-LM, which includes models from both the encoder-decoder and decoder-only families, with varying sizes. We evaluate the performance of our models using automatic metrics on 15 different natural language processing (NLP) benchmarks, as well as through human assessment. The results demonstrate that our proposed LaMini-LM models are comparable to competitive baselines, while being much smaller in size.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Builds on 45 of its 76 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

45 of 76 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
33 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  13. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  14. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  15. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  16. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  17. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  18. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  19. 2016Sequence-Level Knowledge Distillationcited by 761
  20. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  21. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  22. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  23. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  24. 2022Decoupled Knowledge Distillationcited by 380
  25. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  26. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  27. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  28. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  29. 2020Learning from Task Descriptionscited by 78
  30. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  31. 2023Phoenix: Democratizing ChatGPT across Languagescited by 32
  32. 2022ConTinTin: Continual Learning from Task Instructionscited by 26
  33. 2020Costs to Consider in Adopting NLP for Your Businesscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 40; 2024: 55; 2025: 21; 2026: 8.

124 papers.

Sort:

By area:

  1. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  2. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2023A Survey on Model Compression for Large Language Modelscited by 263
  5. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  6. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  7. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  8. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  9. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  10. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  11. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  12. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
38 more
  1. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  2. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  3. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  4. 2023A Survey of Reasoning with Foundation Modelscited by 70
  5. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  6. 2023Language Model Inversioncited by 66
  7. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  8. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  9. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
  10. 2024Adapting Large Language Models for Document-Level Machine Translationcited by 59
  11. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  12. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  13. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  14. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  15. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  16. 2024A Survey on Transformer Compressioncited by 44
  17. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  18. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  19. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  20. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  21. 2024MobiLlama: Towards Accurate and Lightweight Fully Transparent GPTcited by 34
  22. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  23. 2023Aligning Large Language Models with Human: A Surveycited by 31
  24. 2023MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Followingcited by 30
  25. 2023LLM Performance Predictors are good initializers for Architecture Searchcited by 29
  26. 2023On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?cited by 25
  27. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  28. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directionscited by 25
  29. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 25
  30. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  31. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
  32. 2023A Paradigm Shift: The Future of Machine Translation Lies with Large Language Modelscited by 19
  33. 2024Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Modelscited by 19
  34. 2023LM4HPC: Towards Effective Language Model Application in High-Performance Computingcited by 18
  35. 2023Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Explorationcited by 18
  36. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  37. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  38. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17

50 most cited of 124 All 124 (JSON)