mlsanityData to 14 Oct 2026

Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty

arXiv 2308.02019 ↗First posted 3 Aug 2023cs.CLcited by 57

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

We present our submission to the BabyLM challenge, whose goal was to improve the sample efficiency of language models. We trained an ensemble consisting of a GPT-2 and small LLaMA models on the developmentally-plausible, 10M-word BabyLM dataset, then distilled it into a small, 58M-parameter LLaMA model, which exceeds in performance both of its teachers as well as a similar model trained without distillation. This suggests that distillation can not only retain the full performance of the teacher model when the latter is trained on a sufficiently small dataset; it can exceed it, and lead to significantly better performance than direct training.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 16 of its 34 references that are in the map. First posted on arXiv 3 Aug 2023.

Builds on

16 of 34 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  11. 2022Galactica: A Large Language Model for Sciencecited by 744
  12. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
4 more
  1. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  2. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  3. 2021Do Data-based Curricula Work?cited by 3
  4. 2023Gradient Ascent Post-training Enhances Language Model Generalizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 27; 2025: 27; 2026: 2.

57 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  3. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  5. 2023Efficient Large Language Models: A Surveycited by 186
  6. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  7. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  8. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 63
  9. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
  10. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  11. 2024A Survey of Small Language Modelscited by 31
  12. 2024GPT or BERT: why not both?cited by 27
38 more
  1. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 26
  2. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  3. 2024Squat: Quant Small Language Models on the Edgecited by 16
  4. 2024From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemescited by 11
  5. 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Modelscited by 11
  6. 2025GLaD: Geometric Latent Distillation for Vision-Language-Action Modelscited by 11
  7. 2024Understanding Large Language Models in Your Pockets: Performance Study on COTS Mobile Devicescited by 10
  8. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 10
  9. 2025Teacher Demonstrations in a BabyLM's Zone of Proximal Development for Contingent Multi-Turn Interactioncited by 9
  10. 2024AntLM: Bridging Causal and Masked Language Modelscited by 8
  11. 2024Emergent Abilities in Reduced-Scale Generative Language Modelscited by 7
  12. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thoughtcited by 7
  13. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  14. 2024BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Datacited by 5
  15. 2024Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamascited by 5
  16. 2025InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusioncited by 5
  17. 2024Proofread: Fixes All Errors with One Tapcited by 4
  18. 2024What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performancecited by 4
  19. 2024Explainable LLM-driven Multi-dimensional Distillation for E-Commerce Relevance Learningcited by 4
  20. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  21. 2025ExpertSteer: Intervening in LLMs through Expert Knowledgecited by 4
  22. 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
  23. 2025The Curious Case of In-Training Compression of State Space Modelscited by 4
  24. 2024Aligning Teacher with Student Preferences for Tailored Training Data Generationcited by 3
  25. 2024BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-trainingcited by 3
  26. 2025GREEN-CODE: Learning to Optimize Energy Efficiency in LLM-based Code Generationcited by 3
  27. 2025CoSMoEs: Compact Sparse Mixture of Expertscited by 3
  28. 2025Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matchingcited by 3
  29. 2025Influence-driven Curriculum Learning for Pre-training on Limited Datacited by 2
  30. 2025Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modellingcited by 2
  31. 2025When LLMs Can't Help: Real-World Evaluation of LLMs in Nutritioncited by 2
  32. 2025iServe: An Intent-based Serving System for LLMscited by 1
  33. 2025Pastiche Novel Generation Creating: Fan Fiction You Love in Your Favorite Author's Stylecited by 1
  34. 2025UNDO: Understanding Distillation as Optimizationcited by 1
  35. 2025Metropolis-Hastings Captioning Game: Knowledge Fusion of Vision Language Models via Decentralized Bayesian Inferencecited by 1
  36. 2025On Membership Inference Attacks in Knowledge Distillationcited by 1
  37. 2025Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Modelscited by 1
  38. 2025FrugalPrompt: Reducing Contextual Overhead in Large Language Models via Token Attributioncited by 1

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models
  2. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data
  3. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Models
  4. 2024Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data
  5. 2024Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting
  6. 2025BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learning
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
  8. 2025BAMBI: Developing Baby Language Models for Italian