mlsanityData to 14 Oct 2026

BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data

arXiv 2409.17312 ↗First posted 25 Sep 2024cs.CL cs.LGcited by 5

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

We present BabyLlama-2, a 345 million parameter model distillation-pretrained from two teachers on a 10 million word corpus for the BabyLM competition. On BLiMP and SuperGLUE benchmarks, BabyLlama-2 outperforms baselines trained on both 10 and 100 million word datasets with the same data mix, as well as its teacher models. Through an extensive hyperparameter sweep, we demonstrate that the advantages of distillation cannot be attributed to suboptimal hyperparameter selection of the teachers. Our findings underscore the need for further investigation into distillation techniques, particularly in data-limited settings.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 14 of its 21 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

14 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty3 Aug

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  7. 2018Born Again Neural Networkscited by 546
  8. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  9. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  10. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  11. 2023FinGPT: Large Generative Models for a Small Languagecited by 49
  12. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
1 more
  1. 2024[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 26

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  2. 2024A Survey of Small Language Modelscited by 31
  3. 2025Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)cited by 11
  4. 2025Pretraining Language Models for Diachronic Linguistic Change Discoverycited by 6
  5. 2025CoSMoEs: Compact Sparse Mixture of Expertscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models
  2. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
  4. 2024When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?
  5. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
  6. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Models
  7. 2024Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights
  8. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models