GUIDE: Guided Initialization and Distillation of Embeddings
Not on the map. Why?
Abstract (arXiv)
Algorithmic efficiency techniques such as distillation (citehinton2015distillation) are useful in improving model quality without increasing serving costs, provided a larger teacher model is available for a smaller student model to learn from during training. Standard distillation methods are limited to only forcing the student to match the teacher's outputs. Given the costs associated with training a large model, we believe we should be extracting more useful information from a teacher model than by just making the student match the teacher's outputs. In this paper, we introduce guide (Guided Initialization and Distillation of Embeddings). guide can be considered a distillation technique that forces the student to match the teacher in the parameter space. Using guide we show 25-26% reduction in the teacher-student quality gap when using large student models (400M - 1B parameters) trained on ≈ 20B tokens. We also present a thorough analysis demonstrating that guide can be combined with knowledge distillation with near additive improvements. Furthermore, we show that applying guide alone leads to substantially better model quality than applying knowledge distillation by itself. Most importantly, guide introduces no training or inference overhead and hence any model quality gains from our method are virtually free.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 25 of its 26 references that are in the map. First posted on arXiv 7 Oct 2025.
Builds on
25 of 26 references.
References · 25
- 2017Attention Is All You Needcited by 59,932
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2014FitNets: Hints for Thin Deep Netscited by 1,619
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
- 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
- 2019The State of Sparsity in Deep Neural Networkscited by 471
13 more
- 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 427
- 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
- 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
- 2021The Efficiency Misnomercited by 98
- 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
- 2020Pre-trained Summarization Distillationcited by 70
- 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
- 2023Initializing Models with Larger Onescited by 39
- 2024Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initializationcited by 29
- 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
- 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
- 2023EmbedDistill: A Geometric Knowledge Distillation for Information Retrievalcited by 11
- 2024LAuReL: Learned Augmented Residual Layercited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.