# GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation

arXiv: https://arxiv.org/abs/2403.19754 · HTML: https://mlsanity.com/paper/2403.19754/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [The False Promise of Imitating Proprietary LLMs](https://mlsanity.com/paper/2305.15717/) (2023)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions](https://mlsanity.com/paper/2303.08233/) (2023)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [Prompt2Model: Generating Deployable Models from Natural Language Instructions](https://mlsanity.com/paper/2308.12261/) (2023)
- [E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Models](https://mlsanity.com/paper/2203.00748/) (2022)
- [EBJR: Energy-Based Joint Reasoning for Adaptive Inference](https://mlsanity.com/paper/2110.10343/) (2021)

## Built on by (most cited first, 6 of 6)

- [Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes](https://mlsanity.com/paper/2509.06266/) (2025)
- [textitTrans-LoRA: towards data-free Transferable Parameter Efficient Finetuning](https://mlsanity.com/paper/2405.17258/) (2024)
- [Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memes](https://mlsanity.com/paper/2411.12174/) (2024)
- [CASP: Compression of Large Multimodal Models Based on Attention Sparsity](https://mlsanity.com/paper/2503.05936/) (2025)
- [Multilinguality at the Edge: Developing Language Models for the Global South](https://mlsanity.com/paper/2604.21637/) (2026)
- [Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer](https://mlsanity.com/paper/2609.22566/) (2026)

## Abstract (arXiv)

Knowledge distillation from LLMs is essential for the efficient deployment of language models. Prior works have proposed data generation using LLMs for preparing distilled models. We argue that generating data with LLMs is prone to sampling mainly from the center of original content distribution. This limitation hinders the distilled model from learning the true underlying data distribution and to forget the tails of the distributions (samples with lower probability). To this end, we propose GOLD, a task-agnostic data generation and knowledge distillation framework, which employs an iterative out-of-distribution-guided feedback mechanism for the LLM. As a result, the generated data improves the generalizability of distilled models. An energy-based OOD evaluation approach is also introduced to deal with noisy generated data. Our extensive experiments on 10 different classification and sequence-to-sequence tasks in NLP show that GOLD respectively outperforms prior arts and the LLM with an average improvement of 5% and 14%. We will also show that the proposed method is applicable to less explored and novel tasks. The code is available.