mlsanityData to 7 Oct 2026

Training Task Experts through Retrieval Based Distillation

arXiv 2407.05463 ↗First posted 7 Jul 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

One of the most reliable ways to create deployable models for specialized tasks is to obtain an adequate amount of high-quality task-specific data. However, for specialized tasks, often such datasets do not exist. Existing methods address this by creating such data from large language models (LLMs) and then distilling such knowledge into smaller models. However, these methods are limited by the quality of the LLMs output, and tend to generate repetitive or incorrect data. In this work, we present Retrieval Based Distillation (ReBase), a method that first retrieves data from rich online sources and then transforms them into domain-specific data. This method greatly enhances data diversity. Moreover, ReBase generates Chain-of-Thought reasoning and distills the reasoning capacity of LLMs. We test our method on 4 benchmarks and results show that our method significantly improves performance by up to 7.8% on SQuAD, 1.37% on MNLI, and 1.94% on BigBench-Hard.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 7 Jul 2024.

Builds on

31 of 42 references.

References · 31

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  12. 2021Datasets: A Community Library for Natural Language Processingcited by 526
19 more
  1. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  2. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  3. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  4. 2022Large Language Models Are Reasoning Teacherscited by 318
  5. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  6. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  7. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  8. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  9. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  10. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  11. 2024DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflowscited by 43
  12. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  13. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40
  14. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasetscited by 36
  15. 2023Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasingcited by 31
  16. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  17. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  18. 2024FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Modelcited by 18
  19. 2023IAG: Induction-Augmented Generation Framework for Answering Reasoning Questionscited by 15

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  2. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  3. 2024CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharingcited by 12

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  2. 2024Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval
  3. 2025Not All Correct Answers Are Equal: Why Your Distillation Source Matters
  4. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data
  5. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks
  6. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  7. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation
  8. 2024Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation