mlsanityData to 7 Oct 2026

Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval

arXiv 2406.12169 ↗First posted 18 Jun 2024cs.IRcited by 3

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Recent research has explored distilling knowledge from large language models (LLMs) to optimize retriever models, especially within the retrieval-augmented generation (RAG) framework. However, most existing training methods rely on extracting supervision signals from LLMs' weights or their output probabilities, which is not only resource-intensive but also incompatible with black-box LLMs. In this paper, we introduce textitIntermediate Distillation, a data-efficient knowledge distillation training scheme that treats LLMs as black boxes and distills their knowledge via an innovative LLM-ranker-retriever pipeline, solely using LLMs' ranking generation as the supervision signal. Extensive experiments demonstrate that our proposed method can significantly improve the performance of retriever models with only 1,000 training instances. Moreover, our distilled retriever model significantly boosts performance in question-answering tasks within the RAG framework, demonstrating the potential of LLMs to economically and effectively train smaller models.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 30 of its 42 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

30 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval23 Jun
  2. 2023In-Context Retrieval-Augmented Language Models31 Jan
  3. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2023BloombergGPT: A Large Language Model for Financecited by 923
  10. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  11. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  12. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
13 more
  1. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  2. 2022Large Language Models Are Reasoning Teacherscited by 318
  3. 2022Teaching language models to support answers with verified quotescited by 290
  4. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  5. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  6. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  7. 2022Training Language Models with Memory Augmentationcited by 112
  8. 2023Extractive Summarization via ChatGPT for Faithful Summary Generationcited by 58
  9. 2024UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systemscited by 38
  10. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  11. 2022Metric-guided Distillation: Distilling Knowledge from the Metric to Ranker and Retriever for Generative Commonsense Reasoningcited by 8
  12. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  13. 2023Efficient Transformer Knowledge Distillation: A Performance Reviewcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answeringcited by 6
  2. 2025ReSCORE: Label-free Iterative Retriever Training for Multi-hop Question Answering with Relevance-Consistency Supervisioncited by 4
  3. 2025Attention Grounded Enhancement for Visual Document Retrievalcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Training Task Experts through Retrieval Based Distillation
  2. 2022QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation
  3. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs
  4. 2024Unveiling the Magic: Investigating Attention Distillation in Retrieval-augmented Generation
  5. 2024Query Optimization for Parametric Knowledge Refinement in Retrieval-Augmented Large Language Models
  6. 2025Post-training an LLM for RAG? Train on Self-Generated Demonstrations
  7. 2024Efficient Knowledge Injection in LLMs via Self-Distillation
  8. 2024KBM: Delineating Knowledge Boundary for Adaptive Retrieval in Large Language Models