Data to 7 Oct 2026

Distilling Knowledge from Reader to Retriever for Question Answering

arXiv 2012.04584 ↗First posted 8 Dec 2020cs.CL cs.LGcited by 226

Not on the map. Why?

Cited by 61 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The task of information retrieval is an important component of many natural language processing systems, such as open domain question answering. While traditional methods were based on hand-crafted features, continuous representations based on neural networks recently obtained competitive results. A challenge of using such methods is to obtain supervised data to train the retriever model, corresponding to pairs of query and support documents. In this paper, we propose a technique to learn retriever models for downstream tasks, inspired by knowledge distillation, and which does not require annotated pairs of query and documents. Our approach leverages attention scores of a reader model, used to solve the task based on retrieved documents, to obtain synthetic labels for the retriever. We evaluate our method on question answering, obtaining state-of-the-art results.

Key facts. Cited by 226 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 39; 2023: 49; 2024: 61; 2025: 28; 2026: 17.

226 papers.

Sort:

By area:

2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
2022Training Compute-Optimal Large Language Modelscited by 2,927
2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
2021Recursively Summarizing Books with Human Feedbackcited by 287
2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204

38 more

2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 118
2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 104
2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentationcited by 98
2024Evaluating Retrieval Quality in Retrieval-Augmented Generationcited by 90
2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
2023Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-Incited by 71
2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
2022Recitation-Augmented Language Modelscited by 65
2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
2021MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documentscited by 53
2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
2021One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrievalcited by 46
2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 42
2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
2021Evidentiality-guided Generation for Knowledge-Intensive NLP Taskscited by 40
2025Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learningcited by 39
2021Hindsight: Posterior-guided training of retrievers for improved open-ended generationcited by 38
2021R2-D2: A Modular Baseline for Open-Domain Question Answeringcited by 37
2023Merging Generated and Retrieved Knowledge for Open-Domain QAcited by 36
2023Evaluating Open-QA Evaluationcited by 35
2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
2021Phrase Retrieval Learns Passage Retrieval, Toocited by 33

50 most cited of 226 All 226 (JSON)