mlsanityData to 14 Oct 2026

RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine

arXiv 2410.18097 ↗First posted 8 Oct 2024cs.IR cs.AI cs.LGcited by 6

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Large Language Models (LLMs) excel at understanding the semantic relationships between queries and documents, even with lengthy and complex long-tail queries. These queries are challenging for feedback-based rankings due to sparse user engagement and limited feedback, making LLMs' ranking ability highly valuable. However, the large size and slow inference of LLMs necessitate the development of smaller, more efficient models (sLLMs). Recently, integrating ranking label generation into distillation techniques has become crucial, but existing methods underutilize LLMs' capabilities and are cumbersome. Our research, RRADistill: Re-Ranking Ability Distillation, propose an efficient label generation pipeline and novel sLLM training methods for both encoder and decoder models. We introduce an encoder-based method using a Term Control Layer to capture term matching signals and a decoder-based model with a ranking layer for enhanced understanding. A/B testing on a Korean-based search platform, validates the effectiveness of our approach in improving re-ranking for long-tail queries.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 30 of its 48 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

30 of 48 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  9. 2019Passage Re-ranking with BERTcited by 645
  10. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  11. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  12. 2020Overview of the TREC 2019 deep learning trackcited by 286
18 more
  1. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 286
  2. 2021Overview of the TREC 2020 deep learning trackcited by 196
  3. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 155
  4. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  5. 2023Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labelscited by 86
  6. 2023Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Searchcited by 75
  7. 2022AutoField: Automating Feature Selection in Deep Recommender Systemscited by 30
  8. 2020DE-RRD: A Knowledge Distillation Framework for Recommender Systemcited by 28
  9. 2021Topology Distillation for Recommender Systemcited by 24
  10. 2024GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulationcited by 23
  11. 2023Distillation from Heterogeneous Models for Top-K Recommendationcited by 13
  12. 2024Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomycited by 11
  13. 2024TWOLAR: a TWO-step LLM-Augmented distillation method for passage Rerankingcited by 11
  14. 2021Dual Correction Strategy for Ranking Distillation in Top-N Recommender Systemcited by 10
  15. 2023MvFS: Multi-view Feature Selection for Recommender Systemcited by 10
  16. 2023Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrievalcited by 7
  17. 2024Continual Collaborative Distillation for Recommender Systemcited by 6
  18. 2022Consensus Learning from Heterogeneous Objectives for One-Class Collaborative Filteringcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2024Taxonomy-guided Semantic Indexing for Academic Paper Searchcited by 12
  2. 2025Reference-Aligned Retrieval-Augmented Question Answering over Heterogeneous Proprietary Documentscited by 1
  3. 2025IRA: Adaptive Interest-aware Representation and Alignment for Personalized Multi-interest Retrievalcited by 0
  4. 2026Fine-tuning Small Language Models as Efficient Enterprise Search Relevance Labelerscited by 0
  5. 2026Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgmentscited by 0
  6. 2026PETRA: Transforming Web Text for Petroleum-Engineering Domain Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval
  2. 2026LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systems
  3. 2024RaCT: Ranking-aware Chain-of-Thought Optimization for LLMs
  4. 2025LLM4Ranking: An Easy-to-use Framework of Utilizing Large Language Models for Document Reranking
  5. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
  6. 2025RankLLM: A Python Package for Reranking with LLMs
  7. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning
  8. 2023Instruction Distillation Makes Large Language Models Efficient Zero-shot Rankers