mlsanityData to 7 Oct 2026

Fine-Grained Distillation for Long Document Retrieval

arXiv 2212.10423 ↗First posted 20 Dec 2022cs.IR cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the scope hypothesis that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 44 of its 55 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

44 of 55 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  10. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  11. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  12. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
32 more
  1. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  2. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  3. 2020Overview of the TREC 2019 deep learning trackcited by 282
  4. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  5. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  6. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  7. 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 164
  8. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  9. 2020Embedding-based Retrieval in Facebook Searchcited by 155
  10. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  11. 2020Learning Dense Representations of Phrases at Scalecited by 91
  12. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  13. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  14. 2017DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understandingcited by 71
  15. 2021Few-Shot Conversational Dense Retrievalcited by 71
  16. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  17. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  18. 2021Semantic Models for the First-stage Retrieval: A Comprehensive Reviewcited by 57
  19. 2020RepBERT: Contextualized Text Embeddings for First-Stage Retrievalcited by 46
  20. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  21. 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
  22. 2022Towards Robust Ranker for Text Retrievalcited by 37
  23. 2021Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performancecited by 35
  24. 2020Complementing Lexical Retrieval with Semantic Residual Embeddingcited by 27
  25. 2021Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrievalcited by 27
  26. 2021B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 25
  27. 2022Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Predictioncited by 23
  28. 2021Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillationscited by 20
  29. 2022LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrievalcited by 18
  30. 2020Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehensioncited by 17
  31. 2022SeDR: Segment Representation Learning for Long Documents Dense Retrievalcited by 6
  32. 2022Multi-Granularity Cross-Modality Representation Learning for Named Entity Recognition on Social Mediacited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 16; 2025: 12.

29 papers.

Sort:

By area:

  1. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  2. 2024Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent Systemcited by 63
  3. 2024MemoryMamba: Memory-Augmented State Space Model for Defect Recognitioncited by 24
  4. 2024Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Modelscited by 15
  5. 2024Exploiting Diffusion Prior for Out-of-Distribution Detectioncited by 7
  6. 2025An LLM-LVLM Driven Agent for Iterative and Fine-Grained Image Editingcited by 6
  7. 2025Dynamic Knowledge Exchange and Dual-diversity Review: Concisely Unleashing the Potential of a Multi-Agent Research Teamcited by 4
  8. 2025Interpretable and Robust Dialogue State Tracking via Natural Language Summarization with LLMscited by 3
  9. 2025Attention Grounded Enhancement for Visual Document Retrievalcited by 3
  10. 2024Optimizing Psychological Counseling with Instruction-Tuned Large Language Modelscited by 2
  11. 2024Improving Visual Storytelling with Multimodal Large Language Modelscited by 2
  12. 2024Large Vision-Language Models for Remote Sensing Visual Question Answeringcited by 2
17 more
  1. 2024End-To-End Training and Testing Gamification Framework to Learn Human Highway Drivingcited by 1
  2. 2024Enhancing Agricultural Machinery Management through Advanced LLM Integrationcited by 1
  3. 2024Coal Mining Question Answering with LLMscited by 1
  4. 2024LLMs for Enhanced Agricultural Meteorological Recommendationscited by 0
  5. 2024Enhancing AI-Driven Psychological Consultation: Layered Prompts with Large Language Modelscited by 0
  6. 2024Consultation on Industrial Machine Faults with Large language Modelscited by 0
  7. 2024NexusIndex: Integrating Advanced Vector Indexing and Multi-Model Embeddings for Robust Fake News Detectioncited by 0
  8. 2024Leveraging Large Language Models for Comparative Literature Summarization with Reflective Incremental Mechanismscited by 0
  9. 2024Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenescited by 0
  10. 2025Cross-Domain Semantic Segmentation with Large Language Model-Assisted Descriptor Generationcited by 0
  11. 2025Weak Supervision Dynamic KL-Weighted Diffusion Models Guided by Large Language Modelscited by 0
  12. 2025Multi-granular Training Strategies for Robust Multi-hop Reasoning Over Noisy and Heterogeneous Knowledge Sourcescited by 0
  13. 2025Enhancing Contrastive Demonstration Selection with Semantic Diversity for Robust In-Context Machine Translationcited by 0
  14. 2025Harnessing Generative LLMs for Enhanced Financial Event Entity Extraction Performancecited by 0
  15. 2025Contextual Candor: Enhancing LLM Trustworthiness Through Hierarchical Unanswerability Detectioncited by 0
  16. 2025VisuCraft: Enhancing Large Vision-Language Models for Complex Visual-Guided Creative Content Generation via Structured Information Extractioncited by 0
  17. 2025VL-MedGuide: A Visual-Linguistic Large Model for Intelligent and Explainable Skin Disease Auxiliary Diagnosiscited by 0