mlsanityData to 7 Oct 2026

D2LLM: Decomposed and Distilled Large Language Models for Semantic Search

arXiv 2406.17262 ↗First posted 25 Jun 2024cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

The key challenge in semantic search is to create models that are both accurate and efficient in pinpointing relevant sentences for queries. While BERT-style bi-encoders excel in efficiency with pre-computed embeddings, they often miss subtle nuances in search tasks. Conversely, GPT-style LLMs with cross-encoder designs capture these nuances but are computationally intensive, hindering real-time applications. In this paper, we present D2LLMs-Decomposed and Distilled LLMs for semantic search-that combines the best of both worlds. We decompose a cross-encoder into an efficient bi-encoder integrated with Pooling by Multihead Attention and an Interaction Emulation Module, achieving nuanced understanding and pre-computability. Knowledge from the LLM is distilled into this model using contrastive, rank, and feature imitation techniques. Our experiments show that D2LLM surpasses five leading baselines in terms of all metrics across three tasks, particularly improving NLI task performance by at least 6.45%. The source code is available at https://github.com/codefuse-ai/D2LLM.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 40 of its 54 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

40 of 54 references.

References · 40

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
28 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  3. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  4. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  5. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  6. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  7. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  8. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  9. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  10. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  11. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  12. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  13. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  14. 2022Knowledge Distillation from A Stronger Teachercited by 204
  15. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  16. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  17. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  18. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  19. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  20. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  21. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 77
  22. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 59
  23. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  24. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  25. 2020Is Retriever Merely an Approximator of Reader?cited by 30
  26. 2022In Defense of Cross-Encoders for Zero-Shot Retrievalcited by 21
  27. 2017Specialising Word Vectors for Lexical Entailmentcited by 18
  28. 2022No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrievalcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 2.

9 papers.

  1. 2025DualRAG: A Dual-Process Approach to Integrate Reasoning and Retrieval for Multi-Hop Question Answeringcited by 20
  2. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual Worldcited by 10
  3. 2025F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Datacited by 8
  4. 2024Towards Boosting LLMs-driven Relevance Modeling with Progressive Retrieved Behavior-augmented Promptingcited by 5
  5. 2024LLM Distillation for Efficient Few-Shot Multiple Choice Question Answeringcited by 3
  6. 2025C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Poolingcited by 3
  7. 2025LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations at eBaycited by 2
  8. 2025LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selectioncited by 2
  9. 2026TingIS: Real-time Risk Event Discovery from Noisy Customer Incidents at Enterprise Scalecited by 0