mlsanityData to 7 Oct 2026

LLM-Oriented Retrieval Tuner

arXiv 2403.01999 ↗First posted 4 Mar 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Dense Retrieval (DR) is now considered as a promising tool to enhance the memorization capacity of Large Language Models (LLM) such as GPT3 and GPT-4 by incorporating external memories. However, due to the paradigm discrepancy between text generation of LLM and DR, it is still an open challenge to integrate the retrieval and generation tasks in a shared LLM. In this paper, we propose an efficient LLM-Oriented Retrieval Tuner, namely LMORT, which decouples DR capacity from base LLM and non-invasively coordinates the optimally aligned and uniform layers of the LLM towards a unified DR space, achieving an efficient and effective DR without tuning the LLM itself. The extensive experiments on six BEIR datasets show that our approach could achieve competitive zero-shot retrieval performance compared to a range of strong DR models while maintaining the generation ability of LLM.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

24 of 37 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  11. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  12. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
12 more
  1. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  2. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  3. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  4. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  5. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  6. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  7. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  8. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  9. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  10. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  11. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
  12. 2022Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negativescited by 5

Built on by

Not cited yet.