mlsanityData to 7 Oct 2026

LLM2IR: simple unsupervised contrastive learning makes long-context LLM great retriever

arXiv 2601.05262 ↗First posted 31 Oct 2025cs.IR cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern dense information retrieval (IR) models usually rely on costly large-scale pretraining. In this paper, we introduce LLM2IR, an efficient unsupervised contrastive learning framework to convert any decoder-only large language model (LLM) to an information retrieval model. Despite its simplicity, the effectiveness is proven among different LLMs on multiple IR benchmarks including LoCo, LongEmbed and BEIR. We also find that models with a longer context length tend to have a stronger IR capacity by comparing task performances of models in the same model family. Our work not only provides an effective way to build IR models on the state-of-the-art LLMs, but also shed light on the relationship between information retrieval ability and model context length, which helps the design of better information retrievers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

31 of 39 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2023Mistral 7Bcited by 3,065
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
19 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  4. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  5. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  6. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  7. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  8. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  9. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  10. 2023Improving Text Embeddings with Large Language Modelscited by 295
  11. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  12. 2024Generative Representational Instruction Tuningcited by 227
  13. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  14. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  15. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  16. 2020SparTerm: Learning Term-based Sparse Representation for Fast Text Retrievalcited by 45
  17. 2024Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTcited by 21
  18. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21
  19. 2024Learning by Analogy: Enhancing Few-Shot Prompting for Math Word Problem Solving with Computational Graph-Based Retrievalcited by 4

Built on by

Not cited yet.