Data to 14 Oct 2026

SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

arXiv 2207.02578 ↗First posted 6 Jul 2022cs.IRcited by 90

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA, to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries. We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings. Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more storage cost. Our code and model check points are available at https://github.com/microsoft/unilm/tree/master/simlm .

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Jul 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 23; 2024: 31; 2025: 19; 2026: 7.

90 papers.

Sort:

By area:

2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
2023Text Embeddings Reveal (Almost) As Much As Textcited by 192
2023Query2doc: Query Expansion with Large Language Modelscited by 149
2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 64
2023SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrievalcited by 62
2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 48

38 more

2023Learning to Rank in Generative Retrievalcited by 46
2023Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Rankingcited by 44
2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
2022Fine-Grained Distillation for Long Document Retrievalcited by 29
2024BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Supporting Evidencecited by 28
2025RaDeR: Reasoning-aware Dense Retrieval Modelscited by 27
2023Large Language Models are Strong Zero-Shot Retrievercited by 26
2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
2023Large Search Model: Redefining Search Stack in the Era of LLMscited by 18
2024CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Taskscited by 18
2023Multiview Identifiers Enhanced Generative Retrievalcited by 16
2024Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAGcited by 16
2024BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Modelscited by 15
2025O1 Embedder: Let Retrievers Think Before Actioncited by 15
2023RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Modelscited by 13
2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12
2023DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queriescited by 11
2025Unsupervised Corpus Poisoning Attacks in Continuous Space for Dense Retrievalcited by 11
2026Agentic-R: Learning to Retrieve for Agentic Searchcited by 11
2024State Space Models are Strong Text Rerankerscited by 10
2025CSPLADE: Learned Sparse Retrieval with Causal Language Modelscited by 10
2022Query-as-context Pre-training for Dense Passage Retrievalcited by 9
2024Understanding and Mitigating the Threat of Vec2Text to Dense Retrieval Systemscited by 9
2024DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignmentcited by 9
2024Fine-Tuning Large Language Models for Stock Return Prediction Using Newsflowcited by 8
2025Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrievalcited by 8
2022MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrieverscited by 7
2023CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrievalcited by 7
2023Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrievalcited by 7
2023Selecting which Dense Retriever to use for Zero-Shot Searchcited by 7
2024LocalRQA: From Generating Data to Locally Training, Testing, and Deploying Retrieval-Augmented QA Systemscited by 7
2025Hypencoder: Hypernetworks for Information Retrievalcited by 7
2025Beyond Chunking: Discourse-Aware Hierarchical Retrieval for Long Document Question Answeringcited by 7
2024UniICL: An Efficient Unified Framework Unifying Compression, Selection, and Generationcited by 6
2025Rethinking the Privacy of Text Embeddings: A Reproducibility Study of "Text Embeddings Reveal (Almost) As Much As Text"cited by 6
2025Distillation versus Contrastive Learning: How to Train Your Rerankerscited by 6
2025Training LLMs to be Better Text Embedders through Bidirectional Reconstructioncited by 6

50 most cited of 90 All 90 (JSON)