# Adapting General-Purpose Embedding Models to Private Datasets Using Keyword-based Retrieval

arXiv: https://arxiv.org/abs/2506.00363 · HTML: https://mlsanity.com/paper/2506.00363/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 31 May 2025.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain](https://mlsanity.com/paper/2408.10343/) (2024)
- [NV-Retriever: Improving text embedding models with effective hard-negative mining](https://mlsanity.com/paper/2407.15831/) (2024)
- [Neural Ranking Models with Weak Supervision](https://mlsanity.com/paper/1704.08803/) (2017)
- [Debiased Contrastive Learning of Unsupervised Sentence Representations](https://mlsanity.com/paper/2205.00656/) (2022)
- [Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations](https://mlsanity.com/paper/2110.07581/) (2021)
- [Pooling And Attention: What Are Effective Designs For LLM-Based Embedding Models?](https://mlsanity.com/paper/2409.02727/) (2024)
- [Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives](https://mlsanity.com/paper/2403.14341/) (2024)
- [Do We Need Domain-Specific Embedding Models? An Empirical Investigation](https://mlsanity.com/paper/2409.18511/) (2024)
- [Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt](https://mlsanity.com/paper/2411.07142/) (2024)
- [A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings](https://mlsanity.com/paper/2203.05877/) (2022)
- [Learning More From Less: Towards Strengthening Weak Supervision for Ad-Hoc Retrieval](https://mlsanity.com/paper/1907.08657/) (2019)

## Built on by (most cited first, 2 of 2)

- [RAG Meets Temporal Graphs: Time-Sensitive Modeling and Retrieval for Evolving Knowledge](https://mlsanity.com/paper/2510.13590/) (2025)
- [GAPrune: Gradient-Alignment Pruning for Domain-Aware Embeddings](https://mlsanity.com/paper/2509.10844/) (2025)

## Abstract (arXiv)

Text embedding models play a cornerstone role in AI applications, such as retrieval-augmented generation (RAG). While general-purpose text embedding models demonstrate strong performance on generic retrieval benchmarks, their effectiveness diminishes when applied to private datasets (e.g., company-specific proprietary data), which often contain specialized terminology and lingo. In this work, we introduce BMEmbed, a novel method for adapting general-purpose text embedding models to private datasets. By leveraging the well-established keyword-based retrieval technique (BM25), we construct supervisory signals from the ranking of keyword-based retrieval results to facilitate model adaptation. We evaluate BMEmbed across a range of domains, datasets, and models, showing consistent improvements in retrieval performance. Moreover, we provide empirical insights into how BM25-based signals contribute to improving embeddings by fostering alignment and uniformity, highlighting the value of this approach in adapting models to domain-specific data. We release the source code available at https://github.com/BaileyWei/BMEmbed for the research community.