# Embedding And Clustering Your Data Can Improve Contrastive Pretraining

arXiv: https://arxiv.org/abs/2407.18887 · HTML: https://mlsanity.com/paper/2407.18887/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 26 Jul 2024.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://mlsanity.com/paper/2402.01613/) (2024)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [Mini-Batch Optimization of Contrastive Loss](https://mlsanity.com/paper/2307.05906/) (2023)

## Built on by (most cited first, 17 of 17)

- [Improved Large Language Model Jailbreak Detection via Pretrained Embeddings](https://mlsanity.com/paper/2412.01547/) (2024)
- [Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models](https://mlsanity.com/paper/2506.03989/) (2025)
- [Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora](https://mlsanity.com/paper/2509.08824/) (2025)
- [DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization](https://mlsanity.com/paper/2505.10989/) (2025)
- [LexSemBridge: Fine-Grained Dense Representation Enhancement through Token-Aware Embedding Augmentation](https://mlsanity.com/paper/2508.17858/) (2025)
- [Learning Task-Agnostic Representations through Multi-Teacher Distillation](https://mlsanity.com/paper/2510.18680/) (2025)
- [DisastIR: A Comprehensive Information Retrieval Benchmark for Disaster Management](https://mlsanity.com/paper/2505.15856/) (2025)
- [Overview of the TREC 2025 Retrieval Augmented Generation (RAG) Track](https://mlsanity.com/paper/2603.09891/) (2026)
- [OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameters](https://mlsanity.com/paper/2409.00286/) (2024)
- [LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention](https://mlsanity.com/paper/2607.11976/) (2026)
- [ICLERB: In-Context Learning Embedding and Reranker Benchmark](https://mlsanity.com/paper/2411.18947/) (2024)
- [SHRAG: AFrameworkfor Combining Human-Inspired Search with RAG](https://mlsanity.com/paper/2512.00772/) (2025)
- [DARE: Aligning LLM Agents with the R Statistical Ecosystem via Distribution-Aware Retrieval](https://mlsanity.com/paper/2603.04743/) (2026)
- [Is a Document Educational or Just Wikipedia-Style? -- Pitfalls of Classifier-Based Quality Filtering](https://mlsanity.com/paper/2605.23721/) (2026)
- [Plan-and-Refine: Diverse and Comprehensive Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.07794/) (2025)
- [GraphMatch: Fusing Language and Graph Representations in a Dynamic Two-Sided Work Marketplace](https://mlsanity.com/paper/2512.02849/) (2025)
- [EmbeddingRWKV: State-Centric Retrieval with Reusable States](https://mlsanity.com/paper/2601.07861/) (2026)

## Abstract (arXiv)

Recent studies of large-scale contrastive pretraining in the text embedding domain show that using single-source minibatches, rather than mixed-source minibatches, can substantially improve overall model accuracy. In this work, we explore extending training data stratification beyond source granularity by leveraging a pretrained text embedding model and the classic k-means clustering algorithm to further split training data apart by the semantic clusters within each source. Experimentally, we observe a notable increase in NDCG@10 when pretraining a BERT-based text embedding model on query-passage pairs from the MSMARCO passage retrieval dataset. Additionally, we conceptually connect our clustering approach to both the Topic Aware Sampling (TAS) aspect of the TAS-B methodology and the nearest-neighbor-based hard-negative mining aspect of the ANCE methodology and discuss how this unified view motivates future lines of research on the organization of contrastive pretraining data.