# Zero-Shot Contextual Embeddings via Offline Synthetic Corpus Generation

arXiv: https://arxiv.org/abs/2506.23662 · HTML: https://mlsanity.com/paper/2506.23662/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 25 of 25)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://mlsanity.com/paper/2402.01613/) (2024)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval](https://mlsanity.com/paper/2112.07577/) (2021)
- [TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning](https://mlsanity.com/paper/2104.06979/) (2021)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval](https://mlsanity.com/paper/2207.02578/) (2022)
- [GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning](https://mlsanity.com/paper/2402.16829/) (2024)
- [Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval](https://mlsanity.com/paper/2106.11251/) (2021)
- [LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval](https://mlsanity.com/paper/2203.06169/) (2022)
- [Contextual Document Embeddings](https://mlsanity.com/paper/2410.02525/) (2024)
- [NPRF: A Neural Pseudo Relevance Feedback Framework for Ad-hoc Information Retrieval](https://mlsanity.com/paper/1810.12936/) (2018)
- [ANCHOR: logically-centralized security for Software-Defined Networks](https://mlsanity.com/paper/1711.03636/) (2017)
- [Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval](https://mlsanity.com/paper/2311.15564/) (2023)

## Abstract (arXiv)

Context-aware embedding methods boost retrieval accuracy by conditioning on corpus statistics (e.g., term co-occurrence and topical patterns) extracted from neighboring documents. However, this context-aware approach requires access to the target corpus or requires domain-specific finetuning, posing practical barriers in privacy-sensitive or resource-constrained settings. We present ZEST, a zero-shot contextual adaptation framework that replaces real corpus access with a one-time offline synthesis of a compact proxy. Given only a handful exemplar documents representative of the general target domain, we use a multi-step hierarchical procedure to generate a synthetic context corpus of several hundred documents that aims to emulate key domain-specific distributions. At inference, the frozen context-aware encoder uses this proxy corpus -- without any finetuning or target corpus access -- to produce domain-adapted embeddings. Across the MTEB benchmark, ZEST's zero-shot synthetic context adaptation using only five example documents performs within 0.5% of models leveraging full target corpus access -- demonstrating remarkable efficacy without any retraining. ZEST thus provides a practical method for deploying high-performance, adaptable embeddings in constrained environments.