# On Synthetic Data Strategies for Domain-Specific Generative Retrieval

arXiv: https://arxiv.org/abs/2502.17957 · HTML: https://mlsanity.com/paper/2502.17957/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [Iterative Reasoning Preference Optimization](https://mlsanity.com/paper/2404.19733/) (2024)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval](https://mlsanity.com/paper/2112.07577/) (2021)
- [Autoregressive Search Engines: Generating Substrings as Document Identifiers](https://mlsanity.com/paper/2204.10628/) (2022)
- [We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://mlsanity.com/paper/2010.05338/) (2020)
- [InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval](https://mlsanity.com/paper/2301.01820/) (2023)
- [Learning to Tokenize for Generative Retrieval](https://mlsanity.com/paper/2304.04171/) (2023)
- [Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation](https://mlsanity.com/paper/2206.10128/) (2022)
- [How Does Generative Retrieval Scale to Millions of Passages?](https://mlsanity.com/paper/2305.11841/) (2023)
- [Learning to Rank in Generative Retrieval](https://mlsanity.com/paper/2306.15222/) (2023)
- [Scalable and Effective Generative Information Retrieval](https://mlsanity.com/paper/2311.09134/) (2023)
- [CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks](https://mlsanity.com/paper/2208.07652/) (2022)
- [Ultron: An Ultimate Retriever on Corpus with a Model-based Indexer](https://mlsanity.com/paper/2208.09257/) (2022)
- [Large Language Models are Built-in Autoregressive Search Engines](https://mlsanity.com/paper/2305.09612/) (2023)
- [Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies](https://mlsanity.com/paper/2305.15115/) (2023)
- [CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks](https://mlsanity.com/paper/2402.01176/) (2024)
- [GLEN: Generative Retrieval via Lexical Index Learning](https://mlsanity.com/paper/2311.03057/) (2023)
- [A Unified Generative Retriever for Knowledge-Intensive Language Tasks via Prompt Learning](https://mlsanity.com/paper/2304.14856/) (2023)
- [Auto Search Indexer for End-to-End Document Retrieval](https://mlsanity.com/paper/2310.12455/) (2023)
- [Generative Retrieval via Term Set Generation](https://mlsanity.com/paper/2305.13859/) (2023)

## Built on by (most cited first, 5 of 5)

- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models](https://mlsanity.com/paper/2511.07581/) (2025)
- [Revela: Dense Retriever Learning via Language Modeling](https://mlsanity.com/paper/2506.16552/) (2025)
- [From Relevance to Authority: Authority-aware Generative Retrieval in Web Search Engines](https://mlsanity.com/paper/2604.13468/) (2026)
- [Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA](https://mlsanity.com/paper/2603.24580/) (2026)

## Abstract (arXiv)

This paper investigates synthetic data generation strategies in developing generative retrieval models for domain-specific corpora, thereby addressing the scalability challenges inherent in manually annotating in-domain queries. We study the data strategies for a two-stage training framework: in the first stage, which focuses on learning to decode document identifiers from queries, we investigate LLM-generated queries across multiple granularity (e.g. chunks, sentences) and domain-relevant search constraints that can better capture nuanced relevancy signals. In the second stage, which aims to refine document ranking through preference learning, we explore the strategies for mining hard negatives based on the initial model's predictions. Experiments on public datasets over diverse domains demonstrate the effectiveness of our synthetic data generation and hard negative sampling approach.