mlsanityData to 14 Oct 2026

On Synthetic Data Strategies for Domain-Specific Generative Retrieval

arXiv 2502.17957 ↗First posted 25 Feb 2025cs.CL cs.IRcited by 5

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

This paper investigates synthetic data generation strategies in developing generative retrieval models for domain-specific corpora, thereby addressing the scalability challenges inherent in manually annotating in-domain queries. We study the data strategies for a two-stage training framework: in the first stage, which focuses on learning to decode document identifiers from queries, we investigate LLM-generated queries across multiple granularity (e.g. chunks, sentences) and domain-relevant search constraints that can better capture nuanced relevancy signals. In the second stage, which aims to refine document ranking through preference learning, we explore the strategies for mining hard negatives based on the initial model's predictions. Experiments on public datasets over diverse domains demonstrate the effectiveness of our synthetic data generation and hard negative sampling approach.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 34 of its 55 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

34 of 55 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  9. 2020Autoregressive Entity Retrievalcited by 334
  10. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  11. 2024Iterative Reasoning Preference Optimizationcited by 246
  12. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
22 more
  1. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  2. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 113
  3. 2020We Can Detect Your Bias: Predicting the Political Ideology of News Articlescited by 92
  4. 2023InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrievalcited by 73
  5. 2023Learning to Tokenize for Generative Retrievalcited by 68
  6. 2022Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generationcited by 48
  7. 2023How Does Generative Retrieval Scale to Millions of Passages?cited by 48
  8. 2023Learning to Rank in Generative Retrievalcited by 46
  9. 2023Scalable and Effective Generative Information Retrievalcited by 39
  10. 2022CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Taskscited by 36
  11. 2022Ultron: An Ultimate Retriever on Corpus with a Model-based Indexercited by 31
  12. 2023Large Language Models are Built-in Autoregressive Search Enginescited by 27
  13. 2023Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategiescited by 25
  14. 2024CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Taskscited by 18
  15. 2023GLEN: Generative Retrieval via Lexical Index Learningcited by 17
  16. 2023A Unified Generative Retriever for Knowledge-Intensive Language Tasks via Prompt Learningcited by 16
  17. 2023Auto Search Indexer for End-to-End Document Retrievalcited by 16
  18. 2023Generative Retrieval via Term Set Generationcited by 13
  19. 2024Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awarenesscited by 12
  20. 2024You Only Read Once (YORO): Learning to Internalize Database Knowledge for Text-to-SQLcited by 11
  21. 2023Understanding Differential Search Index for Text Retrievalcited by 9
  22. 2023Dense Retrieval Adaptation using Target Domain Descriptioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  2. 2025Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Modelscited by 5
  3. 2025Revela: Dense Retriever Learning via Language Modelingcited by 3
  4. 2026From Relevance to Authority: Authority-aware Generative Retrieval in Web Search Enginescited by 1
  5. 2026Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025InPars+: Supercharging Synthetic Data Generation for Information Retrieval Systems
  2. 2025Exploring Training and Inference Scaling Laws in Generative Retrieval
  3. 2023InPars Toolkit: A Unified and Reproducible Synthetic Data Generation Pipeline for Neural Information Retrieval
  4. 2025DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization
  5. 2025Enhancing Transformer-Based Rerankers with Synthetic Data and LLM-Based Supervision
  6. 2025Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevance
  7. 2025Enhancing Domain-Specific Retrieval-Augmented Generation: Synthetic Data Generation and Evaluation using Reasoning Models
  8. 2024Generative Retrieval with Preference Optimization for E-commerce Search