# Exploring the Viability of Synthetic Query Generation for Relevance Prediction

arXiv: https://arxiv.org/abs/2305.11944 · HTML: https://mlsanity.com/paper/2305.11944/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Synthetic QA Corpora Generation with Roundtrip Consistency](https://mlsanity.com/paper/1906.05416/) (2019)
- [RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses](https://mlsanity.com/paper/2210.10634/) (2022)
- [Neural Question Generation from Text: A Preliminary Study](https://mlsanity.com/paper/1704.01792/) (2017)
- [Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search](https://mlsanity.com/paper/2206.06588/) (2022)
- [InPars: Data Augmentation for Information Retrieval using Large Language Models](https://mlsanity.com/paper/2202.05144/) (2022)
- [Harvesting Paragraph-Level Question-Answer Pairs from Wikipedia](https://mlsanity.com/paper/1805.05942/) (2018)
- [Controllable Open-ended Question Generation with A New Question Type Ontology](https://mlsanity.com/paper/2107.00152/) (2021)
- [Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus](https://mlsanity.com/paper/1603.06807/) (2016)
- [Inquisitive Question Generation for High Level Text Comprehension](https://mlsanity.com/paper/2010.01657/) (2020)
- [CONSISTENT: Open-Ended Question Generation From News Articles](https://mlsanity.com/paper/2210.11536/) (2022)

## Built on by (most cited first, 9 of 9)

- [RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation](https://mlsanity.com/paper/2502.10996/) (2025)
- [Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomy](https://mlsanity.com/paper/2403.04160/) (2024)
- [It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction](https://mlsanity.com/paper/2311.07930/) (2023)
- [Large Language Models for Relevance Judgment in Product Search](https://mlsanity.com/paper/2406.00247/) (2024)
- [Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generation](https://mlsanity.com/paper/2502.11181/) (2025)
- [Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learning](https://mlsanity.com/paper/2311.09619/) (2023)
- [OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulation](https://mlsanity.com/paper/2510.03771/) (2025)
- [Improving Scientific Document Retrieval with Academic Concept Index](https://mlsanity.com/paper/2601.00567/) (2026)
- [Beyond Relevance: Structured Semantic Supervision for Product Search with LLM-Augmented Annotations](https://mlsanity.com/paper/2609.23646/) (2026)

## Abstract (arXiv)

Query-document relevance prediction is a critical problem in Information Retrieval systems. This problem has increasingly been tackled using (pretrained) transformer-based models which are finetuned using large collections of labeled data. However, in specialized domains such as e-commerce and healthcare, the viability of this approach is limited by the dearth of large in-domain data. To address this paucity, recent methods leverage these powerful models to generate high-quality task and domain-specific synthetic data. Prior work has largely explored synthetic data generation or query generation (QGen) for Question-Answering (QA) and binary (yes/no) relevance prediction, where for instance, the QGen models are given a document, and trained to generate a query relevant to that document. However in many problems, we have a more fine-grained notion of relevance than a simple yes/no label. Thus, in this work, we conduct a detailed study into how QGen approaches can be leveraged for nuanced relevance prediction. We demonstrate that -- contrary to claims from prior works -- current QGen approaches fall short of the more conventional cross-domain transfer-learning approaches. Via empirical studies spanning 3 public e-commerce benchmarks, we identify new shortcomings of existing QGen approaches -- including their inability to distinguish between different grades of relevance. To address this, we introduce label-conditioned QGen models which incorporates knowledge about the different relevance. While our experiments demonstrate that these modifications help improve performance of QGen techniques, we also find that QGen approaches struggle to capture the full nuance of the relevance label space and as a result the generated queries are not faithful to the desired relevance label.