# Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels

arXiv: https://arxiv.org/abs/2406.11706 · HTML: https://mlsanity.com/paper/2406.11706/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Fine-Tuning LLaMA for Multi-Stage Text Retrieval](https://mlsanity.com/paper/2310.08319/) (2023)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [Improving Passage Retrieval with Zero-Shot Question Generation](https://mlsanity.com/paper/2204.07496/) (2022)
- [RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!](https://mlsanity.com/paper/2312.02724/) (2023)
- [Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline](https://mlsanity.com/paper/2101.08751/) (2021)
- [UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers](https://mlsanity.com/paper/2303.00807/) (2023)
- [Decomposing Complex Queries for Tip-of-the-tongue Retrieval](https://mlsanity.com/paper/2305.15053/) (2023)
- [RELIC: Retrieving Evidence for Literary Claims](https://mlsanity.com/paper/2203.10053/) (2022)
- [BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives](https://mlsanity.com/paper/2402.14151/) (2024)
- [DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queries](https://mlsanity.com/paper/2310.04678/) (2023)

## Built on by (most cited first, 3 of 3)

- [ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models](https://mlsanity.com/paper/2408.05948/) (2024)
- [PrismVAU: Prompt-Refined Inference System for Multimodal Video Anomaly Understanding](https://mlsanity.com/paper/2601.02927/) (2026)
- [Towards Spec Learning: Inference-Time Alignment from Preference Pairs](https://mlsanity.com/paper/2606.24004/) (2026)

## Abstract (arXiv)

We develop a method for training small-scale (under 100M parameter) neural information retrieval models with as few as 10 gold relevance labels. The method depends on generating synthetic queries for documents using a language model (LM), and the key step is that we automatically optimize the LM prompt that is used to generate these queries based on training quality. In experiments with the BIRCO benchmark, we find that models trained with our method outperform RankZephyr and are competitive with RankLLama, both of which are 7B parameter models trained on over 100K labels. These findings point to the power of automatic prompt optimization for synthetic dataset generation.