mlsanityData to 7 Oct 2026

Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels

arXiv 2406.11706 ↗First posted 17 Jun 2024cs.IR cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We develop a method for training small-scale (under 100M parameter) neural information retrieval models with as few as 10 gold relevance labels. The method depends on generating synthetic queries for documents using a language model (LM), and the key step is that we automatically optimize the LM prompt that is used to generate these queries based on training quality. In experiments with the BIRCO benchmark, we find that models trained with our method outperform RankZephyr and are competitive with RankLLama, both of which are 7B parameter models trained on over 100K labels. These findings point to the power of automatic prompt optimization for synthetic dataset generation.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 15 of its 22 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

15 of 22 references.

References · 15

  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  6. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  7. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  8. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  9. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 151
  10. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  11. 2023UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankerscited by 37
  12. 2023Decomposing Complex Queries for Tip-of-the-tongue Retrievalcited by 24
3 more
  1. 2022RELIC: Retrieving Evidence for Literary Claimscited by 18
  2. 2024BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectivescited by 15
  3. 2023DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queriescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 2.

3 papers.

  1. 2024ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Modelscited by 1
  2. 2026PrismVAU: Prompt-Refined Inference System for Multimodal Video Anomaly Understandingcited by 1
  3. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0