# Tip of the Tongue Query Elicitation for Simulated Evaluation

arXiv: https://arxiv.org/abs/2502.17776 · HTML: https://mlsanity.com/paper/2502.17776/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 39 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Builds on (most cited first, 5 of 5)

- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Decomposing Complex Queries for Tip-of-the-tongue Retrieval](https://mlsanity.com/paper/2305.15053/) (2023)
- [Synthetic Test Collections for Retrieval Evaluation](https://mlsanity.com/paper/2405.07767/) (2024)
- [Tip of the Tongue Known-Item Retrieval: A Case Study in Movie Identification](https://mlsanity.com/paper/2101.07124/) (2021)
- [When the Music Stops: Tip-of-the-Tongue Retrieval for Music](https://mlsanity.com/paper/2305.14072/) (2023)

## Built on by (most cited first, 4 of 4)

- [TeamCMU at Touch'e: Adversarial Co-Evolution for Advertisement Integration and Detection in Conversational Search](https://mlsanity.com/paper/2507.00509/) (2025)
- [DS@GT at TREC TOT 2025: Bridging Vague Recollection with Fusion Retrieval and Learned Reranking](https://mlsanity.com/paper/2601.15518/) (2026)
- [DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoning](https://mlsanity.com/paper/2602.00352/) (2026)
- [Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation](https://mlsanity.com/paper/2604.21096/) (2026)

## Abstract (arXiv)

Tip-of-the-tongue (TOT) search occurs when a user struggles to recall a specific identifier, such as a document title. While common, existing search systems often fail to effectively support TOT scenarios. Research on TOT retrieval is further constrained by the challenge of collecting queries, as current approaches rely heavily on community question-answering (CQA) websites, leading to labor-intensive evaluation and domain bias. To overcome these limitations, we introduce two methods for eliciting TOT queries - leveraging large language models (LLMs) and human participants - to facilitate simulated evaluations of TOT retrieval systems. Our LLM-based TOT user simulator generates synthetic TOT queries at scale, achieving high correlations with how CQA-based TOT queries rank TOT retrieval systems when tested in the Movie domain. Additionally, these synthetic queries exhibit high linguistic similarity to CQA-derived queries. For human-elicited queries, we developed an interface that uses visual stimuli to place participants in a TOT state, enabling the collection of natural queries. In the Movie domain, system rank correlation and linguistic similarity analyses confirm that human-elicited queries are both effective and closely resemble CQA-based queries. These approaches reduce reliance on CQA-based data collection while expanding coverage to underrepresented domains, such as Landmark and Person. LLM-elicited queries for the Movie, Landmark, and Person domains have been released as test queries in the TREC 2024 TOT track, with human-elicited queries scheduled for inclusion in the TREC 2025 TOT track. Additionally, we provide source code for synthetic query generation and the human query collection interface, along with curated visual stimuli used for eliciting TOT queries.