# Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation

arXiv: https://arxiv.org/abs/2604.21096 · HTML: https://mlsanity.com/paper/2604.21096/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Builds on (most cited first, 10 of 10)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Decomposing Complex Queries for Tip-of-the-tongue Retrieval](https://mlsanity.com/paper/2305.15053/) (2023)
- [Extrinsic Evaluation of Cultural Competence in Large Language Models](https://mlsanity.com/paper/2406.11565/) (2024)
- [Tip of the Tongue Known-Item Retrieval: A Case Study in Movie Identification](https://mlsanity.com/paper/2101.07124/) (2021)
- [Browsing Lost Unformed Recollections: A Benchmark for Tip-of-the-Tongue Search and Reasoning](https://mlsanity.com/paper/2503.19193/) (2025)
- [When the Music Stops: Tip-of-the-Tongue Retrieval for Music](https://mlsanity.com/paper/2305.14072/) (2023)
- [Tip of the Tongue Query Elicitation for Simulated Evaluation](https://mlsanity.com/paper/2502.17776/) (2025)
- [Overview of the TREC 2025 Tip-of-the-Tongue track](https://mlsanity.com/paper/2601.20671/) (2026)

## Abstract (arXiv)

Tip-of-the-Tongue (ToT) retrieval benchmarks have largely focused on English, limiting their applicability to multilingual information access. In this work, we construct multilingual ToT test collections for Chinese, Japanese, Korean, and English, using an LLM-based query simulation framework. We systematically study how prompt language and source document language affect the fidelity of simulated ToT queries, validating synthetic queries through system rank correlation against real user queries. Our results show that effective ToT simulation requires language-aware design choices: non-English language sources are generally important, while English Wikipedia can be beneficial when non-English sources provide insufficient information for query generation. Based on these findings, we release four ToT test collections with 5,000 queries per language across multiple domains. This work provides the first large-scale multilingual ToT benchmark and offers practical guidance for constructing realistic ToT datasets beyond English.