mlsanityData to 14 Oct 2026

Browsing Lost Unformed Recollections: A Benchmark for Tip-of-the-Tongue Search and Reasoning

arXiv 2503.19193 ↗First posted 24 Mar 2025cs.AI cs.CL cs.IRcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce Browsing Lost Unformed Recollections, a tip-of-the-tongue known-item search and reasoning benchmark for general AI assistants. BLUR introduces a set of 573 real-world validated questions that demand searching and reasoning across multi-modal and multilingual inputs, as well as proficient tool use, in order to excel on. Humans easily ace these questions (scoring on average 98%), while the best-performing system scores around 56%. To facilitate progress toward addressing this challenging and aspirational use case for general AI assistants, we release 350 questions through a public leaderboard, retain the answers to 250 of them, and have the rest as a private test set.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 25 of its 46 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

25 of 46 references.

References · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  7. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  9. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  10. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  11. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  12. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
13 more
  1. 2023OpenAGI: When LLM Meets Domain Expertscited by 197
  2. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  3. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 140
  4. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativenesscited by 77
  5. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gapcited by 56
  6. 2020Towards Ecologically Valid Research on Language User Interfacescited by 42
  7. 2024ECBD: Evidence-Centered Benchmark Design for NLPcited by 28
  8. 2023Decomposing Complex Queries for Tip-of-the-tongue Retrievalcited by 24
  9. 2024CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrievalcited by 20
  10. 2021Tip of the Tongue Known-Item Retrieval: A Case Study in Movie Identificationcited by 14
  11. 2021Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Faircited by 12
  12. 2024Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefitscited by 9
  13. 2023When the Music Stops: Tip-of-the-Tongue Retrieval for Musiccited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
  2. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 62
  3. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  4. 2025InteractComp: Evaluating Search Agents With Ambiguous Queriescited by 12
  5. 2026UIS-Digger: Towards Comprehensive Research Agent Systems for Real-world Unindexed Information Seekingcited by 1
  6. 2026DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoningcited by 0
  7. 2026Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluationcited by 0