mlsanityData to 14 Oct 2026

Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval

arXiv 2510.02938 ↗First posted 3 Oct 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present the Conversational Data Retrieval (CDR) benchmark, the first comprehensive test set for evaluating systems that retrieve conversation data for product insights. With 1.6k queries across five analytical tasks and 9.1k conversations, our benchmark provides a reliable standard for measuring conversational data retrieval performance. Our evaluation of 16 popular embedding models shows that even the best models reach only around NDCG@10 of 0.51, revealing a substantial gap between document and conversational data retrieval capabilities. Our work identifies unique challenges in conversational data retrieval (implicit state recognition, turn dynamics, contextual references) while providing practical query templates and detailed error analysis across different task categories. The benchmark dataset and code are available at https://github.com/l-yohai/CDR-Benchmark.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

36 of 57 references.

References · 36

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  3. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  7. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  8. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  9. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  10. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  11. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  12. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
24 more
  1. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
  2. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  3. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 204
  4. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 183
  5. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 141
  6. 2024Jasper and Stella: distillation of SOTA embedding modelscited by 112
  7. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 105
  8. 2020Open-Retrieval Conversational Question Answeringcited by 99
  9. 2023Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Searchcited by 75
  10. 2022Dialog Inpainting: Turning Documents into Dialogscited by 56
  11. 2025The Widespread Adoption of Large Language Model-Assisted Writing Across Societycited by 51
  12. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  13. 2018Analyzing and Characterizing User Intent in Information-seeking Conversationscited by 36
  14. 2023Learning to Relate to Previous Turns in Conversational Searchcited by 26
  15. 2024Adaptive Retrieval-Augmented Generation for Conversational Systemscited by 22
  16. 2024First-Person Fairness in Chatbotscited by 20
  17. 2024Contextual Document Embeddingscited by 19
  18. 2024The Use of Generative Search Engines for Knowledge Work and Complex Taskscited by 18
  19. 2024REAPER: Reasoning based Retrieval Planning for Complex RAG Systemscited by 16
  20. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  21. 2024MobileConvRec: A Conversational Dataset for Mobile Apps Recommendationscited by 2
  22. 2024Synthesizing Conversations from Unlabeled Documents using Automatic Response Segmentationcited by 2
  23. 2015Mining the Minds of Customers from Online Chat Logscited by 1
  24. 2025HEISIR: Hierarchical Expansion of Inverted Semantic Indexing for Training-free Retrieval of Conversational Data using LLMscited by 1

Built on by

Not cited yet.