mlsanityData to 7 Oct 2026

Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggets

arXiv 2503.09902 ↗First posted 12 Mar 2025cs.IRcited by 8

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

The rise of personalized conversational search systems has been driven by advancements in Large Language Models (LLMs), enabling these systems to retrieve and generate answers for complex information needs. However, the automatic evaluation of responses generated by Retrieval Augmented Generation (RAG) systems remains an understudied challenge. In this paper, we introduce a new resource for assessing the retrieval effectiveness and relevance of response generated by RAG systems, using a nugget-based evaluation framework. Built upon the foundation of TREC iKAT 2023, our dataset extends to the TREC iKAT 2024 collection, which includes 17 conversations and 20,575 relevance passage assessments, together with 2,279 extracted gold nuggets, and 62 manually written gold answers from NIST assessors. While maintaining the core structure of its predecessor, this new collection enables a deeper exploration of generation tasks in conversational settings. Key improvements in iKAT 2024 include: (1) ``gold nuggets'' -- concise, essential pieces of information extracted from relevant passages of the collection -- which serve as a foundation for automatic response evaluation; (2) manually written answers to provide a gold standard for response evaluation; (3) unanswerable questions to evaluate model hallucination; (4) expanded user personas, providing richer contextual grounding; and (5) a transition from Personal Text Knowledge Base (PTKB) ranking to PTKB classification and selection. Built on this resource, we provide a framework for long-form answer generation evaluation, involving nuggets extraction and nuggets matching, linked to retrieval. This establishes a solid resource for advancing research in personalized conversational search and long-form answer generation. Our resources are publicly available at https://github.com/irlabamsterdam/CONE-RAG.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 19 of its 54 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

19 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems7 Jan

Other references · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  4. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  5. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  6. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 136
  7. 2023Large language models can accurately predict searcher preferencescited by 129
  8. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 84
  9. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  10. 2022Conversational Information Seekingcited by 36
  11. 2024CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Searchcited by 29
  12. 2024Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Frameworkcited by 28
6 more
  1. 2024On the Evaluation of Machine-Generated Reportscited by 23
  2. 2024Can We Use Large Language Models to Fill Relevance Judgment Holes?cited by 17
  3. 2024TREC iKAT 2023: The Interactive Knowledge Assistance Track Overviewcited by 15
  4. 2024Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024cited by 8
  5. 2024RALI@TREC iKAT 2024: Achieving Personalization via Retrieval Fusion in Conversational Searchcited by 2
  6. 2023Open-Domain Dialogue Quality Evaluation: Deriving Nugget-level Scores from Turn-level Scorescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Conversational Search: From Fundamentals to Frontiers in the LLM Eracited by 8
  2. 2025All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generationscited by 6
  3. 2025Investigating LLM Variability in Personalized Conversational Information Retrievalcited by 2
  4. 2026Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?cited by 1
  5. 2026Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coveragecited by 1
  6. 2025Adaptive Personalized Conversational Information Retrievalcited by 0
  7. 2026Plans for Evaluating Structured Generative Search Summariescited by 0
  8. 2026uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Rerankingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Adaptive Retrieval-Augmented Generation for Conversational Systems
  2. 2024Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA
  3. 2026Comprehensive Comparison of RAG Methods Across Multi-Domain Conversational QA
  4. 2026MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
  5. 2025Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses
  6. 2024Boosting Conversational Question Answering with Fine-Grained Retrieval-Augmentation and Self-Check
  7. 2026RAC: Retrieval-Augmented Clarification for Faithful Conversational Search
  8. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models