# Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggets

arXiv: https://arxiv.org/abs/2503.09902 · HTML: https://mlsanity.com/paper/2503.09902/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 19 of its 54 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Placed in

- Retrieval-augmented generation / [Conversational and speech RAG](https://mlsanity.com/rag/conversational-and-speech-rag/)

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions](https://mlsanity.com/paper/2312.02913/) (2023)
- [MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2501.03468/) (2025)
- [Conversational Information Seeking](https://mlsanity.com/paper/2201.08808/) (2022)
- [CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search](https://mlsanity.com/paper/2406.05013/) (2024)
- [Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework](https://mlsanity.com/paper/2411.09607/) (2024)
- [On the Evaluation of Machine-Generated Reports](https://mlsanity.com/paper/2405.00982/) (2024)
- [Can We Use Large Language Models to Fill Relevance Judgment Holes?](https://mlsanity.com/paper/2405.05600/) (2024)
- [TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview](https://mlsanity.com/paper/2401.01330/) (2024)
- [Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024](https://mlsanity.com/paper/2408.05388/) (2024)
- [RALI@TREC iKAT 2024: Achieving Personalization via Retrieval Fusion in Conversational Search](https://mlsanity.com/paper/2412.07998/) (2024)
- [Open-Domain Dialogue Quality Evaluation: Deriving Nugget-level Scores from Turn-level Scores](https://mlsanity.com/paper/2310.00410/) (2023)

## Built on by (most cited first, 8 of 8)

- [Conversational Search: From Fundamentals to Frontiers in the LLM Era](https://mlsanity.com/paper/2506.10635/) (2025)
- [All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations](https://mlsanity.com/paper/2510.07083/) (2025)
- [Investigating LLM Variability in Personalized Conversational Information Retrieval](https://mlsanity.com/paper/2510.03795/) (2025)
- [Adaptive Personalized Conversational Information Retrieval](https://mlsanity.com/paper/2508.08634/) (2025)
- [Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?](https://mlsanity.com/paper/2601.13227/) (2026)
- [Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage](https://mlsanity.com/paper/2603.08819/) (2026)
- [Plans for Evaluating Structured Generative Search Summaries](https://mlsanity.com/paper/2605.26400/) (2026)
- [uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Reranking](https://mlsanity.com/paper/2606.11945/) (2026)

## Abstract (arXiv)

The rise of personalized conversational search systems has been driven by advancements in Large Language Models (LLMs), enabling these systems to retrieve and generate answers for complex information needs. However, the automatic evaluation of responses generated by Retrieval Augmented Generation (RAG) systems remains an understudied challenge. In this paper, we introduce a new resource for assessing the retrieval effectiveness and relevance of response generated by RAG systems, using a nugget-based evaluation framework. Built upon the foundation of TREC iKAT 2023, our dataset extends to the TREC iKAT 2024 collection, which includes 17 conversations and 20,575 relevance passage assessments, together with 2,279 extracted gold nuggets, and 62 manually written gold answers from NIST assessors. While maintaining the core structure of its predecessor, this new collection enables a deeper exploration of generation tasks in conversational settings. Key improvements in iKAT 2024 include: (1) ``gold nuggets'' -- concise, essential pieces of information extracted from relevant passages of the collection -- which serve as a foundation for automatic response evaluation; (2) manually written answers to provide a gold standard for response evaluation; (3) unanswerable questions to evaluate model hallucination; (4) expanded user personas, providing richer contextual grounding; and (5) a transition from Personal Text Knowledge Base (PTKB) ranking to PTKB classification and selection. Built on this resource, we provide a framework for long-form answer generation evaluation, involving nuggets extraction and nuggets matching, linked to retrieval. This establishes a solid resource for advancing research in personalized conversational search and long-form answer generation. Our resources are publicly available at https://github.com/irlabamsterdam/CONE-RAG.