# TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants

arXiv: https://arxiv.org/abs/2405.02637 · HTML: https://mlsanity.com/paper/2405.02637/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 4 May 2024.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores](https://mlsanity.com/paper/2311.09766/) (2023)
- [Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions](https://mlsanity.com/paper/2312.02913/) (2023)
- [TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview](https://mlsanity.com/paper/2401.01330/) (2024)
- [A Large-Scale Analysis of Mixed Initiative in Information-Seeking Dialogues for Conversational Search](https://mlsanity.com/paper/2104.07096/) (2021)

## Built on by (most cited first, 17 of 17)

- [MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2501.03468/) (2025)
- [A Survey on Recent Advances in Conversational Data Generation](https://mlsanity.com/paper/2405.13003/) (2024)
- [Benchmarking Poisoning Attacks against Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.18543/) (2025)
- [Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges](https://mlsanity.com/paper/2504.15205/) (2025)
- [DiSCo: LLM Knowledge Distillation for Efficient Sparse Retrieval in Conversational Search](https://mlsanity.com/paper/2410.14609/) (2024)
- [MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations](https://mlsanity.com/paper/2602.23184/) (2026)
- [Conversational Search: From Fundamentals to Frontiers in the LLM Era](https://mlsanity.com/paper/2506.10635/) (2025)
- [Query Understanding in LLM-based Conversational Information Seeking](https://mlsanity.com/paper/2504.06356/) (2025)
- [IRLab@iKAT24: Learned Sparse Retrieval with Multi-aspect LLM Query Generation for Conversational Search](https://mlsanity.com/paper/2411.14739/) (2024)
- [Investigating LLM Variability in Personalized Conversational Information Retrieval](https://mlsanity.com/paper/2510.03795/) (2025)
- [Evaluation of Agents under Simulated AI Marketplace Dynamics](https://mlsanity.com/paper/2604.14256/) (2026)
- [Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifesto](https://mlsanity.com/paper/2506.11112/) (2025)
- [CFDA & CLIP at TREC iKAT 2025: Enhancing Personalized Conversational Search via Query Reformulation and Rank Fusion](https://mlsanity.com/paper/2509.15588/) (2025)
- [Assessing the Effectiveness of LLMs in Delivering Cognitive Behavioral Therapy](https://mlsanity.com/paper/2603.03862/) (2026)
- [AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations](https://mlsanity.com/paper/2603.10524/) (2026)
- [uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Reranking](https://mlsanity.com/paper/2606.11945/) (2026)
- [RALI@iKAT 2026: Retrieval-aware User Profile Denoising via Personalized Conversational Dense Retrieval](https://mlsanity.com/paper/2610.17673/) (2026)

## Abstract (arXiv)

Conversational information seeking has evolved rapidly in the last few years with the development of Large Language Models (LLMs), providing the basis for interpreting and responding in a naturalistic manner to user requests. The extended TREC Interactive Knowledge Assistance Track (iKAT) collection aims to enable researchers to test and evaluate their Conversational Search Agents (CSA). The collection contains a set of 36 personalized dialogues over 20 different topics each coupled with a Personal Text Knowledge Base (PTKB) that defines the bespoke user personas. A total of 344 turns with approximately 26,000 passages are provided as assessments on relevance, as well as additional assessments on generated responses over four key dimensions: relevance, completeness, groundedness, and naturalness. The collection challenges CSA to efficiently navigate diverse personal contexts, elicit pertinent persona information, and employ context for relevant conversations. The integration of a PTKB and the emphasis on decisional search tasks contribute to the uniqueness of this test collection, making it an essential benchmark for advancing research in conversational and interactive knowledge assistants.