mlsanityData to 7 Oct 2026

TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants

arXiv 2405.02637 ↗First posted 4 May 2024cs.IR cs.AI cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Conversational information seeking has evolved rapidly in the last few years with the development of Large Language Models (LLMs), providing the basis for interpreting and responding in a naturalistic manner to user requests. The extended TREC Interactive Knowledge Assistance Track (iKAT) collection aims to enable researchers to test and evaluate their Conversational Search Agents (CSA). The collection contains a set of 36 personalized dialogues over 20 different topics each coupled with a Personal Text Knowledge Base (PTKB) that defines the bespoke user personas. A total of 344 turns with approximately 26,000 passages are provided as assessments on relevance, as well as additional assessments on generated responses over four key dimensions: relevance, completeness, groundedness, and naturalness. The collection challenges CSA to efficiently navigate diverse personal contexts, elicit pertinent persona information, and employ context for relevant conversations. The integration of a PTKB and the emphasis on decisional search tasks contribute to the uniqueness of this test collection, making it an essential benchmark for advancing research in conversational and interactive knowledge assistants.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 11 of its 27 references that are in the map. First posted on arXiv 4 May 2024.

Builds on

11 of 27 references.

References · 11

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2018CoQA: A Conversational Question Answering Challengecited by 674
  7. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  8. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  9. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  10. 2024TREC iKAT 2023: The Interactive Knowledge Assistance Track Overviewcited by 15
  11. 2021A Large-Scale Analysis of Mixed Initiative in Information-Seeking Dialogues for Conversational Searchcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 50
  2. 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
  3. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 17
  4. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judgescited by 14
  5. 2024DiSCo: LLM Knowledge Distillation for Efficient Sparse Retrieval in Conversational Searchcited by 8
  6. 2025Conversational Search: From Fundamentals to Frontiers in the LLM Eracited by 8
  7. 2026MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversationscited by 8
  8. 2025Query Understanding in LLM-based Conversational Information Seekingcited by 4
  9. 2024IRLab@iKAT24: Learned Sparse Retrieval with Multi-aspect LLM Query Generation for Conversational Searchcited by 3
  10. 2025Investigating LLM Variability in Personalized Conversational Information Retrievalcited by 2
  11. 2026Evaluation of Agents under Simulated AI Marketplace Dynamicscited by 1
  12. 2025Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifestocited by 0
4 more
  1. 2025CFDA & CLIP at TREC iKAT 2025: Enhancing Personalized Conversational Search via Query Reformulation and Rank Fusioncited by 0
  2. 2026Assessing the Effectiveness of LLMs in Delivering Cognitive Behavioral Therapycited by 0
  3. 2026AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversationscited by 0
  4. 2026uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Rerankingcited by 0