mlsanityData to 14 Oct 2026

Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifesto

arXiv 2506.11112 ↗First posted 8 Jun 2025cs.CL cs.HC cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

During the workshop, we deeply discussed what CONversational Information ACcess (CONIAC) is and its unique features, proposing a world model abstracting it, and defined the Conversational Agents Framework for Evaluation (CAFE) for the evaluation of CONIAC systems, consisting of six major components: 1) goals of the system's stakeholders, 2) user tasks to be studied in the evaluation, 3) aspects of the users carrying out the tasks, 4) evaluation criteria to be considered, 5) evaluation methodology to be applied, and 6) measures for the quantitative criteria chosen.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 159 references that are in the map. First posted on arXiv 8 Jun 2025.

Builds on

30 of 159 references.

References · 30

  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  3. 2018Towards Deep Conversational Recommendationscited by 176
  4. 2018Neural Approaches to Conversational AIcited by 165
  5. 2020A Survey on Conversational Recommender Systemscited by 143
  6. 2020TREC CAsT 2019: The Conversational Assistance Track Overviewcited by 64
  7. 2019Improving Multi-turn Dialogue Modelling with Utterance ReWritercited by 58
  8. 2022Conversational Information Seekingcited by 37
  9. 2024USimAgent: Large Language Models for Simulating Search Userscited by 32
  10. 2021Fairness in Information Access Systemscited by 29
  11. 2023A Survey on Asking Clarification Questions Datasets in Conversational Systemscited by 28
  12. 2019Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Datasetcited by 26
18 more
  1. 2024Reliable LLM-based User Simulator for Task-Oriented Dialogue Systemscited by 25
  2. 2024Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Searchcited by 24
  3. 2024TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistantscited by 17
  4. 2024TREC iKAT 2023: The Interactive Knowledge Assistance Track Overviewcited by 15
  5. 2018QRFA: A Data-Driven Model of Information-Seeking Dialoguescited by 11
  6. 2023Evaluating Generative Ad Hoc Information Retrievalcited by 11
  7. 2022Impacts of Personal Characteristics on User Trust in Conversational Recommender Systemscited by 8
  8. 2023Analyzing User Engagement with TikTok's Short Format Video Recommendations using Data Donationscited by 8
  9. 2023Query Understanding in the Age of Large Language Modelscited by 7
  10. 2021Meta-evaluation of Conversational Search Evaluation Metricscited by 6
  11. 2024A Survey of Ontology Expansion for Conversational Understandingcited by 6
  12. 2023SWAN: A Generic Framework for Auditing Textual Conversational Systemscited by 4
  13. 2025De-centering the (Traditional) User: Multistakeholder Evaluation of Recommender Systemscited by 4
  14. 2024Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policiescited by 3
  15. 2019Leveraging Multi-Method Evaluation for Multi-Stakeholder Settingscited by 2
  16. 2024Future of Information Retrieval Research in the Age of Generative AIcited by 2
  17. 2015Mobile-Based Experience Sampling for Behaviour Researchcited by 1
  18. 2024Ah, that's the great puzzle: On the Quest of a Holistic Understanding of the Harms of Recommender Systems on Childrencited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions
  2. 2026ATOD: An Evaluation Framework and Benchmark for Agentic Task-Oriented Dialogue Systems
  3. 2024MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation
  4. 2026Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
  5. 2025ConDABench: Interactive Evaluation of Language Models for Data Analysis
  6. 2026Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis
  7. 2023Exploring Links between Conversational Agent Design Challenges and Interdisciplinary Collaboration
  8. 2024AntEval: Evaluation of Social Interaction Competencies in LLM-Driven Agents