# Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifesto

arXiv: https://arxiv.org/abs/2506.11112 · HTML: https://mlsanity.com/paper/2506.11112/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 159 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Builds on (most cited first, 30 of 30)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Towards Deep Conversational Recommendations](https://mlsanity.com/paper/1812.07617/) (2018)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [TREC CAsT 2019: The Conversational Assistance Track Overview](https://mlsanity.com/paper/2003.13624/) (2020)
- [Improving Multi-turn Dialogue Modelling with Utterance ReWriter](https://mlsanity.com/paper/1906.07004/) (2019)
- [Conversational Information Seeking](https://mlsanity.com/paper/2201.08808/) (2022)
- [USimAgent: Large Language Models for Simulating Search Users](https://mlsanity.com/paper/2403.09142/) (2024)
- [Fairness in Information Access Systems](https://mlsanity.com/paper/2105.05779/) (2021)
- [A Survey on Asking Clarification Questions Datasets in Conversational Systems](https://mlsanity.com/paper/2305.15933/) (2023)
- [Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Dataset](https://mlsanity.com/paper/1912.04639/) (2019)
- [Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2402.13374/) (2024)
- [Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search](https://mlsanity.com/paper/2405.03480/) (2024)
- [TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants](https://mlsanity.com/paper/2405.02637/) (2024)
- [TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview](https://mlsanity.com/paper/2401.01330/) (2024)
- [QRFA: A Data-Driven Model of Information-Seeking Dialogues](https://mlsanity.com/paper/1812.10720/) (2018)
- [Evaluating Generative Ad Hoc Information Retrieval](https://mlsanity.com/paper/2311.04694/) (2023)
- [Impacts of Personal Characteristics on User Trust in Conversational Recommender Systems](https://mlsanity.com/paper/2203.12981/) (2022)
- [Analyzing User Engagement with TikTok's Short Format Video Recommendations using Data Donations](https://mlsanity.com/paper/2301.04945/) (2023)
- [Query Understanding in the Age of Large Language Models](https://mlsanity.com/paper/2306.16004/) (2023)
- [Meta-evaluation of Conversational Search Evaluation Metrics](https://mlsanity.com/paper/2104.13453/) (2021)
- [A Survey of Ontology Expansion for Conversational Understanding](https://mlsanity.com/paper/2410.15019/) (2024)
- [SWAN: A Generic Framework for Auditing Textual Conversational Systems](https://mlsanity.com/paper/2305.08290/) (2023)
- [De-centering the (Traditional) User: Multistakeholder Evaluation of Recommender Systems](https://mlsanity.com/paper/2501.05170/) (2025)
- [Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies](https://mlsanity.com/paper/2409.17436/) (2024)
- [Leveraging Multi-Method Evaluation for Multi-Stakeholder Settings](https://mlsanity.com/paper/2001.04348/) (2019)
- [Future of Information Retrieval Research in the Age of Generative AI](https://mlsanity.com/paper/2412.02043/) (2024)
- [Mobile-Based Experience Sampling for Behaviour Research](https://mlsanity.com/paper/1508.03725/) (2015)
- [Ah, that's the great puzzle: On the Quest of a Holistic Understanding of the Harms of Recommender Systems on Children](https://mlsanity.com/paper/2405.02050/) (2024)

## Abstract (arXiv)

During the workshop, we deeply discussed what CONversational Information ACcess (CONIAC) is and its unique features, proposing a world model abstracting it, and defined the Conversational Agents Framework for Evaluation (CAFE) for the evaluation of CONIAC systems, consisting of six major components: 1) goals of the system's stakeholders, 2) user tasks to be studied in the evaluation, 3) aspects of the users carrying out the tasks, 4) evaluation criteria to be considered, 5) evaluation methodology to be applied, and 6) measures for the quantitative criteria chosen.