Evaluating Agent Interactions Through Episodic Knowledge Graphs
Not on the map. Why?
Abstract (arXiv)
We present a new method based on episodic Knowledge Graphs (eKGs) for evaluating (multimodal) conversational agents in open domains. This graph is generated by interpreting raw signals during conversation and is able to capture the accumulation of knowledge over time. We apply structural and semantic analysis of the resulting graphs and translate the properties into qualitative measures. We compare these measures with existing automatic and manual evaluation metrics commonly used for conversational agents. Our results show that our Knowledge-Graph-based evaluation provides more qualitative insights into interaction and the agent's behavior.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 14 of its 34 references that are in the map. First posted on arXiv 22 Sep 2022.
Builds on
14 of 34 references.
References · 14
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
- 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
- 2020Recipes for building an open-domain chatbotcited by 593
- 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
- 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
- 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
- 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
- 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
- 2020Overview of the Ninth Dialog System Technology Challenge: DSTC9cited by 49
- 2022Interactive Evaluation of Dialog Track at DSTC9cited by 9
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.