# INFACT: An Online Human Evaluation Framework for Conversational Recommendation

arXiv: https://arxiv.org/abs/2209.03213 · HTML: https://mlsanity.com/paper/2209.03213/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 7 Sep 2022.

## Builds on (most cited first, 12 of 12)

- [Towards Deep Conversational Recommendations](https://mlsanity.com/paper/1812.07617/) (2018)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusion](https://mlsanity.com/paper/2007.04032/) (2020)
- [Towards Knowledge-Based Recommender Dialog System](https://mlsanity.com/paper/1908.05391/) (2019)
- [Towards Topic-Guided Conversational Recommender System](https://mlsanity.com/paper/2010.04125/) (2020)
- [INSPIRED: Toward Sociable Recommendation Dialog Systems](https://mlsanity.com/paper/2009.14306/) (2020)
- [Recommendation as a Communication Game: Self-Supervised Bot-Play for Goal-oriented Dialogue](https://mlsanity.com/paper/1909.03922/) (2019)
- [Towards Knowledge-Based Personalized Product Description Generation in E-commerce](https://mlsanity.com/paper/1903.12457/) (2019)
- [Conversational Recommendation: A Grand AI Challenge](https://mlsanity.com/paper/2203.09126/) (2022)
- [Towards Retrieval-based Conversational Recommendation](https://mlsanity.com/paper/2109.02311/) (2021)
- [Evaluating Conversational Recommender Systems: A Landscape of Research](https://mlsanity.com/paper/2208.12061/) (2022)
- [A retrieval-based dialogue system utilizing utterance and context embeddings](https://mlsanity.com/paper/1710.05780/) (2017)

## Abstract (arXiv)

Conversational recommender systems (CRS) are interactive agents that support their users in recommendation-related goals through multi-turn conversations. Generally, a CRS can be evaluated in various dimensions. Today's CRS mainly rely on offline(computational) measures to assess the performance of their algorithms in comparison to different baselines. However, offline measures can have limitations, for example, when the metrics for comparing a newly generated response with a ground truth do not correlate with human perceptions, because various alternative generated responses might be suitable too in a given dialog situation. Current research on machine learning-based CRS models therefore acknowledges the importance of humans in the evaluation process, knowing that pure offline measures may not be sufficient in evaluating a highly interactive system like a CRS.