# Rating Prediction in Conversational Task Assistants with Behavioral and Conversational-Flow Features

arXiv: https://arxiv.org/abs/2309.11307 · HTML: https://mlsanity.com/paper/2309.11307/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2](https://mlsanity.com/paper/2012.03539/) (2020)
- [Conversational AI: The Science Behind the Alexa Prize](https://mlsanity.com/paper/1801.03604/) (2018)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [Conversational Information Seeking](https://mlsanity.com/paper/2201.08808/) (2022)
- [An End-to-End Dialogue State Tracking System with Machine Reading Comprehension and Wide & Deep Classification](https://mlsanity.com/paper/1912.09297/) (2019)
- [IART: Intent-aware Response Ranking with Transformers in Information-seeking Conversation Systems](https://mlsanity.com/paper/2002.00571/) (2020)
- [Alexa, Let's Work Together: Introducing the First Alexa Prize TaskBot Challenge on Conversational Task Assistance](https://mlsanity.com/paper/2209.06321/) (2022)
- [Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems](https://mlsanity.com/paper/2006.01921/) (2020)
- [Task2Dial: A Novel Task and Dataset for Commonsense enhanced Task-based Dialogue Grounded in Documents](https://mlsanity.com/paper/2204.01061/) (2022)
- [Understanding User Satisfaction with Task-oriented Dialogue Systems](https://mlsanity.com/paper/2204.12195/) (2022)

## Built on by (most cited first, 1 of 1)

- [TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus](https://mlsanity.com/paper/2310.02118/) (2023)

## Abstract (arXiv)

Predicting the success of Conversational Task Assistants (CTA) can be critical to understand user behavior and act accordingly. In this paper, we propose TB-Rater, a Transformer model which combines conversational-flow features with user behavior features for predicting user ratings in a CTA scenario. In particular, we use real human-agent conversations and ratings collected in the Alexa TaskBot challenge, a novel multimodal and multi-turn conversational context. Our results show the advantages of modeling both the conversational-flow and behavioral aspects of the conversation in a single model for offline rating prediction. Additionally, an analysis of the CTA-specific behavioral features brings insights into this setting and can be used to bootstrap future systems.