# Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models

arXiv: https://arxiv.org/abs/2403.12388 · HTML: https://mlsanity.com/paper/2403.12388/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Survey on Evaluation Methods for Dialogue Systems](https://mlsanity.com/paper/1905.04071/) (2019)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations](https://mlsanity.com/paper/2310.11207/) (2023)
- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems](https://mlsanity.com/paper/2105.03748/) (2021)
- [Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue System](https://mlsanity.com/paper/2306.09821/) (2023)
- [User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systems](https://mlsanity.com/paper/2202.02912/) (2022)
- [Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations](https://mlsanity.com/paper/2010.02495/) (2020)
- [Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents](https://mlsanity.com/paper/2010.11230/) (2020)
- [Multi-domain Conversation Quality Evaluation via User Satisfaction Estimation](https://mlsanity.com/paper/1911.08567/) (2019)
- [Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting](https://mlsanity.com/paper/2302.04813/) (2023)

## Built on by (most cited first, 19 of 19)

- [Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale](https://mlsanity.com/paper/2504.14225/) (2025)
- [When Search Engine Services meet Large Language Models: Visions and Challenges](https://mlsanity.com/paper/2407.00128/) (2024)
- [WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback](https://mlsanity.com/paper/2408.15549/) (2024)
- [On Overcoming Miscalibrated Conversational Priors in LLM-based Chatbots](https://mlsanity.com/paper/2406.01633/) (2024)
- [DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning](https://mlsanity.com/paper/2510.02341/) (2025)
- [How can we assess human-agent interactions? Case studies in software agent design](https://mlsanity.com/paper/2510.09801/) (2025)
- [ConDABench: Interactive Evaluation of Language Models for Data Analysis](https://mlsanity.com/paper/2510.13835/) (2025)
- [CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments](https://mlsanity.com/paper/2603.28569/) (2026)
- [Continuous Latent Contexts Enable Efficient Online Learning in Transformers](https://mlsanity.com/paper/2605.09867/) (2026)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [User Simulator-Guided Multi-Turn Preference Optimization for Reasoning LLM-based Conversational Recommendation](https://mlsanity.com/paper/2604.03671/) (2026)
- [Mitigating Label Bias with Interpretable Rubric Embeddings](https://mlsanity.com/paper/2605.21455/) (2026)
- [POET: Protocol Optimization via Eligibility Tuning](https://mlsanity.com/paper/2602.00370/) (2026)
- [Interaction-Grounded Learning for Contextual Markov Decision Processes with Personalized Feedback](https://mlsanity.com/paper/2602.08307/) (2026)
- [A benchmark for joint dialogue satisfaction, emotion recognition, and emotion state transition prediction](https://mlsanity.com/paper/2603.03327/) (2026)
- [Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?](https://mlsanity.com/paper/2605.00964/) (2026)
- [Verifiable Self-Evolution for Open-Ended Dialogue Skills via Future-Feedback Prediction](https://mlsanity.com/paper/2607.18973/) (2026)
- [Expectation Alignment of Language Models for Real-World User Expectations](https://mlsanity.com/paper/2607.20485/) (2026)
- [Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration](https://mlsanity.com/paper/2608.11460/) (2026)

## Abstract (arXiv)

Accurate and interpretable user satisfaction estimation (USE) is critical for understanding, evaluating, and continuously improving conversational systems. Users express their satisfaction or dissatisfaction with diverse conversational patterns in both general-purpose (ChatGPT and Bing Copilot) and task-oriented (customer service chatbot) conversational systems. Existing approaches based on featurized ML models or text embeddings fall short in extracting generalizable patterns and are hard to interpret. In this work, we show that LLMs can extract interpretable signals of user satisfaction from their natural language utterances more effectively than embedding-based approaches. Moreover, an LLM can be tailored for USE via an iterative prompting framework using supervision from labeled examples. The resulting method, Supervised Prompting for User satisfaction Rubrics (SPUR), not only has higher accuracy but is more interpretable as it scores user satisfaction via learned rubrics with a detailed breakdown.