# Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue System

arXiv: https://arxiv.org/abs/2306.09821 · HTML: https://mlsanity.com/paper/2306.09821/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](https://mlsanity.com/paper/2109.14739/) (2021)
- [GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection](https://mlsanity.com/paper/2111.14592/) (2021)
- [Guiding Large Language Models via Directional Stimulus Prompting](https://mlsanity.com/paper/2302.11520/) (2023)
- [Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems](https://mlsanity.com/paper/2105.03748/) (2021)
- [Controllable Dialogue Simulation with In-Context Learning](https://mlsanity.com/paper/2210.04185/) (2022)
- [Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System](https://mlsanity.com/paper/2305.02468/) (2023)
- [Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue System](https://mlsanity.com/paper/2006.06814/) (2020)
- [Multi-Domain Dialogue Acts and Response Co-Generation](https://mlsanity.com/paper/2004.12363/) (2020)
- [User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systems](https://mlsanity.com/paper/2202.02912/) (2022)
- [Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues](https://mlsanity.com/paper/2305.16798/) (2023)
- [Modeling User Satisfaction Dynamics in Dialogue via Hawkes Process](https://mlsanity.com/paper/2305.12594/) (2023)
- [A Comparative Study on Language Models for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2201.08687/) (2022)
- [Interactive Query Clarification and Refinement via User Simulation](https://mlsanity.com/paper/2205.15918/) (2022)

## Built on by (most cited first, 30 of 32)

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions](https://mlsanity.com/paper/2312.02913/) (2023)
- [Towards Human-centered Proactive Conversational Agents](https://mlsanity.com/paper/2404.12670/) (2024)
- [PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants](https://mlsanity.com/paper/2506.09902/) (2025)
- [Exploring the Potential of Large Language Models in Computational Argumentation](https://mlsanity.com/paper/2311.09022/) (2023)
- [Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlook](https://mlsanity.com/paper/2402.19348/) (2024)
- [Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models](https://mlsanity.com/paper/2403.12388/) (2024)
- [User Modeling in the Era of Large Language Models: Current Research and Future Directions](https://mlsanity.com/paper/2312.11518/) (2023)
- [Self-seeding and Multi-intent Self-instructing LLMs for Generating Intent-aware Information-Seeking dialogs](https://mlsanity.com/paper/2402.11633/) (2024)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems](https://mlsanity.com/paper/2404.06762/) (2024)
- [ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders](https://mlsanity.com/paper/2602.16938/) (2026)
- [Human vs. Agent in Task-Oriented Conversations](https://mlsanity.com/paper/2509.17619/) (2025)
- [Enhancing Large Language Model Induced Task-Oriented Dialogue Systems Through Look-Forward Motivated Goals](https://mlsanity.com/paper/2309.08949/) (2023)
- [Demonstration Selection for In-Context Learning via Reinforcement Learning](https://mlsanity.com/paper/2412.03966/) (2024)
- [Bridging the User-side Knowledge Gap in Knowledge-aware Recommendations with Large Language Models](https://mlsanity.com/paper/2412.13544/) (2024)
- [DiscoverLLM: From Executing Intents to Discovering Them](https://mlsanity.com/paper/2602.03429/) (2026)
- [Lusifer: LLM-based User SImulated Feedback Environment for online Recommender systems](https://mlsanity.com/paper/2405.13362/) (2024)
- [How can we assess human-agent interactions? Case studies in software agent design](https://mlsanity.com/paper/2510.09801/) (2025)
- [CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2403.19056/) (2024)
- [Simulating Tabular Datasets through LLMs to Rapidly Explore Hypotheses about Real-World Entities](https://mlsanity.com/paper/2411.18071/) (2024)
- [LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation](https://mlsanity.com/paper/2505.19667/) (2025)
- [LLM-guided Plan and Retrieval: A Strategic Alignment for Interpretable User Satisfaction Estimation in Dialogue](https://mlsanity.com/paper/2503.04675/) (2025)
- [Evaluating the Efficacy of Foundational Models: Advancing Benchmarking Practices to Enhance Fine-Tuning Decision-Making](https://mlsanity.com/paper/2407.11006/) (2024)
- [CoReflect: A Reflective Co-Evolution Framework for Improving Conversational Evaluation](https://mlsanity.com/paper/2601.12208/) (2026)
- [CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators](https://mlsanity.com/paper/2605.08334/) (2026)
- [From Static Context to Calibrated Interactive RL: Mitigating Distribution Shift in Multi-turn Dialogue with Aligned Simulator](https://mlsanity.com/paper/2605.26403/) (2026)
- [ANCHOR: Agentic Noise Creation Framework for Human Simulation and Denoising Recommendation](https://mlsanity.com/paper/2606.05621/) (2026)

## Abstract (arXiv)

Dialogue systems and large language models (LLMs) have gained considerable attention. However, the direct utilization of LLMs as task-oriented dialogue (TOD) models has been found to underperform compared to smaller task-specific models. Nonetheless, it is crucial to acknowledge the significant potential of LLMs and explore improved approaches for leveraging their impressive abilities. Motivated by the goal of leveraging LLMs, we propose an alternative approach called User-Guided Response Optimization (UGRO) to combine it with a smaller TOD model. This approach uses LLM as annotation-free user simulator to assess dialogue responses, combining them with smaller fine-tuned end-to-end TOD models. By utilizing the satisfaction feedback generated by LLMs, UGRO further optimizes the supervised fine-tuned TOD model. Specifically, the TOD model takes the dialogue history as input and, with the assistance of the user simulator's feedback, generates high-satisfaction responses that meet the user's requirements. Through empirical experiments on two TOD benchmarks, we validate the effectiveness of our method. The results demonstrate that our approach outperforms previous state-of-the-art (SOTA) results.