# Human vs. Agent in Task-Oriented Conversations

arXiv: https://arxiv.org/abs/2509.17619 · HTML: https://mlsanity.com/paper/2509.17619/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 20 of 20)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems](https://mlsanity.com/paper/2310.09233/) (2023)
- [Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?](https://mlsanity.com/paper/2406.13605/) (2024)
- [Description-Driven Task-Oriented Dialog Modeling](https://mlsanity.com/paper/2201.08904/) (2022)
- [Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models](https://mlsanity.com/paper/2305.13112/) (2023)
- [Evaluating Conversational Recommender Systems via User Simulation](https://mlsanity.com/paper/2006.08732/) (2020)
- [Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems](https://mlsanity.com/paper/2105.03748/) (2021)
- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue System](https://mlsanity.com/paper/2306.09821/) (2023)
- [How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendation](https://mlsanity.com/paper/2403.16416/) (2024)
- [Evaluating Mixed-initiative Conversational Search Systems via User Simulation](https://mlsanity.com/paper/2204.08046/) (2022)
- [DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialogues](https://mlsanity.com/paper/2405.13028/) (2024)
- [A LLM-based Controllable, Scalable, Human-Involved User Simulator Framework for Conversational Recommender Systems](https://mlsanity.com/paper/2405.08035/) (2024)
- [UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems](https://mlsanity.com/paper/2301.05544/) (2023)
- [LLM-based Frameworks for API Argument Filling in Task-Oriented Conversational Systems](https://mlsanity.com/paper/2407.12016/) (2024)
- [LLM Agents Display Human Biases but Exhibit Distinct Learning Patterns](https://mlsanity.com/paper/2503.10248/) (2025)

## Built on by (most cited first, 9 of 9)

- [Mind the Sim2Real Gap in User Simulation for Agentic Tasks](https://mlsanity.com/paper/2603.11245/) (2026)
- [Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations](https://mlsanity.com/paper/2601.17087/) (2026)
- [Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants](https://mlsanity.com/paper/2605.09808/) (2026)
- [Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M+ Real Search Requests](https://mlsanity.com/paper/2601.17617/) (2026)
- [VISTA: A Versatile Interactive User Simulation Toolkit for Agent Evaluation](https://mlsanity.com/paper/2606.11079/) (2026)
- [Personalized Turn-Level User Conversation Satisfaction Benchmark](https://mlsanity.com/paper/2605.29711/) (2026)
- [Communication Policy Evolution for Proactive LLM Agents](https://mlsanity.com/paper/2606.14314/) (2026)
- [Measuring the Behavioral Fidelity of Long-Horizon Human Activity Simulations](https://mlsanity.com/paper/2609.01257/) (2026)
- [Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution](https://mlsanity.com/paper/2609.07594/) (2026)

## Abstract (arXiv)

Task-oriented conversational systems are essential for efficiently addressing diverse user needs, yet their development requires substantial amounts of high-quality conversational data that is challenging and costly to obtain. While large language models (LLMs) have demonstrated potential in generating synthetic conversations, the extent to which these agent-generated interactions can effectively substitute real human conversations remains unclear. This work presents the first systematic comparison between LLM-simulated users and human users in personalized task-oriented conversations. We propose a comprehensive analytical framework encompassing three key aspects (conversation strategy, interaction style, and conversation evaluation) and ten distinct dimensions for evaluating user behaviors, and collect parallel conversational datasets from both human users and LLM agent users across four representative scenarios under identical conditions. Our analysis reveals significant behavioral differences between the two user types in problem-solving approaches, question broadness, user engagement, context dependency, feedback polarity and promise, language style, and hallucination awareness. We found consistency in the agent users and human users across the depth-first or breadth-first dimensions, as well as the usefulness dimensions. These findings provide critical insights for advancing LLM-based user simulation. Our multi-dimensional taxonomy constructed a generalizable framework for analyzing user behavior patterns, offering insights from LLM agent users and human users. By this work, we provide perspectives on rethinking how to use user simulation in conversational systems in the future.