# DialogBench: Evaluating LLMs as Human-like Dialogue Systems

arXiv: https://arxiv.org/abs/2311.01677 · HTML: https://mlsanity.com/paper/2311.01677/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 86 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Builds on (most cited first, 30 of 62)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)

## Built on by (most cited first, 30 of 31)

- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents](https://mlsanity.com/paper/2601.03515/) (2026)
- [TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Models](https://mlsanity.com/paper/2503.11656/) (2025)
- [RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems](https://mlsanity.com/paper/2409.16727/) (2024)
- [NarraBench: A Comprehensive Framework for Narrative Benchmarking](https://mlsanity.com/paper/2510.09869/) (2025)
- [Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency](https://mlsanity.com/paper/2402.19366/) (2024)
- [MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems](https://mlsanity.com/paper/2412.15557/) (2024)
- [SEAD: Self-Evolving Agent for Multi-Turn Service Dialogue](https://mlsanity.com/paper/2602.03548/) (2026)
- [Seven simple steps for log analysis in AI systems](https://mlsanity.com/paper/2604.09563/) (2026)
- [Enhancing Role-playing Systems through Aggressive Queries: Evaluation and Improvement](https://mlsanity.com/paper/2402.10618/) (2024)
- [LLM-Driven NPCs: Cross-Platform Dialogue System for Games and Social Platforms](https://mlsanity.com/paper/2504.13928/) (2025)
- [CAPE: Context-Aware Personality Evaluation Framework for Large Language Models](https://mlsanity.com/paper/2508.20385/) (2025)
- [FileGram: Grounding Agent Personalization in File-System Behavioral Traces](https://mlsanity.com/paper/2604.04901/) (2026)
- [NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews](https://mlsanity.com/paper/2411.13779/) (2024)
- [Adaptive Multi-Agent Response Refinement in Conversational Systems](https://mlsanity.com/paper/2511.08319/) (2025)
- [SAGE: A Service Agent Graph-guided Evaluation Benchmark](https://mlsanity.com/paper/2604.09285/) (2026)
- [EiCAP: Beyond Fluency, Probing and Improving Emotional Intelligence in LLMs via Psychologically Grounded Multi-Turn Dialogue](https://mlsanity.com/paper/2508.06196/) (2025)
- [Testing the Testers: Human-Driven Quality Assessment of Voice AI Testing Platforms](https://mlsanity.com/paper/2511.04133/) (2025)
- [A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents](https://mlsanity.com/paper/2601.10809/) (2026)
- [Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialogue](https://mlsanity.com/paper/2602.22697/) (2026)
- [On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Models](https://mlsanity.com/paper/2410.03996/) (2024)
- [CAPE: A Chinese Dataset for Appraisal-based Emotional Generation using Large Language Models](https://mlsanity.com/paper/2410.14145/) (2024)
- [Planning with Diffusion Models for Target-Oriented Dialogue Systems](https://mlsanity.com/paper/2504.16858/) (2025)
- [Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques](https://mlsanity.com/paper/2507.13629/) (2025)
- [CoReflect: A Reflective Co-Evolution Framework for Improving Conversational Evaluation](https://mlsanity.com/paper/2601.12208/) (2026)
- [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026)
- [Are Emotion and Rhetoric Neurons in LLM? Neuron Recognition and Adaptive Masking for Emotion-Rhetoric Prediction Steering](https://mlsanity.com/paper/2604.17255/) (2026)
- [BLAST: Benchmarking LLMs with ASP-based Structured Testing](https://mlsanity.com/paper/2604.22306/) (2026)
- [Measuring Semantic Progress in Multi-turn Dialogue via Information Gain](https://mlsanity.com/paper/2606.12332/) (2026)
- [Conversation as Measurement in Clinical Encounters: Observable Phase Structure, Partially Observable Patient State](https://mlsanity.com/paper/2608.08868/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved remarkable breakthroughs in new dialogue capabilities by leveraging instruction tuning, which refreshes human impressions of dialogue systems. The long-standing goal of dialogue systems is to be human-like enough to establish long-term connections with users. Therefore, there has been an urgent need to evaluate LLMs as human-like dialogue systems. In this paper, we propose DialogBench, a dialogue evaluation benchmark that contains 12 dialogue tasks to probe the capabilities of LLMs as human-like dialogue systems should have. Specifically, we prompt GPT-4 to generate evaluation instances for each task. We first design the basic prompt based on widely used design principles and further mitigate the existing biases to generate higher-quality evaluation instances. Our extensive tests on English and Chinese DialogBench of 26 LLMs show that instruction tuning improves the human likeness of LLMs to a certain extent, but most LLMs still have much room for improvement as human-like dialogue systems. Interestingly, results also show that the positioning of assistant AI can make instruction tuning weaken the human emotional perception of LLMs and their mastery of information about human daily life.