# Multi-Party Chat: Conversational Agents in Group Settings with Humans and Models

arXiv: https://arxiv.org/abs/2304.13835 · HTML: https://mlsanity.com/paper/2304.13835/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 19 of 19)

- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://mlsanity.com/paper/2208.03188/) (2022)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://mlsanity.com/paper/1506.08909/) (2015)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [Learning to Speak and Act in a Fantasy Text Adventure Game](https://mlsanity.com/paper/1903.03094/) (2019)
- [Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion](https://mlsanity.com/paper/2203.13224/) (2022)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [The CRINGE Loss: Learning what language not to model](https://mlsanity.com/paper/2211.05826/) (2022)
- [Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions](https://mlsanity.com/paper/2006.12442/) (2020)
- [A Survey of Available Corpora for Building Data-Driven Dialogue Systems](https://mlsanity.com/paper/1512.05742/) (2015)
- [Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity](https://mlsanity.com/paper/2112.05843/) (2021)
- [Generating Interactive Worlds with Text](https://mlsanity.com/paper/1911.09194/) (2019)
- [Mephisto: A Framework for Portable, Reproducible, and Iterative Crowdsourcing](https://mlsanity.com/paper/2301.05154/) (2023)

## Built on by (most cited first, 30 of 54)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models](https://mlsanity.com/paper/2310.00746/) (2023)
- [Small LLMs Are Weak Tool Learners: A Multi-LLM Agent](https://mlsanity.com/paper/2401.07324/) (2024)
- [Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character](https://mlsanity.com/paper/2405.20773/) (2024)
- [LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play](https://mlsanity.com/paper/2405.06373/) (2024)
- [MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models](https://mlsanity.com/paper/2402.01620/) (2024)
- [Proactive Conversational Agents with Inner Thoughts](https://mlsanity.com/paper/2501.00383/) (2024)
- [CodeAgent: Autonomous Communicative Agents for Code Review](https://mlsanity.com/paper/2402.02172/) (2024)
- [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023)
- [Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning](https://mlsanity.com/paper/2503.17662/) (2025)
- [Balancing Autonomy and Alignment: A Multi-Dimensional Taxonomy for Autonomous LLM-powered Multi-Agent Architectures](https://mlsanity.com/paper/2310.03659/) (2023)
- [RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2312.16132/) (2023)
- [RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems](https://mlsanity.com/paper/2409.16727/) (2024)
- [MASTER: A Multi-Agent System with LLM Specialized MCTS](https://mlsanity.com/paper/2501.14304/) (2025)
- [LawLuo: A Multi-Agent Collaborative Framework for Multi-Round Chinese Legal Consultation](https://mlsanity.com/paper/2407.16252/) (2024)
- [Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning](https://mlsanity.com/paper/2506.01748/) (2025)
- [Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulations](https://mlsanity.com/paper/2311.06330/) (2023)
- [CharacterGPT: A Persona Reconstruction Framework for Role-Playing Agents](https://mlsanity.com/paper/2405.19778/) (2024)
- [SimulBench: Evaluating Language Models with Creative Simulation Tasks](https://mlsanity.com/paper/2409.07641/) (2024)
- [Mixed-Session Conversation with Egocentric Memory](https://mlsanity.com/paper/2410.02503/) (2024)
- [Unveiling the Secrets of Engaging Conversations: Factors that Keep Users Hooked on Role-Playing Dialog Agents](https://mlsanity.com/paper/2402.11522/) (2024)
- [The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agents](https://mlsanity.com/paper/2502.20757/) (2025)
- [Multi-Party Conversational Agents: A Survey](https://mlsanity.com/paper/2505.18845/) (2025)
- [PIPPA: A Partially Synthetic Conversational Dataset](https://mlsanity.com/paper/2308.05884/) (2023)
- [RNR: Teaching Large Language Models to Follow Roles and Rules](https://mlsanity.com/paper/2409.13733/) (2024)
- [Decoding Hate: Exploring Language Models' Reactions to Hate Speech](https://mlsanity.com/paper/2410.00775/) (2024)
- [Enhancing Role-playing Systems through Aggressive Queries: Evaluation and Improvement](https://mlsanity.com/paper/2402.10618/) (2024)

## Abstract (arXiv)

Current dialogue research primarily studies pairwise (two-party) conversations, and does not address the everyday setting where more than two speakers converse together. In this work, we both collect and evaluate multi-party conversations to study this more general case. We use the LIGHT environment to construct grounded conversations, where each participant has an assigned character to role-play. We thus evaluate the ability of language models to act as one or more characters in such conversations. Models require two skills that pairwise-trained models appear to lack: (1) being able to decide when to talk; (2) producing coherent utterances grounded on multiple characters. We compare models trained on our new dataset to existing pairwise-trained dialogue models, as well as large language models with few-shot prompting. We find that our new dataset, MultiLIGHT, which we will publicly release, can help bring significant improvements in the group setting.