First Steps Towards Overhearing LLM Agents: A Case Study With Dungeons & Dragons Gameplay
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
Much work has been done on conversational LLM agents which directly assist human users with tasks. We present an alternative paradigm for interacting with LLM agents, which we call "overhearing agents". These overhearing agents do not actively participate in conversation -- instead, they "listen in" on human-to-human conversations and perform background tasks or provide suggestions to assist the user. In this work, we explore the overhearing agents paradigm through the lens of Dungeons & Dragons gameplay. We present an in-depth study using large multimodal audio-language models as overhearing agents to assist a Dungeon Master. We perform a human evaluation to examine the helpfulness of such agents and find that some large audio-language models have the emergent ability to perform overhearing agent tasks using implicit audio cues. Finally, we release Python libraries and our project code to support further research into the overhearing agents paradigm at https://github.com/zhudotexe/overhearing_agents.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 18 of its 81 references that are in the map. First posted on arXiv 28 May 2025.
Builds on
18 of 81 references.
References · 18
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2019The Curious Case of Neural Text Degenerationcited by 2,881
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
- 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
- 2023Efficient Guided Generation for Large Language Modelscited by 192
- 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
- 2024Large Language Models and Games: A Survey and Roadmapcited by 91
- 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
6 more
- 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learningcited by 74
- 2022Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writerscited by 69
- 2022Dungeons and Dragons as a Dialog Challenge for Artificial Intelligencecited by 35
- 2020Computer-Generated Music for Tabletop Role-Playing Gamescited by 23
- 2022I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragonscited by 22
- 2023Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applicationscited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap
- 2025Does Reasoning Help LLM Agents Play Dungeons and Dragons? A Prompt Engineering Experiment
- 2025SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents
- 2024Speaker Verification in Agent-Generated Conversations
- 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents
- 2024clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agents
- 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
- 2023Character-LLM: A Trainable Agent for Role-Playing