# GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

arXiv: https://arxiv.org/abs/2605.14498 · HTML: https://mlsanity.com/paper/2605.14498/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 14 May 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 25 of 25)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://mlsanity.com/paper/2507.05257/) (2025)
- [FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions](https://mlsanity.com/paper/2310.15421/) (2023)
- [Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward](https://mlsanity.com/paper/2602.12430/) (2026)
- [In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents](https://mlsanity.com/paper/2503.08026/) (2025)
- [OpenClaw-RL: Train Any Agent Simply by Talking](https://mlsanity.com/paper/2603.10165/) (2026)
- [PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory](https://mlsanity.com/paper/2512.06688/) (2025)
- [MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents](https://mlsanity.com/paper/2506.21605/) (2025)
- [MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks](https://mlsanity.com/paper/2602.16313/) (2026)
- [MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems](https://mlsanity.com/paper/2510.17281/) (2025)
- [AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications](https://mlsanity.com/paper/2602.22769/) (2026)
- [MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents](https://mlsanity.com/paper/2601.03236/) (2026)
- [Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects](https://mlsanity.com/paper/2512.12818/) (2025)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind](https://mlsanity.com/paper/2501.08838/) (2025)

## Built on by (most cited first, 17 of 17)

- [TOKI: A Bitemporal Operator Algebra for Contradiction Resolution in LLM-Agent Persistent Memory](https://mlsanity.com/paper/2606.06240/) (2026)
- [MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations](https://mlsanity.com/paper/2607.12893/) (2026)
- [Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgents](https://mlsanity.com/paper/2606.30306/) (2026)
- [DarkForest: Less Talk, Higher Accuracy for Multi-Agent LLMs](https://mlsanity.com/paper/2605.25188/) (2026)
- [Authorization Before Context: A Model-Neutral Audience Boundary Against Cross-Audience Memory Leakage in Agentic Systems](https://mlsanity.com/paper/2608.17148/) (2026)
- [CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologies](https://mlsanity.com/paper/2609.32192/) (2026)
- [MemFuse: Multi-Source Memory Fusion from Fragmented Observations](https://mlsanity.com/paper/2608.18704/) (2026)
- [Propose, Verify, Commit: Evidence-Grounded Memory for Long-Horizon Multi-Actor Conversations](https://mlsanity.com/paper/2609.23465/) (2026)
- [SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue](https://mlsanity.com/paper/2609.26780/) (2026)
- [Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations](https://mlsanity.com/paper/2609.32522/) (2026)
- [Stashbird: Efficient Speaker-Indexed Memory for Conversational Agents](https://mlsanity.com/paper/2609.34242/) (2026)
- [Org-Agent: Beyond Personal Assistants Towards Organizational Agents](https://mlsanity.com/paper/2609.34392/) (2026)
- [Audience-Bound Persistent Memory: Authorization Across the Memory Lifecycle](https://mlsanity.com/paper/2609.36373/) (2026)
- [PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue](https://mlsanity.com/paper/2610.07725/) (2026)
- [Temporal Evidence Forests for Long-Horizon Multi-Party Conversational Memory](https://mlsanity.com/paper/2610.12874/) (2026)
- [Knowing the User Is Not Knowing What We Established: Grounding Awareness in Personal Agents](https://mlsanity.com/paper/2610.18684/) (2026)
- [EGCForest: Structured Agent Memory Organization for Enterprise Group Chats](https://mlsanity.com/paper/2610.19836/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) agents increasingly serve as personal assistants and workplace collaborators, where their utility depends on memory systems that extract, retrieve, and apply information across long-running conversations. However, both existing memory systems and benchmarks are built around the dyadic, single-user setup, even though real deployments routinely span groups and channels with multiple users interacting with the agent and with each other. This mismatch leaves three properties of group memory unmeasured: (i) group dynamics that go beyond concatenated one-on-one chats, (ii) speaker-grounded belief tracking, where the per-user memory modeling is needed, and (iii) audience-adapted language, where Theory-of-Mind shifts produce role-specific vocabulary. We introduce GroupMemBench, a benchmark that exposes all three. A graph-grounded synthesis pipeline produces multi-party conversations with controllable reply structure and conditions each message on per-user personas and target audiences. An adversarial query pipeline then binds every question to a specific asker across six categories, spanning multi-hop reasoning, knowledge update, term ambiguity, user-implicit reasoning, temporal reasoning, and abstention, and iteratively searches challenging, realistic queries that reflect comprehensive memory capability. Benchmarking leading memory systems exposes a sharp collapse: the strongest one reaches only 46.0% average accuracy, with knowledge update at 27.1% and term ambiguity at 37.7%, while a simple BM25 baseline matches or exceeds most agent memory systems. This indicates current memory ingestion erases the structural and lexical features group memory depends on, leaving multi-user memory far from solved.